[00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:22:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:15:39] (ProbeDown) firing: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:20:38] (ProbeDown) resolved: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:24:22] (03PS1) 10AntiCompositeNumber: SULWatcher: attempt to catch EventStreams exceptions [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023559 [01:24:51] (03CR) 10CI reject: [V:04-1] SULWatcher: attempt to catch EventStreams exceptions [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023559 (owner: 10AntiCompositeNumber) [01:30:14] (03PS2) 10AntiCompositeNumber: SULWatcher: attempt to catch EventStreams exceptions [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023559 [02:00:41] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286 (10Huji) 03NEW [02:01:21] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9739011 (10Huji) [02:01:46] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9739009 (10Huji) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace... [02:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:16:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:21:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:26:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:06:49] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290 (10matmarex) 03NEW [03:52:00] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:57:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:59:30] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:03:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:05:30] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:08:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:40:03] (ToolforgeKubernetesWorkerTooManyDProcesses) firing: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:42:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:45:03] (ToolforgeKubernetesWorkerTooManyDProcesses) resolved: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [04:52:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:55:43] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290#9739118 (10hashar) On the counterpart, if the listed reviewers are added to the CC field, they would not be put in the attention set which kind... [06:28:28] (PuppetAgentNoResources) firing: (10) No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:33:28] (PuppetAgentNoResources) firing: (10) No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:38:28] (PuppetAgentNoResources) firing: (10) No Puppet resources found on instance runner-1021 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:48:28] (PuppetAgentNoResources) firing: (8) No Puppet resources found on instance runner-1022 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [06:49:36] 14Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9739155 (10MBH) Thank you, but I prefer traditional "build locally - transfer to server - run exe file" way as way more convenient and simple. Another reason is that I rea... [06:53:28] (PuppetAgentNoResources) resolved: (5) No Puppet resources found on instance runner-1025 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:23:21] 10Data-Services, 06DBA: Prepare and check storage layer for sysop_plwiki - https://phabricator.wikimedia.org/T363276#9739204 (10ABran-WMF) p:05Triage→03Medium [07:23:37] 10Data-Services, 06DBA: Prepare and check storage layer for mywikisource - https://phabricator.wikimedia.org/T363269#9739207 (10ABran-WMF) p:05Triage→03Medium [07:24:39] 10Data-Services, 06DBA: Prepare and check storage layer for mswikisource - https://phabricator.wikimedia.org/T363249#9739215 (10ABran-WMF) p:05Triage→03Medium [07:24:40] 10Data-Services, 06DBA: Prepare and check storage layer for iglwiki - https://phabricator.wikimedia.org/T363262#9739211 (10ABran-WMF) p:05Triage→03Medium [07:24:52] 10Data-Services, 06DBA: Prepare and check storage layer for kawikisource - https://phabricator.wikimedia.org/T363242#9739217 (10ABran-WMF) p:05Triage→03Medium [07:25:16] 10Data-Services, 06DBA: Prepare and check storage layer for kaawiktionary - https://phabricator.wikimedia.org/T363255#9739213 (10ABran-WMF) p:05Triage→03Medium [07:32:24] 10Tool-itwiki, 06Commons: Auto-calculate the bearing from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#9739231 (10valerio.bozzolan) [08:08:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:09:50] 06cloud-services-team, 10Toolforge: toolforge: explore options to introduce egress network quotas - https://phabricator.wikimedia.org/T363296 (10aborrero) 03NEW [08:11:14] 10Toolforge, 10Tools, 06Data-Engineering, 10EventStreams, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9739324 (10aborrero) >>! In T329327#9738585, @bd808 wrote: >... [08:11:57] 06cloud-services-team, 10Toolforge: toolforge: explore options to introduce egress network quotas - https://phabricator.wikimedia.org/T363296#9739320 (10aborrero) p:05Triage→03Medium [08:12:31] 14Toolforge (Toolforge iteration 07): [toolforge] several tools get periods of connection refused (104) when connecting to wikis - https://phabricator.wikimedia.org/T356164#9739327 (10aborrero) >>! In T356164#9621026, @dcaro wrote: >>>! In T356164#9559316, @aborrero wrote: >> Maybe an idea: have a per-tool n... [09:26:40] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: cloud-vps monitoring - https://phabricator.wikimedia.org/T362452#9739581 (10dcaro) Notes: Apr 24, 2024 | Cloud VPS monitoring systems Attendees: Arturo Borrero David Caro Francesco Negri Taavi Väänänen Notes * https://wikitech.wiki... [09:28:28] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Taavi knowledge transfer: Cloud VPS OpenTofu provider - https://phabricator.wikimedia.org/T362450#9739588 (10dcaro) Notes: Apr 24, 2024 | Cloud VPS OpenTofu provider Attendees: Arturo Borrero c_7b9ad6d28760abb302f0909412d1ed85b8d1db6ade03cbf2242fededb17164f1@... [09:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:48:12] 10Data-Services, 06DBA: Prepare and check storage layer for kuswiki - https://phabricator.wikimedia.org/T360302#9739632 (10Ladsgroup) p:05Triage→03Medium [09:48:19] 10Data-Services, 06DBA: Prepare and check storage layer for bewwiki - https://phabricator.wikimedia.org/T360309#9739633 (10Ladsgroup) p:05Triage→03Medium [09:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:56:01] 10Data-Services, 06DBA: Prepare and check storage layer for kawikisource - https://phabricator.wikimedia.org/T363242#9739641 (10Ladsgroup) Ready for data engineering people to take over [09:56:03] 10Data-Services, 06DBA: Prepare and check storage layer for mswikisource - https://phabricator.wikimedia.org/T363249#9739644 (10Ladsgroup) Ready for data engineering people to take over [09:56:32] 10Data-Services, 06DBA: Prepare and check storage layer for bewwiki - https://phabricator.wikimedia.org/T360309#9739653 (10Ladsgroup) Ready for data engineering people to take over [09:57:02] 10Data-Services, 06DBA: Prepare and check storage layer for kuswiki - https://phabricator.wikimedia.org/T360302#9739650 (10Ladsgroup) Ready for data engineering people to take over [09:57:03] 10Data-Services, 06DBA: Prepare and check storage layer for kaawiktionary - https://phabricator.wikimedia.org/T363255#9739647 (10Ladsgroup) Ready for data engineering people to take over [09:57:14] 10Data-Services, 06DBA: Prepare and check storage layer for mywikisource - https://phabricator.wikimedia.org/T363269#9739660 (10Ladsgroup) Ready for data engineering people to take over [09:57:35] 10Data-Services, 06DBA: Prepare and check storage layer for iglwiki - https://phabricator.wikimedia.org/T363262#9739657 (10Ladsgroup) Ready for data engineering people to take over [09:59:10] 10Data-Services, 06DBA: Prepare and check storage layer for sysop_plwiki - https://phabricator.wikimedia.org/T363276#9739671 (10Ladsgroup) This will be messy. It's a private wiki. [10:24:51] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for kuswiki - https://phabricator.wikimedia.org/T360302#9739755 (10taavi) 05Open→03Resolved a:03taavi [10:37:31] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for iglwiki - https://phabricator.wikimedia.org/T363262#9739820 (10taavi) [10:39:49] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for mswikisource - https://phabricator.wikimedia.org/T363249#9739851 (10taavi) [10:40:13] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for iglwiki - https://phabricator.wikimedia.org/T363262#9739834 (10taavi) 05Open→03Resolved a:03taavi [10:40:21] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for mswikisource - https://phabricator.wikimedia.org/T363249#9739847 (10taavi) 05Open→03Resolved a:03taavi [10:41:08] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for kaawiktionary - https://phabricator.wikimedia.org/T363255#9739841 (10taavi) 05Open→03Resolved a:03taavi [10:45:25] 10Toolforge: Buildservice "network is unreachable" error - https://phabricator.wikimedia.org/T362958#9739867 (10dcaro) @Magnus Is this happening often? I just created an MR to show the node the build is running on, so we can debug this better (https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/... [10:47:26] 10Tool-openstack-browser: openstack-browser: List extra allowed service IPs on server detail - https://phabricator.wikimedia.org/T360541#9739868 (10taavi) [10:48:11] 10Tool-openstack-browser: openstack-browser: Show DNS subzone delegation - https://phabricator.wikimedia.org/T363311 (10taavi) 03NEW [10:52:43] 10Toolforge: Query appears to run for a longer time when invoked via toolforge jobs framework - https://phabricator.wikimedia.org/T363286#9739886 (10taavi) Which database is the query running against? fawiki? And are you running it on the analytics or web replicas? [10:55:35] 10Toolforge: /mnt/nfs/labstore-secondary-tools-project no longer seems to be mounted in the new container on Toolforge - https://phabricator.wikimedia.org/T363087#9739890 (10taavi) > Since /data/project is a symbolic link to /mnt/nfs/labstore-secondary-tools-project on Toolforge, some tools such as pnpm will wri... [11:11:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:21:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:26:01] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9739931 (10aborrero) 05In progress→03Resolved a:03aborrero I'm fine with option 1 too, so I'm declaring this decision request done. [11:26:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:28:01] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9739945 (10aborrero) [11:29:23] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9739948 (10Tgr) > Con (long-term): Existing value (if any) of the separation between Wikitech and Mediawiki.org would be lost. E.g. it might be less obvi... [11:38:44] (03CR) 10Samtar: [C:03+2] SULWatcher: attempt to catch EventStreams exceptions [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023559 (owner: 10AntiCompositeNumber) [11:39:18] (03Merged) 10jenkins-bot: SULWatcher: attempt to catch EventStreams exceptions [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023559 (owner: 10AntiCompositeNumber) [11:41:57] (03CR) 10Samtar: [C:03+2] SULWatcher: add PingServer mixin to handle ping timeouts [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023548 (owner: 10AntiCompositeNumber) [11:42:18] (03PS2) 10AntiCompositeNumber: SULWatcher: add PingServer mixin to handle ping timeouts [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023548 [11:43:31] (03CR) 10Samtar: [C:03+2] SULWatcher: add PingServer mixin to handle ping timeouts [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023548 (owner: 10AntiCompositeNumber) [11:44:07] (03Merged) 10jenkins-bot: SULWatcher: add PingServer mixin to handle ping timeouts [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023548 (owner: 10AntiCompositeNumber) [11:44:35] (03PS4) 10AntiCompositeNumber: StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 [11:45:53] (03CR) 10Samtar: [C:03+2] StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 (owner: 10AntiCompositeNumber) [11:46:32] (03Merged) 10jenkins-bot: StewardBot: add health check script based on canary events [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1021004 (owner: 10AntiCompositeNumber) [11:59:23] 10Toolforge, 10Tools, 06Data-Engineering, 10EventStreams, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9740030 (10Ottomata) This is probably not helpful, but EventSt... [12:03:18] 10Tool-masto-collab: HTTP status client error (422 Unprocessable Entity) on posting with media - https://phabricator.wikimedia.org/T363314 (10TheresNoTime) 03NEW [12:13:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:13:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:15:54] 10Striker: Searching for "tools" in Striker does not work - https://phabricator.wikimedia.org/T363320 (10taavi) 03NEW [12:17:58] 10Tool-masto-collab: HTTP status client error (422 Unprocessable Entity) on posting with media - https://phabricator.wikimedia.org/T363314#9740158 (10TheresNoTime) nb. I have logged some `ROCKET_LOG_LEVEL=debug` output to `/data/project/masto-collab/debug.log` if it helps [12:23:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:52:44] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for kawikisource - https://phabricator.wikimedia.org/T363242#9740341 (10taavi) 05Open→03Resolved a:03taavi [13:28:57] 10Tool-global-search: Even if I change the language it doesn't reflect - https://phabricator.wikimedia.org/T363335 (10Chqaz) 03NEW [13:30:16] 10Tools, 06Tech-Docs-Team, 07Documentation, 03Wikimedia-Hackathon-2024: [Hackathon 2024] Improve technical documentation of tools - https://phabricator.wikimedia.org/T358040#9740497 (10TBurmeister) a:05TBurmeister→03JorisDarlingtonQuarshie [13:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:46:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:51:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:52:56] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Learn how to do what Taavi does - https://phabricator.wikimedia.org/T362443#9740599 (10dcaro) For the "other tools + striker" sync meeting that has no task, the notes are: Apr 24, 2024 | Misc tools + Striker Attendees: Arturo Borrero c_7b9ad6d28760abb302f090... [13:56:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:45:14] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9740734 (10Jclark-ctr) 05Open→03Resolved [15:08:53] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9740835 (10aborrero) [15:09:12] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9740822 (10aborrero) 05Resolved→03Open reopening, as I just noticed an important data point: as of today PodSecurityPolicy work on mutati... [15:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:12:50] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9740862 (10aborrero) [15:12:54] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install cloudcephosd10[39-41] - https://phabricator.wikimedia.org/T363341 (10RobH) 03NEW [15:13:10] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install cloudcephosd10[39-41] - https://phabricator.wikimedia.org/T363341#9740879 (10RobH) [15:13:17] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9740880 (10dcaro) >>! In T362872#9740822, @aborrero wrote: > reopening, as I just noticed an important data point: as of today PodSecurityPol... [15:16:39] 10PAWS: Test PAWS on k8s 1.25 - https://phabricator.wikimedia.org/T326985#9740915 (10rook) 05Stalled→03In progress [15:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:20:17] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9740931 (10dcaro) 05Resolved→03Open @Jclark-ctr the disk does not show up: ` root@cloudcephosd1017:~# lsblk NAME... [15:20:39] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install cloudcephosd10[35-38] - https://phabricator.wikimedia.org/T363344 (10RobH) 03NEW [15:21:05] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install cloudcephosd10[35-38] - https://phabricator.wikimedia.org/T363344#9740953 (10RobH) [15:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:25:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:25:33] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [15:26:57] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: Decision Request - Toolforge policy agent enforcement model - https://phabricator.wikimedia.org/T362872#9740978 (10aborrero) >>! In T362872#9740880, @dcaro wrote: > > Is there a way for us to see how many objects are currently not meeting the p... [15:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:28:22] 10Toolforge: [jobs-api,builds-api,envvars-api,api-gateway] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346 (10dcaro) 03NEW [15:28:29] 10Toolforge: [jobs-api,builds-api,envvars-api,api-gateway] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9740995 (10dcaro) p:05Triage→03High [15:29:48] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [15:29:49] 06cloud-services-team, 10Toolforge: toolforge lima-kilo: PodSecurityPolicy admission is disabled - https://phabricator.wikimedia.org/T363347 (10aborrero) 03NEW [15:30:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [15:31:41] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09): [builds-api] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#9741019 (10fnegri) The dashboard is at https://grafana-rw.wmcloud.org/d/f4Sxgf-Sz/builds-api I think it includes all the data listed... [15:32:04] 06cloud-services-team, 10Toolforge: toolforge lima-kilo: PodSecurityPolicy admission is disabled - https://phabricator.wikimedia.org/T363347#9741013 (10aborrero) 05Open→03In progress p:05Triage→03High [16:11:16] 10Toolforge, 10Tools, 06Data-Engineering, 10EventStreams, and 2 others: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327#9741202 (10bd808) >>! In T329327#9739317, @aborrero wrote: > I... [16:18:45] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:20:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:21:15] (03PS1) 10Btullis: Add dummy keytabs for new stats servers [labs/private] - 10https://gerrit.wikimedia.org/r/1023889 (https://phabricator.wikimedia.org/T336040) [16:21:35] (03CR) 10Btullis: [V:03+2 C:03+2] Add dummy keytabs for new stats servers [labs/private] - 10https://gerrit.wikimedia.org/r/1023889 (https://phabricator.wikimedia.org/T336040) (owner: 10Btullis) [16:23:56] 06cloud-services-team, 10Toolforge: toolforge: explore options to introduce egress network quotas - https://phabricator.wikimedia.org/T363296#9741291 (10bd808) I love the idea of attempting to create a more equitable resource distribution for networking in Toolforge. I'm not sure yet however how this would act... [16:30:00] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:37:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:39:57] 10Tool-refill: Make reFill available for Urdu - https://phabricator.wikimedia.org/T352382#9741354 (10Pppery) [16:40:30] 10Tool-bridgebot, 07Upstream: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487#9741362 (10bd808) [16:40:31] 10Tool-bridgebot: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9741361 (10bd808) [16:40:44] 10Tool-bridgebot: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9741356 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [16:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:43:49] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [prometheus] [grafana] set scrape interval in data source config - https://phabricator.wikimedia.org/T363176#9741377 (10fnegri) p:05Triage→03Medium [16:46:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:51:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:56:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:21:07] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9741547 (10Jclark-ctr) cloudcephosd1017 looks like the drive was listed as foreign I cleared the foreign status can you verify it... [17:24:13] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T359049) [17:24:47] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T359049) [17:24:58] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T359049) [17:29:26] !log dcaro@urcuchillay admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T359049) [17:29:39] (ProbeDown) firing: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:29:51] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9741664 (10Jclark-ctr) [17:30:03] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T359049) [17:31:00] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: hw troubleshooting: /dev/sdg disk not working properly in cloudcephosd1017.eqiad.wmnet - https://phabricator.wikimedia.org/T359049#9741671 (10dcaro) \o/ the drive is listed now, will add it to the cluster (will take a bit), and close the task once... [17:34:38] (ProbeDown) resolved: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:37:09] (CephClusterInWarning) firing: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:42:00] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:42:09] (CephClusterInWarning) resolved: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [18:11:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:26:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:50:12] (03PS1) 10AntiCompositeNumber: SULWatcher: Fix inheritance order for LiberaBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023931 [18:51:27] (03CR) 10AntiCompositeNumber: [C:03+2] SULWatcher: Fix inheritance order for LiberaBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023931 (owner: 10AntiCompositeNumber) [18:52:02] (03Merged) 10jenkins-bot: SULWatcher: Fix inheritance order for LiberaBot [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1023931 (owner: 10AntiCompositeNumber) [19:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:17:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:40:00] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742399 (10Dzahn) [19:40:41] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9742391 (10Dzahn) 05Open→03Resolved Things have been working better since we gave it more resources. Closing ag... [19:54:52] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742594 (10Dzahn) created new "puppet prefix" in Horizon called "deploy" - then used it to apply role and needed Hiera keys for a dep... [20:00:48] 06cloud-services-team, 10wikitech.wikimedia.org, 07Epic, 07Security: sustainability of wikitech.wikimedia.org - https://phabricator.wikimedia.org/T363125#9742634 (10nshahquinn-wmf) >>! In T363125#9739948, @Tgr wrote: >> Con (long-term): Existing value (if any) of the separation between Wikitech and Mediawi... [20:00:58] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742631 (10Dzahn) 05Open→03Stalled deploy-1006 is on bullseye and the current status is "puppet error with duplicate declaration r... [20:09:58] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742677 (10Dzahn) per comments on https://gerrit.wikimedia.org/r/c/operations/puppet/+/820749/14/modules/scap/manifests/master.pp#69... [20:11:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:16:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:23:46] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:04:17] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742835 (10Dzahn) [21:04:43] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services, 13Patch-For-Review: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9742841 (10Dzahn) created new subtask about deployment servers on bullseye in general - stalled on that [21:11:24] 10Toolforge: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417 (10bd808) 03NEW [21:16:17] 10Toolforge: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9742882 (10bd808) We are a few versions behind on https://github.com/heroku/buildpacks-go, but I don't see anything [[https://github.com/heroku/buildpacks-go/compare/v0.1.13...v0.2.1|in the co... [21:27:21] 10Tool-bridgebot: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9742914 (10bd808) `lang=shell-session $ ssh login.toolforge.org $ become bridgebot $ webservice buildservice shell --mount all -m 2G -c 1 $ /layers/heroku_go/go_target/bin/bridgebot -conf /a... [21:54:26] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290#9742949 (10matmarex) >>! In T363290#9739118, @hashar wrote: > On the counterpart, if the listed reviewers are added to the CC field, they would... [21:57:10] 10Toolforge: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9742955 (10bd808) Adding tiny shell wrappers for the Procfile to call seems to work around the issue. [22:46:52] 10Tools, 10Gerrit, 03Wikimedia-Hackathon-2024: Gerrit reviewer bot should add reviewers as CC instead of actual reviewers - https://phabricator.wikimedia.org/T363290#9743037 (10bd808) >>! In T363290#9742949, @matmarex wrote: > I feel like people mostly use it for notifications, and don't intend to actually r... [23:07:23] 10Toolforge, 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9743103 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/1 Fix issues found during live testing of initial implemen... [23:07:40] 10Tool-bridgebot, 13Patch-For-Review: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9743105 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/1 Fix issues found during live testing of initial implem... [23:13:02] 10Toolforge, 13Patch-For-Review: Golang and Procfile buildpacks not working together as expected - https://phabricator.wikimedia.org/T363417#9743132 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/1 Fix issues found during live testing of initial implemen... [23:13:05] 10Tool-bridgebot, 13Patch-For-Review: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9743136 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/1 Fix issues found during live testing of initial implem... [23:17:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:22:39] 10VPS-project-Wikistats: Add mywikisource to wikistats - https://phabricator.wikimedia.org/T363274#9743143 (10Dzahn) a:03Dzahn [23:31:35] 10Tool-bridgebot: Replace custom deployment with build service and job service - https://phabricator.wikimedia.org/T363028#9743151 (10bd808) I have seen one crash on startup in testing but it was not repeatable. It looks like it was triggered by something the irc client saw in scrollback when attaching: `lang=go...