[00:11:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:12:26] 10Wikibugs, 13Patch-For-Review: Reimagine channel configuration (re)loading to avoid need for git pull - https://phabricator.wikimedia.org/T360860#9671982 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/23 Reimagine channel configuration (re)loading to av... [00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:45:56] (CloudVPSDesignateLeaks) firing: (5) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:04:01] (03PS1) 10Krinkle: app: Enable CORS on error response to improve error handling [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:04:13] 10Wikibugs: Reimagine channel configuration (re)loading to avoid need for git pull - https://phabricator.wikimedia.org/T360860#9672029 (10bd808) This is "fun". [[https://github.com/ronf/asyncssh/blob/a93224fb52d0da5e95e98ce23d596f288e3842ae/asyncssh/connection.py#L7675|asyncssh/connection.py]] raises an exceptio... [01:17:45] (03PS2) 10Krinkle: Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:18:31] (03CR) 10CI reject: [V:04-1] Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 (owner: 10Krinkle) [01:20:57] (03PS3) 10Krinkle: Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:21:23] (03PS4) 10Krinkle: Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:22:09] (03CR) 10CI reject: [V:04-1] Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 (owner: 10Krinkle) [01:23:25] (03PS5) 10Krinkle: Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:24:09] (03CR) 10CI reject: [V:04-1] Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 (owner: 10Krinkle) [01:25:09] (03PS6) 10Krinkle: Enable CORS for app.py errors, and display in frontend [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1015432 [01:28:35] 10VPS-project-Codesearch, 13Patch-For-Review: 14Codesearch: Fix ShoutHow repository viewer links to link to a viewable version of the matching file instead of prompting the user to download the file - 14https://phabricator.wikimedia.org/T304879#9672076 (10Krinkle) 05Open→03Resolved p:05Triage→03Medi... [01:55:47] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9672100 (10Diskdance) [02:38:05] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9672123 (10Cstone) [02:45:56] (03CR) 10Krinkle: [V:03+2 C:03+2] build: Add .gitreview file [labs/tools/editorinteract] - 10https://gerrit.wikimedia.org/r/1015431 (owner: 10Krinkle) [02:53:15] (03PS2) 10Reedy: releases: Bump php-parallel-lint/php-parallel-lint to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015205 (https://phabricator.wikimedia.org/T361217) [04:45:56] (CloudVPSDesignateLeaks) firing: (5) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:17:28] (03CR) 10Krinkle: [C:03+2] releases: Bump php-parallel-lint/php-parallel-lint to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015205 (https://phabricator.wikimedia.org/T361217) (owner: 10Reedy) [05:18:02] (03Merged) 10jenkins-bot: releases: Bump php-parallel-lint/php-parallel-lint to 1.4.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015205 (https://phabricator.wikimedia.org/T361217) (owner: 10Reedy) [08:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:06:30] 10Cloud-VPS, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools: spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are: Another puppet instance is already running and the waitforlock setting is set to 0; e... - https://phabricator.wikimedia.org/T361218#9672481 [10:06:38] 10Cloud-VPS, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools: spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are: Another puppet instance is already running and the waitforlock setting is set to 0; e... - https://phabricator.wikimedia.org/T361218#9672483 [10:08:23] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459#9672522 (10Jelto) [12:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:00:40] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update gitlab-runners project puppetmaster - https://phabricator.wikimedia.org/T360459#9672737 (10Andrew) This is done. The old puppetmaster (gitlab-runners-puppetmaster-01) is shut down; please confirm that you're happ... [13:03:05] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9672738 (10Andrew) 05Resolved→03Open a:05Jhancock.wm→03Andrew [13:22:06] (ProbeDown) firing: (2) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:27:06] (ProbeDown) resolved: (2) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:22:28] (InstanceDown) firing: Project toolsbeta instance toolsbeta-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:27:28] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-harbor-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:40:46] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q#:rack/setup/install (2) cloudbackup hosts - https://phabricator.wikimedia.org/T356216#9672959 (10Andrew) These are now set up and should start running a few backup jobs over the weekend. I need to check back and make su... [14:55:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [15:01:16] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update mailman project puppetmaster - https://phabricator.wikimedia.org/T361371 (10Andrew) 03NEW [15:12:20] 10wikitech.wikimedia.org, 10DiscussionTools, 10Editing-team (Kanban Board), 13Patch-For-Review: Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - https://phabricator.wikimedia.org/T361322#9673022 (10Esanders) > We could easily move away form this prefix, but that wouldn't hel... [15:45:16] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update mailman project puppetmaster - https://phabricator.wikimedia.org/T361371#9673106 (10Andrew) I've built a new puppetserver (mailman-puppetserver-1.mailman.eqiad1.wikimedia.cloud) and shut off the old one. Please delete when you're satisfied... [15:54:31] (ToolsToolsDBReplicationMissing) firing: ToolsDB replication is not running on tools-db-3 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [16:22:15] 10wikitech.wikimedia.org, 10DiscussionTools, 06Editing QA, 10Editing-team (Kanban Board), and 2 others: Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - https://phabricator.wikimedia.org/T361322#9673194 (10DLynch) [16:26:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-test-localdisk on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:31:29] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-cumin with Bullseye or Bookworm host - https://phabricator.wikimedia.org/T361380 (10Andrew) 03NEW [16:32:48] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-cumin with Bullseye or Bookworm host - https://phabricator.wikimedia.org/T361380#9673234 (10Andrew) a:03jhathaway @jhathaway, assigning to you because I think you built the latest production cu... [16:33:32] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673238 (10Andrew) [16:33:55] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673247 (10Andrew) [16:41:48] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381 (10Andrew) 03NEW [16:42:56] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#9673293 (10Andrew) a:03hnowlan @hnowlan, a glance at the puppet repo suggests that you're the pers... [16:43:37] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673296 (10Andrew) [16:45:44] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-prep kafka hosts with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361382 (10Andrew) 03NEW [16:45:52] 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth, 07TestMe, 07Wikimedia-production-error: OATHAuth's disableOATHAuthForUser.php script triggers a Notification that can't be sent as MW isn't initialised yet, so causes a production error - https://phabricator.wikimedia.org/T306184#9673316 (10Reedy) [16:45:57] (CloudVPSDesignateLeaks) firing: (5) Detected 17 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:46:35] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-prep kafka hosts with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361382#9673318 (10Andrew) a:03herron @herron, you seem to have had some recent kafka involvement. If you're not t... [16:47:03] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-prep kafka hosts with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361382#9673325 (10Andrew) [16:48:29] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673326 (10Andrew) [16:50:42] (CloudVPSDesignateLeaks) firing: (5) Detected 17 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:55:42] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:56:58] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383 (10Andrew) 03NEW [16:58:27] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#9673359 (10Andrew) a:03Eevans @Eevans you win this award since you created the instance... [16:58:44] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673362 (10Andrew) [16:59:03] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673367 (10thcipriani) We talked about this a bit in the #together team meeting on Wednesday—we discussed whether we had the a... [16:59:31] (ToolsToolsDBReplicationMissing) resolved: ToolsDB replication is not running on tools-db-3 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [17:00:42] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:04:01] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-memc[08-10] with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361384 (10Andrew) 03NEW [17:07:09] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673398 (10Andrew) [17:07:10] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-memc[08-10] with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361384#9673393 (10Andrew) a:03elukey @elukey, it looks to me like you are the most recent rebuilder of mc hosts. Feel... [17:10:07] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 06serviceops: Replace deployment-memc[08-10] with Bullseye or Bookworm - https://phabricator.wikimedia.org/T361384#9673411 (10elukey) a:05elukey→03jijiki [17:25:42] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:25:46] (03PS1) 10Jforrester: releases: Upgrade eslint-config-wikimedia to 0.27.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015566 [17:26:03] (03CR) 10Jforrester: [C:03+2] releases: Upgrade eslint-config-wikimedia to 0.27.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015566 (owner: 10Jforrester) [17:26:38] (03Merged) 10jenkins-bot: releases: Upgrade eslint-config-wikimedia to 0.27.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1015566 (owner: 10Jforrester) [17:29:04] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#9673474 (10thcipriani) >>! In T361383#9673359, @Andrew wrote: > @Eevans you win this award... [18:02:45] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-ores02 - https://phabricator.wikimedia.org/T361385 (10Andrew) 03NEW [18:04:01] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673549 (10Andrew) [18:05:57] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361386 (10Andrew) 03NEW [18:07:26] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Remove or replace deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361386#9673578 (10Andrew) a:03jijiki @jijiki, you seem to have touched some parsoid puppet code r... [18:08:26] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673581 (10Andrew) [18:08:53] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387 (10Andrew) 03NEW [18:10:15] 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9673596 (10Andrew) [18:10:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:13:30] 06cloud-services-team, 10Cloud-VPS, 05Goal, 10Puppet (Puppet 7.0): Update maps-experiments project puppetmaster - https://phabricator.wikimedia.org/T361388 (10Andrew) 03NEW [18:15:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:29:31] (ToolsToolsDBReplicationMissing) firing: ToolsDB replication is not running on tools-db-3 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [18:36:55] 06cloud-services-team, 10VPS-Projects, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#9673688 (10Andrew) [18:36:59] 06cloud-services-team, 10Cloud-VPS, 05Goal, 10Puppet (Puppet 7.0): 14Update maps-experiments project puppetmaster - 14https://phabricator.wikimedia.org/T361388#9673680 (10Andrew) 05Open→03Resolved 14I've moved clients from maps-puppetmaster02 to maps-experiments-puppetserver-1 and shut down maps-p... [18:55:15] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:24:54] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 05Goal, 13Patch-For-Review: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9673773 (10fnegri) After a few attempts, the procedure at https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Toolsdb#Cre... [19:42:45] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#9673804 (10Eevans) >>! In T361383#9673474, @thcipriani wrote: >>>! In T361383#9673359, @An... [20:44:57] 10Toolforge: [jobs-cli,jobs-api] Provide a means to configure a task to be restarted indefinately upon error, but terminate normally otherwise - https://phabricator.wikimedia.org/T361405 (10bd808) 03NEW [20:45:56] 10Toolforge: [jobs-cli,jobs-api] Provide a means to configure a task to be restarted indefinately upon error, but terminate normally otherwise - https://phabricator.wikimedia.org/T361405#9673974 (10bd808) [21:21:28] 10Toolforge: [buildservice] Determine the least invasive/smallest extra output buildpack needed to pair with Apt - https://phabricator.wikimedia.org/T361409 (10bd808) 03NEW [21:25:23] 10Toolforge: [buildservice] Determine the least invasive/smallest extra output buildpack needed to pair with Apt - https://phabricator.wikimedia.org/T361409#9674057 (10bd808) Adding @Anomie as a subscriber because my conversations with him about AnomieBOT led to this request as noted in the description. [21:37:24] 10Toolforge: [webservice] Apply a `app.kubernetes.io/name` label to `webservice` created pods - https://phabricator.wikimedia.org/T361410 (10bd808) 03NEW [21:50:41] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:55:41] (CloudVPSDesignateLeaks) resolved: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:55:15] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:56:12] 06cloud-services-team, 10VPS-Projects, 10Puppet (Puppet 7.0): Update mailman project puppetmaster - https://phabricator.wikimedia.org/T361371#9674196 (10Ladsgroup) We don't have any more unmerged puppet patches AFAIK. For the newer upgrade, we should have some. [23:40:41] (CloudVPSDesignateLeaks) firing: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:45:41] (CloudVPSDesignateLeaks) firing: (5) Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks