[00:02:35] <wikibugs>	 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10342527 (10LibUp-bot)
[01:45:50] <wikibugs>	 (03merge) 10ivi104: Fix error in [A-z] range [toolforge-repos/hrwp] - 10https://gitlab.wikimedia.org/toolforge-repos/hrwp/-/merge_requests/2 (owner: 10hartman)
[01:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:00:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[03:58:44] <wikibugs>	 10PAWS: [feature] <your request here> - https://phabricator.wikimedia.org/T380442 (10Vinzs) 03NEW
[04:04:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:43:58] <wikibugs>	 10tool-wscontest, 07good first task: Make all interface messages translatable - https://phabricator.wikimedia.org/T346994#10342730 (10AS1100K) a:03AS1100K
[07:43:18] <wikibugs>	 (03approved) 10raymond-ndibe: [maintain-harbor] comment about the importance of some logs in tests [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/38 (https://phabricator.wikimedia.org/T358225)
[07:43:22] <wikibugs>	 (03merge) 10raymond-ndibe: [maintain-harbor] comment about the importance of some logs in tests [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/38 (https://phabricator.wikimedia.org/T358225)
[08:03:03] <wikibugs>	 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, 13Patch-For-Review: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10342863 (10SLyngshede-WMF) > Is there a particular reason that this data validation API requires authentication?   Probably...
[09:43:45] <wikibugs>	 10tool-wscontest, 07good first task: Make all interface messages translatable - https://phabricator.wikimedia.org/T346994#10343044 (10AS1100K) I have opened [[ https://github.com/wikisource/wscontest/pull/77 | PR #77 ]] on GitHub.  I have a question, for the text `Index Pages:`, is it okay to do `{{ msg('index...
[09:48:45] <wikibugs>	 (03approved) 10sstefanova: toolforge_deploy_mr: add clis to the bash completion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/215 (owner: 10dcaro)
[09:49:50] <wikibugs>	 (03merge) 10dcaro: toolforge_deploy_mr: add clis to the bash completion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/215
[09:55:01] <wikibugs>	 (03update) 10raymond-ndibe: toolforge_deploy_mr: add clis to the bash completion [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/215 (owner: 10dcaro)
[10:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:00:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:09:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: codfw1dev: fullstack tests failing - https://phabricator.wikimedia.org/T380208#10343267 (10aborrero) 05In progress→03Resolved nova-fullstack works normally now.
[11:56:15] <wikibugs>	 (03update) 10aborrero: eqiad1: add IPv6 support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132 (https://phabricator.wikimedia.org/T380174)
[11:57:27] <wikibugs>	 (03update) 10aborrero: eqiad1: add IPv6 support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132 (https://phabricator.wikimedia.org/T380174)
[12:21:33] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/24
[12:44:50] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: netbox: allocate CIDRs for openstack VXLAN-based flat networks - https://phabricator.wikimedia.org/T374111#10343747 (10aborrero) updates: * cloud-flat-1-ipv4only (eqiad1) 172.16.8.0/21 https://netbox.wikimedia.org/ipam/prefixes/1076/ * cloud-flat-1-du...
[12:45:19] <wikibugs>	 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10343748 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/462
[12:45:27] <notefromgithub>	 vivian-rook opened https://github.com/toolforge/paws/pull/462
[12:53:09] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10343763 (10aborrero) TODO next: * activate IPv6 support within openstack (merge https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132) * enable...
[13:36:29] <wikibugs>	 (03update) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225)
[13:36:46] <wikibugs>	 (03update) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225)
[13:38:57] <wikibugs>	 (03update) 10raymond-ndibe: [builds-builder] create toolforge harbor project [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/63 (https://phabricator.wikimedia.org/T358225)
[14:21:12] <wikibugs>	 10Tools, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Listeria no longer translates property numbers to labels in the local language for Wikidatalist headings - https://phabricator.wikimedia.org/T247047#10344036 (10WMDECyn) a:03seanleong-WMDE
[14:22:54] <wikibugs>	 10Tools, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Listeria no longer translates property numbers to labels in the local language for Wikidatalist headings - https://phabricator.wikimedia.org/T247047#10344035 (10WMDECyn)
[14:27:26] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare  documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10344062 (10aborrero) expanded https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_VXLAN_IPv6_migration#Common_questions_and_problems
[14:31:10] <wikibugs>	 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T380474 (10rook) 03NEW
[14:31:45] <wikibugs>	 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T380474#10344093 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/463
[14:31:56] <notefromgithub>	 vivian-rook opened https://github.com/toolforge/paws/pull/463
[14:47:46] <wikibugs>	 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T380474#10344133 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/463
[14:47:47] <wikibugs>	 10PAWS: upgrade jupyterlab - https://phabricator.wikimedia.org/T380474#10344134 (10rook) 05Open→03Resolved
[14:48:03] <notefromgithub>	 vivian-rook closed https://github.com/toolforge/paws/pull/463
[14:49:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10WikiWho: Enable use of web proxy for wikiwho.net domain - https://phabricator.wikimedia.org/T376637#10344138 (10Kkzo) @MusikAnimal invited me to engage directly. I represent what you called above the "owners".  As mentioned, I need to wait for replies from the IT departm...
[15:06:30] <wikibugs>	 10Striker: [toolsadmin] Striker cannot create Developer accounts with names matching existing SUL accounts - https://phabricator.wikimedia.org/T380384#10344199 (10Andrew) Thank you for the complete explanation @bd808!  @RoyZuo can you give the suggested workaround a try for now? It looks like it's going to be a...
[15:11:44] <wikibugs>	 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, 13Patch-For-Review: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10344221 (10SLyngshede-WMF) Authentication has been disabled on idm-test.wikimedia.org. If everything works we can deploy th...
[15:11:58] <wikibugs>	 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, 13Patch-For-Review: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10344223 (10SLyngshede-WMF) 05Open→03In progress
[15:16:37] <jinxer-wm>	 FIRING: PowerSupplyFailure: Power Supply - Status - issue on cloudbackup2003:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Power_Supply_Failures - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&var-Sensor=Power%20Supply&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DPowerSupplyFailure
[15:16:53] <wikibugs>	 06cloud-services-team: PowerSupplyFailure Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T380479 (10phaultfinder) 03NEW
[15:28:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484 (10fnegri) 03NEW
[15:32:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10344421 (10Andrew) > I am not coming up with reasons to keep mirroring the Keystone project membership data int...
[15:33:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "eqiad.wmflabs." DNS zone to cloudinfra - https://phabricator.wikimedia.org/T380488 (10fnegri) 03NEW
[15:35:04] <icinga-wm>	 PROBLEM - Host checker.tools.wmflabs.org is DOWN: CRITICAL - Host Unreachable (checker.tools.wmflabs.org)
[15:35:09] <jinxer-wm>	 FIRING: CephSlowOps: Ceph cluster in eqiad has 253 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps
[15:36:19] <icinga-wm>	 RECOVERY - Host checker.tools.wmflabs.org is UP: PING OK - Packet loss = 0%, RTA = 1.92 ms
[15:36:21] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489 (10phaultfinder) 03NEW
[15:36:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "svc.eqiad.wmflabs." to cloudinfra project - https://phabricator.wikimedia.org/T380490 (10fnegri) 03NEW
[15:36:38] <wmcs-alerts>	 FIRING: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:36:50] <wmcs-alerts>	 FIRING: [6x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:37:11] <wmcs-alerts>	 FIRING: ProjectProxyMainProxyDown: Proxy on proxy-03 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown
[15:38:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate db.svc.eqiad.wmflabs. DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491 (10fnegri) 03NEW
[15:39:09] <jinxer-wm>	 FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[15:41:38] <wmcs-alerts>	 RESOLVED: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:41:50] <wmcs-alerts>	 RESOLVED: [6x] ProbeDown: Service tools-k8s-haproxy-5:30000 has failed probes (http_admin_toolforge_org_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[15:42:11] <wmcs-alerts>	 RESOLVED: ProjectProxyMainProxyDown: Proxy on proxy-03 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown
[15:45:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate web.db.svc.eqiad.wmflabs. and analytics.db.svc.eqiad.wmflabs. to cloudinfra project - https://phabricator.wikimedia.org/T380493 (10fnegri) 03NEW
[15:48:30] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344542 (10dcaro) Having slow heartbeats: `             Slow OSD heartbeats on back (longest 6757.204ms)             Slow OSD heartbeats on front (longest 6091.493ms)  `  ` root@cloudcephmon1001...
[15:48:38] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344545 (10dcaro) a:03dcaro
[15:49:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: opentofu shouldn't delete openstack resources - https://phabricator.wikimedia.org/T380310#10344543 (10Andrew) 05Open→03Invalid Thanks to @fnegri's patient explanation (and an actual meeting we held about this yesterday) I'm now convinced that the specific delete-by-s...
[15:49:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "16.172.in-addr.arpa." DNS zone to cloudinfra - https://phabricator.wikimedia.org/T380495 (10fnegri) 03NEW
[15:50:26] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344560 (10dcaro) From the switch stats: {F57729487} https://grafana.wikimedia.org/d/f61a7d56-e132-44dc-b9da-d722b11566cf/network-totals-by-site?orgId=1&refresh=30s&var-site=eqiad%20prometheus%2...
[15:50:52] <wikibugs>	 (03PS2) 10BryanDavis: dev: Allow insecure apt actions in Keystone and MediaWiki Docker [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474
[15:50:52] <wikibugs>	 (03PS1) 10BryanDavis: dev: Add .mailmap for git author remapping [labs/striker] - 10https://gerrit.wikimedia.org/r/1093937
[15:51:13] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380488#10344566 (10fnegri)
[15:51:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380490#10344567 (10fnegri)
[15:51:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10344570 (10fnegri)
[15:51:39] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "16.172.in-addr.arpa." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380495#10344571 (10fnegri)
[15:51:44] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344565 (10dcaro) {F57729491} https://grafana.wikimedia.org/d/5p97dAASz/network-interface-queue-and-error-stats?orgId=1&var-site=eqiad+prometheus%2Fops&var-device=cloudsw1-d5-eqiad&var-interface...
[15:53:07] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10344574 (10fnegri) p:05Triage→03High a:03fnegri Changing this specific subtask to "High" and assigning to myself as I want to do it as part o...
[15:53:22] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10344581 (10fnegri)
[15:53:25] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10344582 (10fnegri)
[15:54:00] <wikibugs>	 (03PS2) 10BryanDavis: dev: Add .mailmap for git author remapping [labs/striker] - 10https://gerrit.wikimedia.org/r/1093937
[15:54:11] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484#10344583 (10fnegri) p:05Triage→03Medium
[15:54:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380488#10344584 (10fnegri) p:05Triage→03Medium
[15:54:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380490#10344585 (10fnegri) p:05Triage→03Medium
[15:54:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate web.db.svc.eqiad.wmflabs. and analytics.db.svc.eqiad.wmflabs. to cloudinfra project - https://phabricator.wikimedia.org/T380493#10344587 (10fnegri) p:05Triage→03Medium
[15:54:26] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10344586 (10Andrew) The other database on that host is codfw1dev striker. Quoth bd808:  > I'm not sure the Striker in codfw1dev ever worked, but maybe t.aa...
[15:54:29] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10344588 (10aborrero)
[15:54:31] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Migrate "16.172.in-addr.arpa." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380495#10344592 (10fnegri) p:05Triage→03Medium
[15:55:06] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 1 slow ops - https://phabricator.wikimedia.org/T380239#10344596 (10dcaro) From the switch graphs, something made it blip: {F57729497}  https://grafana.wikimedia.org/d/5p97dAASz/network-interface-queue-and-error-stats?orgId=1&var-site=eqiad%20prometheus...
[15:55:07] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344590 (10dcaro) Might be related to T380239
[15:55:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [designate] migrate DNS records in noauth-project to cloudinfra project - https://phabricator.wikimedia.org/T380484#10344597 (10aborrero)
[15:58:11] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10344599 (10fnegri) @aborrero I will try to import the records to tofu-infra in the process!
[16:00:43] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] test failover procedure - https://phabricator.wikimedia.org/T344719#10344605 (10fnegri) I have added many more details to the procedure in Wikitech, and I will test it next Monday as part of  {T352206}.
[16:07:51] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:12:50] <wmcs-alerts>	 RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:16:40] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install cloudcontrol1011 - https://phabricator.wikimedia.org/T380499 (10RobH) 03NEW
[16:17:30] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install cloudcontrol1011 - https://phabricator.wikimedia.org/T380499#10344684 (10RobH)
[16:18:58] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install cloudcontrol1011 - https://phabricator.wikimedia.org/T380499#10344687 (10RobH) a:03Andrew Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving...
[16:21:19] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 1 slow ops - https://phabricator.wikimedia.org/T380239#10344736 (10dcaro) 05Open→03In progress p:05Triage→03High
[16:21:29] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance ntp-03 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:21:41] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10344729 (10dcaro) 05Open→03In progress p:05Triage→03High
[16:22:29] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-74 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:22:45] <wikibugs>	 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T380436#10344756 (10rook) Update not quite updating? https://github.com/OpenRefine/OpenRefine/issues/7001
[16:23:17] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:32:35] <wikibugs>	 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503 (10cmooney) 03NEW p:05Triage→03High The #Cloud-Services project tag is not intended to have any...
[16:35:55] <wikibugs>	 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10344824 (10JJMC89)
[16:42:02] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 1 slow ops - https://phabricator.wikimedia.org/T380239#10344917 (10dcaro)
[16:42:05] <wikibugs>	 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, 10ops-eqiad: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10344918 (10dcaro)
[16:46:59] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-74 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:47:59] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance ntp-03 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[16:48:17] <wmcs-alerts>	 RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[17:03:35] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345086 (10Reedy) Some cross ref to {T376267}, thought it's a newer account that was only created after wikitech was moved...
[17:04:30] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345107 (10Reedy) https://ldap.toolforge.org/user/jly  https://meta.wikimedia.org/wiki/Special:CentralAuth?target=JLy-WMF
[17:07:08] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345126 (10Reedy) ` reedy@deploy2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=labswiki "JLy-WMF" DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See...
[17:07:35] <wikibugs>	 (03open) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953)
[17:08:18] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225)
[17:10:23] <wikibugs>	 (03update) 10fnegri: Draft: Import DNS records for db.svc.wikimedia.cloud [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/135 (https://phabricator.wikimedia.org/T374953)
[17:11:53] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225)
[17:12:36] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] allow for running both admin and tools tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/605 (https://phabricator.wikimedia.org/T358225)
[17:14:16] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511 (10dcaro) 03NEW p:05Triage→03High
[17:14:37] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345235 (10dcaro) The original exception is: ` [Thu Nov 21 17:09:40.191562 2024] [wsgi:error]...
[17:15:12] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345250 (10Jly) @Reedy Thanks! Access to wikitech and wikimedia is working now. Please ignore the dev account tied to wrong SUL, this was before and I managed to do a refresh to connect to JLy-WMF.
[17:21:52] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380513 (10Keith_D) 03NEW
[17:21:54] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345317 (10Reedy) Can you try and load up some other wikis (like the German Wikipedia, or the French Wikivoyage etc; any of the 900+ on https://meta.wikimedia.org/wiki/Special:SiteMatrix :)), and see whet...
[17:21:58] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345318 (10Andrew) The Striker DB is empty but for the schema. So, abandoning.
[17:22:04] <wikibugs>	 (03CR) 10BryanDavis: [C:03+2] dev: Add .mailmap for git author remapping [labs/striker] - 10https://gerrit.wikimedia.org/r/1093937 (owner: 10BryanDavis)
[17:22:08] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10345322 (10fnegri)
[17:22:12] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4: (Need By: TBD) rack/setup/install cloudcontrol2005-dev, clouddb2002-dev, cloudgw2003-dev - https://phabricator.wikimedia.org/T306854#10345324 (10Andrew)
[17:22:18] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: tofu-infra: replace wmcs-wikireplica-dns.py with tofu - https://phabricator.wikimedia.org/T374953#10345323 (10fnegri)
[17:22:22] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345325 (10Andrew)
[17:23:11] <wikibugs>	 (03Merged) 10jenkins-bot: dev: Add .mailmap for git author remapping [labs/striker] - 10https://gerrit.wikimedia.org/r/1093937 (owner: 10BryanDavis)
[17:24:09] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380513#10345338 (10JJMC89) →14Duplicate dup:03T380426
[17:24:37] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10345349 (10fnegri) 05Open→03In progress
[17:24:56] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345352 (10Andrew)
[17:25:02] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate "db.svc.eqiad.wmflabs." DNS zone to cloudinfra project - https://phabricator.wikimedia.org/T380491#10345320 (10fnegri) We should also make sure that these records are no longer managed by the `wmcs-wikireplica-dns` script: {T374953}
[17:25:05] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: tofu-infra: replace wmcs-wikireplica-dns.py with tofu - https://phabricator.wikimedia.org/T374953#10345343 (10fnegri) 05Open→03In progress
[17:25:08] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380426#10345340 (10JJMC89)
[17:25:15] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345359 (10dcaro) From our code for the special view: ` /opt/lib/python/site-packages/wikimedi...
[17:26:48] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345360 (10Jly) I've tried a bunch (https://meta.wikimedia.org/wiki/Special:CentralAuth?target=JLy-WMF), it signs me in automatically or I manually signed in, no issues :)
[17:26:56] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345362 (10dcaro) This should have fixed it actually: https://gerrit.wikimedia.org/r/c/opensta...
[17:31:03] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345397 (10dcaro) Yep, we are not running latest:  docker-registry.wikimedia.org/repos/cloud/c...
[17:31:47] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225)
[17:32:17] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345400 (10Reedy) 05Open→03Resolved a:03Reedy Sweet, thanks.  I'm guessing it's just a weird edge case regarding the migration of the auth systems of wikitech, at least. Not sure why the mediawi...
[17:32:37] <wikibugs>	 10wikitech.wikimedia.org: Wikitech and Mediawiki sign in errors - https://phabricator.wikimedia.org/T380506#10345410 (10JLam-WMF) Thanks, all.
[17:32:53] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence, 13Patch-For-Review: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345418 (10Andrew)
[17:33:28] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345432 (10dcaro) :facepalm: I'm adding `docker pull` there xd  It does work and has the right...
[17:33:50] <wikibugs>	 (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225)
[17:33:55] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380426#10345433 (10Curb_Safe_Charmer) a:03Curb_Safe_Charmer
[17:36:57] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345456 (10dcaro) it's managed by puppet: https://gerrit.wikimedia.org/g/operations/puppet/+/7...
[17:39:49] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T380426#10345499 (10Curb_Safe_Charmer) This time, seems to have come back to life after simply running  ` webservice restart ` on refill-api.
[17:41:00] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence, 13Patch-For-Review: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345510 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `clouddb2002-dev.codfw.wmnet...
[17:43:01] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence, 06DC-Ops, and 2 others: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345520 (10Andrew)
[17:46:09] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence, 06DC-Ops, and 2 others: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10345516 (10Andrew) a:05Andrew→03None This host is shut down and removed from puppet. It's only a couple of years old so shou...
[17:46:37] <jinxer-wm>	 RESOLVED: PowerSupplyFailure: Power Supply - Status - issue on cloudbackup2003:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook#Power_Supply_Failures - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&var-Sensor=Power%20Supply&var-server=cloudbackup2003 - https://alerts.wikimedia.org/?q=alertname%3DPowerSupplyFailure
[17:46:50] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345574 (10dcaro) a:05dcaro→03Andrew
[17:47:56] <wikibugs>	 06cloud-services-team: CephSlowOps Ceph cluster in eqiad has 253 slow ops - https://phabricator.wikimedia.org/T380489#10345577 (10dcaro)
[17:48:02] <wikibugs>	 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10345576 (10dcaro)
[18:34:26] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345862 (10Andrew) 05Open→03Resolved After deploying the latest Horizon build things s...
[18:43:03] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [horizon] failing to show the proxies tab for some projects - https://phabricator.wikimedia.org/T380511#10345902 (10dancy) Thank you!
[19:06:25] <wikibugs>	 10Cloud-Services: Interfaces tab of deployment-deploy04.deployment-prep VM never finishes loading - https://phabricator.wikimedia.org/T380531 (10dancy) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and re...
[19:16:15] <wikibugs>	 10Horizon, 05Cloud-Services-Origin-User: Interfaces tab of deployment-deploy04.deployment-prep VM never finishes loading - https://phabricator.wikimedia.org/T380531#10346082 (10JJMC89)
[19:19:46] <wikibugs>	 10Horizon, 05Cloud-Services-Origin-User: Interfaces tab of deployment-deploy04.deployment-prep VM never finishes loading - https://phabricator.wikimedia.org/T380531#10346091 (10dancy) p:05Triage→03Low
[19:20:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:35:25] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] "Indeed. I've confirmed:" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1079381 (https://phabricator.wikimedia.org/T373708) (owner: 10Krinkle)
[19:35:53] <wikibugs>	 (03PS1) 10Andrew Bogott: Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030)
[19:36:38] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott)
[19:37:17] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535 (10Krinkle) 03NEW
[19:37:39] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] "I filed T380535 for the Gerrit API issue." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1079381 (https://phabricator.wikimedia.org/T373708) (owner: 10Krinkle)
[19:38:21] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10346167 (10Krinkle) Note that on https://gerrit.wikimedia.org/r/admin/repos/q/filter:design/, Gerrit does list `design/codex-php`.
[19:38:29] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10346168 (10Krinkle)
[19:52:50] <wikibugs>	 (03PS2) 10Andrew Bogott: Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030)
[19:53:23] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott)
[20:00:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:22:15] <wikibugs>	 10Tools: bullseye's Spur API key has expired - https://phabricator.wikimedia.org/T380193#10346270 (10TheresNoTime) >>! In T380193#10342014, @RoySmith wrote: > I love Bullseye, but I just don't see the point of continuing to maintain it when there's a WMF-supported service that basically does the same thing.  If...
[20:31:09] <wikibugs>	 (03PS3) 10Andrew Bogott: Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030)
[20:31:41] <wikibugs>	 06Toolforge-standards-committee: Adoption request for bullseye - https://phabricator.wikimedia.org/T380537 (10TheresNoTime) 03NEW
[20:31:43] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030) (owner: 10Andrew Bogott)
[20:40:29] <wikibugs>	 (03PS4) 10Andrew Bogott: Get openstack project list from keystone [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/1093997 (https://phabricator.wikimedia.org/T379030)
[20:50:58] <wikibugs>	 06Toolforge-standards-committee: Adoption request for bullseye - https://phabricator.wikimedia.org/T380537#10346360 (10LucasWerkmeister) > I request that the secrets not be removed in this case — the only sensitive secret is the Spur.us API key which is already expired per T380193  I suppose it’s a somewhat moot...
[21:49:17] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: OpenStack project ids rather than names are being used in hiera settings archiving - https://phabricator.wikimedia.org/T379128#10346556 (10Andrew) 05Open→03Resolved a:03Andrew Changes should now use names rather than IDs. I did my best to retc...
[22:16:06] <wmcs-alerts>	 FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:21:06] <wmcs-alerts>	 FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:26:06] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:36:10] <wikibugs>	 06Toolforge-standards-committee: Adoption request for bullseye - https://phabricator.wikimedia.org/T380537#10346787 (10TheresNoTime) >>! In T380537#10346360, @LucasWerkmeister wrote: >  > - a `SECRET_KEY` – if it’s anything like Flask / ItsDangerous (and [it sounds like it is](https://docs.djangoproject.com/en/5...
[22:42:41] <wikibugs>	 (03update) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/24 (owner: 10l10n-bot)
[22:43:43] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/24 (owner: 10l10n-bot)
[22:43:46] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/24 (owner: 10l10n-bot)
[22:51:22] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10346834 (10bd808) Trying some variations: * https://gerrit.wikimedia.org/r/projects/?p=design/c `lan...
[23:02:12] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10346862 (10thcipriani) A strange datapoint here:  https://gerrit.wikimedia.org/g/ shows `design/code...
[23:04:08] <wikibugs>	 10VPS-project-Codesearch, 10Gerrit, 06Release-Engineering-Team: Gerrit Projects API listing does not include design/codex-php (Codesearch does not index codex-php) - https://phabricator.wikimedia.org/T380535#10346866 (10thcipriani) >>! In T380535#10346862, @thcipriani wrote: > A strange datapoint here: >  >...
[23:38:58] <wikibugs>	 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10346945 (10Dwisehaupt) Ok. I think I'm going to need some help at this point. I have gone back to the puppetserver (puppetserver-01.civicrm-prototype) I cre...