[00:59:53] (03update) 10raymond-ndibe: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) [02:17:59] (03PS1) 10Krinkle: Add support for mediawiki.org [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1152873 [02:18:06] (03CR) 10Krinkle: [C:03+2] Add support for mediawiki.org [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1152873 (owner: 10Krinkle) [02:19:04] (03Merged) 10jenkins-bot: Add support for mediawiki.org [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/1152873 (owner: 10Krinkle) [02:30:01] 06cloud-services-team, 10Cloud-VPS: Octavia infra followup - https://phabricator.wikimedia.org/T395864 (10Andrew) 03NEW [06:39:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s8 backend clouddb1016.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:44:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s8 backend clouddb1016.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:23:00] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878495 (10fnegri) Reminder that I'm gonna merge and apply this later today, dropping the old `ipblocks*` views. [08:46:04] 10Cloud-Services, 06Data-Engineering, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10878585 (10Gehel) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabric... [09:06:35] 06cloud-services-team, 06Data-Engineering, 06Data-Persistence, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10878660 (10fnegri) @Gehel I don't think there's anything actionable for #cloud-services-team yet.... [09:09:37] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10878667 (10taavi) 05Open→03Stalled (replicating `x3` was already recently a... [09:21:13] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881 (10fnegri) 03NEW [09:21:32] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering, 07SecTeam-Processed: Add "wikishared" database to wiki replicas - https://phabricator.wikimedia.org/T395072#10878728 (10fnegri) > I think there is already a ticket for adding x1 to wikireplicas. I couldn't find one, so I... [09:21:57] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10878731 (10fnegri) [09:22:07] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10878733 (10fnegri) [09:22:10] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10878732 (10fnegri) [09:22:47] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10878735 (10fnegri) > but likely should have its own task? Agree, I created {T3... [09:42:46] (03PS7) 10Majavah: Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [09:47:25] (03CR) 10Majavah: [C:04-1] Switch username validation to Bitu API (037 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [09:50:54] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10878825 (10Marostegui) This definitely needs a review from security before we can proceed. This will also likely need specific triggers/view... [09:51:49] (03update) 10fnegri: [api] Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) (owner: 10chuckonwumelu) [09:55:22] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10878832 (10fnegri) > we need to get an indication on Who is best placed to provide the answers here? I'm looping in #data-engineering as well. [09:58:33] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#10878844 (10Ladsgroup) >>! In T395881#10878832, @fnegri wrote: >> we need to get an indication on > > Who is best placed to provide the answ... [10:08:53] 10Toolforge (Toolforge iteration 20), 07good first task: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#10878879 (10fnegri) [10:18:26] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878892 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by fnegri: Started updating wiki repl... [10:43:50] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878966 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by fnegri executed with errors: - an-redacteddb1001.eqi... [10:44:44] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878969 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by fnegri: Started updating wiki replica views [10:48:15] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878975 (10fnegri) The cookbook crashed on clouddb1018 while creating the view `cswiki_p.logging`: ` 2025-06-03 10:41:28,289 INFO [cswiki_p.lo... [10:52:42] 06cloud-services-team, 10Data-Services: maintain-views: skip new databases that have not been sanitized yet - https://phabricator.wikimedia.org/T375779#10878976 (10fnegri) 05Open→03Resolved a:03fnegri Marking this as resolved, as {T392105} should prevent this situation. [10:55:45] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10878984 (10Marostegui) @taavi for now I've set up clouddb1016:3363 - I'd need someome from #cloud-services-team to deploy views, add it to the LB etc,... [10:56:29] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10878993 (10taavi) Thank you! I will have a look at that. [11:00:24] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10878999 (10fnegri) > The cookbook crashed on clouddb1018 while creating the view cswiki_p.logging: The same view failed to create in T395122#1... [11:09:35] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10879053 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by fnegri executed with errors: - an-redacteddb1001.eqi... [11:28:22] FIRING: HAProxyServiceUnavailable: HAProxy service wikireplica-db-analytics-x3 has no available backends on cloudlb1002:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyServiceUnavailable [11:28:32] 06cloud-services-team: HAProxyServiceUnavailable HAProxy service wikireplica-db-analytics-x3 has no available backends on cloudlb1002:9900 - https://phabricator.wikimedia.org/T395895 (10phaultfinder) 03NEW [11:35:47] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879207 (10taavi) maintain-views needed adjusting as extension sections don't have dblists with their list of databases, patches for that are above. Wi... [11:36:50] 06cloud-services-team: HAProxyServiceUnavailable HAProxy service wikireplica-db-analytics-x3 has no available backends on cloudlb1002:9900 - https://phabricator.wikimedia.org/T395895#10879213 (10taavi) →14Duplicate dup:03T390954 [11:36:57] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879215 (10taavi) [11:39:06] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879218 (10Marostegui) >>! In T390954#10879207, @taavi wrote: > > I added the load balancer backend definition to check that is working as expected,... [11:45:00] (03PS9) 10Arendpieter: Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) [11:46:48] (03CR) 10Arendpieter: Switch username validation to Bitu API (034 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [12:22:49] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446#10879326 (10taavi) 05Open→03Resolved Done for all deployment-prep alerts. Customizing which alerts are sent... [12:30:55] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10879344 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye [12:31:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10879348 (10Jclark-ctr) [12:32:55] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879349 (10taavi) >>! In T390954#10879218, @Marostegui wrote: > Do you want me to setup the other server then? Yes please! [13:01:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10879389 (10fnegri) This time it failed on a different table: ` 2025-06-03 11:07:08,328 INFO [thwiki_p.templatelinks] 2025-06-03 11:08:08,329 W... [13:06:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-x3 backend clouddb1016.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [13:08:22] FIRING: [2x] HAProxyServiceUnavailable: HAProxy service wikireplica-db-web-x3 has no available backends on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyServiceUnavailable [13:08:28] 06cloud-services-team: HAProxyServiceUnavailable - https://phabricator.wikimedia.org/T395900 (10phaultfinder) 03NEW [13:38:22] PROBLEM - mysqld processes on clouddb1020 is CRITICAL: PROCS CRITICAL: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:39:55] 06cloud-services-team: HAProxyServiceUnavailable - https://phabricator.wikimedia.org/T395900#10879489 (10taavi) 05Open→03Resolved [13:40:57] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10879490 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudcephosd1048.eqiad.wmnet with OS bullseye com... [13:41:19] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10879491 (10Jclark-ctr) [13:41:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10879492 (10Jclark-ctr) 05Open→03Resolved [13:44:19] 06cloud-services-team, 10Toolforge: Toolforge `sql` utility should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395901#10879508 (10taavi) [13:44:22] 06cloud-services-team, 10Toolforge: Toolforge `sql` utility should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395901#10879510 (10taavi) [13:44:32] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879511 (10taavi) [13:45:22] RECOVERY - mysqld processes on clouddb1020 is OK: PROCS OK: 3 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [13:46:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-x3 backend clouddb1016.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [13:47:13] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879515 (10Marostegui) @BTullis I assume an-redacttedb host also needs x3? (a set of tables split from s8) [13:52:54] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879549 (10BTullis) @Marostegui - Thanks. I'll double-check with the #data-engineering team, but my current understanding is that we **do not** need a... [13:54:55] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10879551 (10Marostegui) >>! In T390954#10879549, @BTullis wrote: > @Marostegui - Thanks. I'll double-check with the #data-engineering team, but my curre... [14:07:27] (03update) 10chuckonwumelu: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) [14:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:10:29] (03update) 10chuckonwumelu: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) [14:16:04] 10Toolforge (Toolforge iteration 20), 07good first task: [components-api] add `GET` endpoint `/v1/tool//deployments/latest` - https://phabricator.wikimedia.org/T394990#10879625 (10Chuckonwumelu) a:03Chuckonwumelu [14:44:04] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10879800 (10fnegri) I managed to run the script manually on clouddb1018, but I had to kill a few threads as the script got stuck with `Waiting f... [14:45:41] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10879801 (10fnegri) [14:57:13] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10879848 (10fnegri) clouddb1019 is trickier because there are some old queries that are still there even after a `kill`: ` | 33282161 | s52788... [15:06:17] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 (10Andrew) 03NEW [15:06:24] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910#10879896 (10Andrew) a:05Jclark-ctr→03Andrew [15:13:42] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-cli,components-api] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277#10879926 (10Chuckonwumelu) [15:15:45] (03update) 10chuckonwumelu: [build] Return image_name when retrieving build info [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T395035) [15:59:16] (03PS2) 10Acamicamacaraca: Change Serbocroatian to Serbo-Croatian [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1153275 (https://phabricator.wikimedia.org/T395915) [16:18:46] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10880601 (10Jhancock.wm) @Andrew hey I'm running into an issue in this rack with port availability. because of the servers using 2x1G ports. how much a pa... [16:20:57] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1153285 (owner: 10L10n-bot) [16:25:18] (03CR) 10Nikerabbit: [C:03+2] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1153285 (owner: 10L10n-bot) [16:45:46] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10880712 (10fnegri) Those queries are badly stuck, with execution time exceeding 8 days. wmf-pt-kill also failed to kill them, and they are stuc... [16:52:53] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1153296 (owner: 10L10n-bot) [16:53:04] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/qrcode-generator] - 10https://gerrit.wikimedia.org/r/1153298 (owner: 10L10n-bot) [16:53:15] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1153299 (owner: 10L10n-bot) [16:53:39] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/1153297 (owner: 10L10n-bot) [16:53:51] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1153295 (owner: 10L10n-bot) [17:02:30] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10880782 (10fnegri) `STOP SLAVE` worked fine. `systemctl stop mariadb@s4` is stuck. This was logged in journald: ` Jun 03 16:57:49 clouddb1019... [17:23:51] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10880925 (10fnegri) [17:25:00] 06cloud-services-team, 10Toolforge: Toolforge `sql` utility should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395901#10880929 (10taavi) a:03taavi [17:25:21] (03PS1) 10Majavah: sql: Remove unnecessary parens [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153314 [17:25:21] (03PS1) 10Majavah: sql: Support connecting to extension databases [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) [17:25:48] (03CR) 10CI reject: [V:04-1] sql: Support connecting to extension databases [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) (owner: 10Majavah) [17:26:22] (03PS2) 10Majavah: sql: Support connecting to extension databases [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) [17:27:10] (03update) 10chuckonwumelu: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) [17:32:35] 06cloud-services-team, 10Cloud-VPS: YAML aliases do not seem to pass cleanly through from Horizon to the Puppetserver - https://phabricator.wikimedia.org/T395940 (10bd808) 03NEW [17:36:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [17:36:40] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) [17:37:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T395910) [17:37:08] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [17:37:14] (03update) 10chuckonwumelu: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) [17:37:42] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10880997 (10fnegri) [17:39:49] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Remove the compatibility layer of block schema in wikireplicas - https://phabricator.wikimedia.org/T390767#10881012 (10fnegri) After checking with @jcrespo and @Marostegui I did a `kill -9` and restarted mariadb. The server is now back in sync and re... [17:42:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T395910) [17:42:14] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [17:43:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:43:24] (03update) 10fnegri: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) (owner: 10chuckonwumelu) [17:43:32] (03update) 10fnegri: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) (owner: 10chuckonwumelu) [17:50:23] (03CR) 10FNegri: [C:03+1] "LGTM, but I'd wait for a second +1 because I've never touched this script before :)" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) (owner: 10Majavah) [17:50:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T395910) [17:50:30] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [17:51:10] (03approved) 10fnegri: Adding warning message for beta [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T394277) (owner: 10chuckonwumelu) [17:54:35] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-cli,components-api] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277#10881092 (10fnegri) [17:54:58] (03PS4) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support overriding number of expected drives [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137802 (https://phabricator.wikimedia.org/T392366) [17:54:58] (03PS1) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [17:55:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T395910) [17:55:49] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [17:57:48] (03PS2) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [17:58:44] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [17:59:25] 10Toolforge (Toolforge iteration 20), 07good first task: [components-api] use the `build.params.image_name` to compare with the `component` - https://phabricator.wikimedia.org/T395076#10881126 (10fnegri) @Chuckonwumelu this task is also a good one for you to work on, after you complete {T395035}. [18:00:03] (03update) 10fnegri: [build] Return image_name when retrieving build info [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T395035) (owner: 10chuckonwumelu) [18:00:21] (03PS3) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:03:55] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [18:06:08] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, and 2 others: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10881195 (10fnegri) [18:07:51] (03PS4) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:11:41] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [18:15:36] (03PS5) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:19:22] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [18:21:47] (03PS6) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:21:53] 06cloud-services-team, 10Data-Services, 10Tool-wikiloves: Frequent "Aborted connections" to Wiki Replicas - https://phabricator.wikimedia.org/T395952 (10fnegri) 03NEW [18:25:43] (03PS7) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:28:19] (03approved) 10fnegri: [build] Return image_name when retrieving build info [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T395035) (owner: 10chuckonwumelu) [18:28:55] (03PS8) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:29:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T395910) [18:29:07] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [18:32:33] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [18:34:14] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T395910) [18:34:21] T395910: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910 [18:43:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:43:04] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) [18:44:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:44:37] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) [18:45:27] (03PS9) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [18:45:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [18:51:07] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [19:21:51] (03PS10) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [19:22:35] (03PS11) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [19:22:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:25:47] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [19:27:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [19:35:57] (03PS12) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [19:36:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [19:40:41] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [19:41:51] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [19:51:43] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10881532 (10Andrew) 05Resolved→03Open a:05Jclark-ctr→03cmooney The cookbook is rejecting cloudcephosd1048 for failing network tests, so I assume th... [19:53:59] (03PS13) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [19:57:26] (03CR) 10BryanDavis: [C:03+2] sql: Remove unnecessary parens [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153314 (owner: 10Majavah) [19:57:53] (03CR) 10CI reject: [V:04-1] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [19:58:29] (03Merged) 10jenkins-bot: sql: Remove unnecessary parens [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153314 (owner: 10Majavah) [19:59:03] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959 (10Duckmather) 03NEW [19:59:48] (03CR) 10BryanDavis: [C:03+2] sql: Support connecting to extension databases [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) (owner: 10Majavah) [20:03:16] (03PS14) 10Andrew Bogott: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 [20:06:07] (03Merged) 10jenkins-bot: sql: Support connecting to extension databases [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153315 (https://phabricator.wikimedia.org/T395901) (owner: 10Majavah) [20:08:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [20:12:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [20:15:02] (03PS1) 10Majavah: build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 [20:16:30] (03CR) 10CI reject: [V:04-1] build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 (owner: 10Majavah) [20:17:22] (03PS2) 10Majavah: build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 [20:17:27] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881601 (10Curb_Safe_Charmer) a:03Curb_Safe_Charmer [20:17:35] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881602 (10Curb_Safe_Charmer) 05Open→03In progress [20:19:24] (03CR) 10CI reject: [V:04-1] build: Add gbp.conf with id-length [labs/toollabs] - 10https://gerrit.wikimedia.org/r/1153344 (owner: 10Majavah) [20:26:42] (03CR) 10Andrew Bogott: [C:03+2] wmcs.ceph.osd.bootstrap_and_add: support overriding number of expected drives [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137802 (https://phabricator.wikimedia.org/T392366) (owner: 10Andrew Bogott) [20:26:44] (03CR) 10Andrew Bogott: [C:03+2] wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [20:30:16] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881683 (10Curb_Safe_Charmer) That edit had an edit summary of "Filled in 13 bare reference(s) with reFill 2, Script-assisted style fixes and per CS1" and included a bunch of manual of style changes that reFill is incapable... [20:31:10] (03Merged) 10jenkins-bot: wmcs.ceph.osd.bootstrap_and_add: support overriding number of expected drives [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1137802 (https://phabricator.wikimedia.org/T392366) (owner: 10Andrew Bogott) [20:31:11] (03Merged) 10jenkins-bot: wmcs.ceph.osd.bootstrap_and_add: support OS drives in hw raid (mostly for boss cards) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1153326 (owner: 10Andrew Bogott) [20:32:53] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881708 (10Pppery) This is the fault of the website - if I disable JavaScript and load the website that genuinely is its title. Perhaps reFill (or Zotero, or Citoid) could filer out the garbage input and not GIGO it, though. [20:33:43] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881713 (10Curb_Safe_Charmer) I used the insert citation button in Visual Editor to test whether this is specific to reFill or affects Citoid more generally. I can confirm that the same behaviour happens when using Visual E... [20:38:32] 10Tool-refill: pagetitles.main bug in reFill - https://phabricator.wikimedia.org/T395959#10881724 (10Curb_Safe_Charmer) I agree with Pppery. If I browse to https://collections.ram.ac.uk/IMU/#/details/ecatalogue/893 and view the source, the title element is as follows: {{ 'pagetitles.main' | translate}}{{... [20:43:17] <wikibugs> (03update) 10raymond-ndibe: [components-smoke-test] reduce testcase dep on each other [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) [20:45:47] <wikibugs> 06cloud-services-team, 10Toolforge, 10Tools: Flickr blocking image requests from Toolforge k8s, breaking multiple tools - https://phabricator.wikimedia.org/T384468#10881760 (10Don-vip) It works again! Thank you so much Andrew! [21:05:54] <wikibugs> 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10881855 (10Andrew) I think I don't care what port it's in -- that's the same rack, right? [21:09:27] <wikibugs> 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcephosd200[567] - https://phabricator.wikimedia.org/T393614#10881864 (10Jhancock.wm) yeah same rack. I just need someone to migrate it for me since cloud is more complex than the netbox script can handle. and possibl... [22:00:09] <wikibugs> (03CR) 10Deni: [C:03+1] Change Serbocroatian to Serbo-Croatian [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1153275 (https://phabricator.wikimedia.org/T395915) (owner: 10Acamicamacaraca) [22:03:46] <wikibugs> (03open) 10andrew: Update to networks and secgroups for octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/243 [22:04:14] <wikibugs> (03close) 10andrew: Update to networks and secgroups for octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/243 [22:06:31] <wikibugs> (03open) 10andrew: Update secgroups and dns for octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:06:52] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:07:14] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:07:28] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:07:46] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:07:54] <wikibugs> (03merge) 10andrew: Update secgroups and dns for octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/244 [22:07:56] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [22:08:31] <logmsgbot_cloud> !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [23:36:56] <wikibugs> (03update) 10raymond-ndibe: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] (fix_do_run_image_name_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) [23:39:42] <wikibugs> (03update) 10raymond-ndibe: [deploy] add force-build and force-run query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044) [23:55:07] <wmcs-alerts> FIRING: HarborComponentDown: A Harbor component is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [23:57:48] <jinxer-wm> FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure