[00:02:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:07:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:16:18] FIRING: [2x] KernelErrors: Server cloudcephmon1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [00:39:11] FIRING: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [03:13:13] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1137741 (owner: 10L10n-bot) [04:16:18] FIRING: [2x] KernelErrors: Server cloudcephmon1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [06:36:52] 06cloud-services-team: KernelErrors Server cloudcephmon1004 logged kernel errors - https://phabricator.wikimedia.org/T392423#10759379 (10taavi) [06:37:03] 06cloud-services-team, 10Cloud-VPS: KernelErrors Server cloudcephmon1004 logged kernel errors - https://phabricator.wikimedia.org/T392423#10759381 (10taavi) [06:39:47] 06cloud-services-team, 10Toolforge: Toolforge OpenTofu support - https://phabricator.wikimedia.org/T329425#10759386 (10taavi) [06:42:31] supertassu closed https://github.com/toolforge/quarry/pull/78 [07:04:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [07:05:58] 10Quarry: Quarry test suite is not being run anymore - https://phabricator.wikimedia.org/T392385#10759427 (10taavi) 05Open→03Resolved [07:10:24] 10Quarry: Remove gerrit git from quarry puppet - https://phabricator.wikimedia.org/T348748#10759437 (10taavi) a:05rook→03taavi [07:10:40] 10Quarry: Remove gerrit git from quarry puppet - https://phabricator.wikimedia.org/T348748#10759438 (10taavi) 05Open→03Resolved a:05taavi→03rook [07:40:20] 06cloud-services-team, 10Cloud-VPS: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10759606 (10fgiunchedi) The user-data hint is what I was missing: I provisioned `filippo-centrallog-03.o11y.eqiad1.wikimedia.cloud` from Horizon with user-data set (from the Configuration... [07:50:54] 06cloud-services-team, 10Quarry: Update quarry redis deployment - https://phabricator.wikimedia.org/T392141#10759674 (10taavi) 05Open→03Resolved [07:57:11] supertassu opened https://github.com/toolforge/quarry/pull/80 [08:17:05] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10759766 (10fgiunchedi) 05Open→03Resolved All good, resolving [08:34:28] 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Reconsider severity of ProjectProxyMainProxyDown - https://phabricator.wikimedia.org/T381107#10759810 (10taavi) The runbook part is already tracked as {T361873}. [08:34:35] 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Reconsider severity of ProjectProxyMainProxyDown - https://phabricator.wikimedia.org/T381107#10759814 (10taavi) [08:36:33] 06cloud-services-team, 10Cloud-VPS, 10Sustainability (Incident Followup): Reconsider severity of ProjectProxyMainProxyDown - https://phabricator.wikimedia.org/T381107#10759817 (10taavi) There are two options for paging only when both are down: * Add a new scrape job for the VIP, and alert on that (more relia... [08:38:09] 06cloud-services-team, 10Cloud-VPS, 10Tool-spacemedia: DNS name resolution failure with cdn.esahubble.org from Cloud VPS & Toolforge - https://phabricator.wikimedia.org/T368439#10759830 (10taavi) 05Open→03Resolved [08:38:31] 06cloud-services-team, 10Cloud-VPS: Remove matanya as an admin from VPS projects - https://phabricator.wikimedia.org/T368330#10759832 (10taavi) 05Open→03Resolved a:03bd808 [08:42:44] (03update) 10aborrero: eqiad1: cloudinfra: introduce PTR zones for 2a02:ec80:a000:: [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/205 (https://phabricator.wikimedia.org/T380746) [08:44:34] (03update) 10aborrero: eqiad1: cloudinfra: introduce PTR zones for 2a02:ec80:a000:: [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/205 (https://phabricator.wikimedia.org/T380746) [08:54:10] (03open) 10aborrero: codfw1dev: dns: drop unused IPv6 reverse zones [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/206 [09:02:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#10759920 (10taavi) 05Open→03Resolved [09:04:52] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10759927 (10ops-monitoring-bot) Host rebooted by aborrero@cumin1002 with reason: enable IPv6 [09:07:43] supertassu closed https://github.com/toolforge/quarry/pull/80 [09:13:21] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Error parsing "/var/lib/prometheus/node.d/node_cloudvirt_libvirt_stats.prom" - https://phabricator.wikimedia.org/T289563#10759968 (10taavi) 05Open→03Resolved [09:22:54] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10760010 (10fnegri) 05In progress→03Resolved There was definitely an improvement, but Inlet Temp for clouddumps1001 remains about... [09:35:51] (03merge) 10aborrero: eqiad1: cloudinfra: introduce PTR zones for 2a02:ec80:a000:: [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/205 (https://phabricator.wikimedia.org/T380746) [09:35:55] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [09:38:39] (03update) 10aborrero: eqiad1: enable VXLAN/dualstack network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/204 (https://phabricator.wikimedia.org/T380174) [09:39:01] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [09:44:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10760105 (10dcaro) @Nokib_Sarkar I created https://github.com/nokibsarkar/campwiz-test/pull/1 , that should fix the issue :) Tested it on my local dev environment... [09:54:50] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468 (10taavi) 03NEW [10:07:26] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10760156 (10ops-monitoring-bot) Host rebooted by aborrero@cumin1002 with reason: enable IPv6 [10:10:09] (03PS1) 10Vgutierrez: secret: Add wmfuniq snakeoil [labs/private] - 10https://gerrit.wikimedia.org/r/1138307 (https://phabricator.wikimedia.org/T391411) [10:11:18] (03CR) 10Vgutierrez: [C:03+2] secret: Add wmfuniq snakeoil [labs/private] - 10https://gerrit.wikimedia.org/r/1138307 (https://phabricator.wikimedia.org/T391411) (owner: 10Vgutierrez) [10:11:24] (03CR) 10Vgutierrez: [V:03+2 C:03+2] secret: Add wmfuniq snakeoil [labs/private] - 10https://gerrit.wikimedia.org/r/1138307 (https://phabricator.wikimedia.org/T391411) (owner: 10Vgutierrez) [10:14:09] (03merge) 10aborrero: eqiad1: enable VXLAN/dualstack network [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/204 (https://phabricator.wikimedia.org/T380174) [10:14:20] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:15:18] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:17:01] (03open) 10aborrero: Revert "eqiad1: enable VXLAN/dualstack network" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/207 [10:17:29] (03update) 10raymond-ndibe: [jobs-cli] only send timeout if it's set by the user [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T389118) [10:18:26] (03merge) 10aborrero: Revert "eqiad1: enable VXLAN/dualstack network" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/207 [10:18:33] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:42:07] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:44:09] (03update) 10taavi: Upgrade dependencies [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/4 [10:46:08] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: dns: add PTR support for 2a02:ec80:a000:: - https://phabricator.wikimedia.org/T380746#10760220 (10cmooney) The openstack DNS is now returning SOA records for the one /64 configured thus far: ` cmooney@cumin1002:~$ dig +noall +answer SOA 1.0.0.... [10:52:55] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:05:30] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:18:04] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:22:07] (03open) 10aborrero: eqiad1: network: add VXLAN/dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/208 (https://phabricator.wikimedia.org/T380174) [11:28:29] (03approved) 10taavi: eqiad1: network: add VXLAN/dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/208 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:29:05] (03merge) 10aborrero: eqiad1: network: add VXLAN/dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/208 (https://phabricator.wikimedia.org/T380174) [11:29:27] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:30:05] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:33:31] (03open) 10aborrero: eqiad1: subnets: introduce vxlan-dualstack-ipv4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/209 (https://phabricator.wikimedia.org/T380174) [11:34:15] (03approved) 10dcaro: run: mark as skipped if the deploy failed [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/66 [11:34:18] (03merge) 10dcaro: run: mark as skipped if the deploy failed [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/66 [11:36:44] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.102-20250423113430-2abb5ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/751 [11:40:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [11:40:42] (03approved) 10taavi: eqiad1: subnets: introduce vxlan-dualstack-ipv4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/209 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:42:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10760362 (10dcaro) > There seems to be a weird behavior of the procfile buildpack, where if you only declare the web entry it will not override the golang buildpac... [11:42:50] (03merge) 10aborrero: eqiad1: subnets: introduce vxlan-dualstack-ipv4 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/209 (https://phabricator.wikimedia.org/T380174) [11:42:54] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:44:15] (03open) 10aborrero: eqiad1: subnets: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/210 (https://phabricator.wikimedia.org/T380174) [11:44:30] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:45:06] (03approved) 10taavi: eqiad1: subnets: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/210 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:45:20] (03merge) 10aborrero: eqiad1: subnets: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/210 (https://phabricator.wikimedia.org/T380174) [11:45:21] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:45:58] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:47:06] (03open) 10aborrero: eqiad1: routers: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/211 (https://phabricator.wikimedia.org/T380174) [11:47:54] (03approved) 10taavi: eqiad1: routers: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/211 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:48:07] (03merge) 10aborrero: eqiad1: routers: introduce vxlan-dualstack-ipv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/211 (https://phabricator.wikimedia.org/T380174) [11:48:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:48:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Environment variables are not being passed - https://phabricator.wikimedia.org/T390845#10760378 (10dcaro) [11:49:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#10760379 (10dcaro) [11:49:05] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:49:17] (03open) 10aborrero: eqiad1: ports: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/212 (https://phabricator.wikimedia.org/T380174) [11:49:34] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [11:52:22] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:52:50] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:53:31] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:54:04] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:54:08] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10760389 (10Marostegui) The `_p` database isn't created on Sanitariums, they are created on the wikirep... [11:54:08] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:54:36] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:55:39] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [11:55:42] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikifunctions: Make wikifunctionsclient_usage table available on cloud wiki replicas - https://phabricator.wikimedia.org/T392475 (10LucasWerkmeister) 03NEW [11:56:36] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikifunctions: Make wikifunctionsclient_usage table available on cloud wiki replicas - https://phabricator.wikimedia.org/T392475#10760405 (10Ladsgroup) Please catalog the table first. [11:56:51] (03approved) 10taavi: eqiad1: ports: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/212 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:57:04] (03merge) 10aborrero: eqiad1: ports: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/212 (https://phabricator.wikimedia.org/T380174) [11:57:12] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:57:54] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:58:38] (03open) 10aborrero: eqiad1: ports: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/213 (https://phabricator.wikimedia.org/T380174) [11:59:21] (03approved) 10taavi: eqiad1: ports: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/213 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [11:59:51] (03merge) 10aborrero: eqiad1: ports: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/213 (https://phabricator.wikimedia.org/T380174) [11:59:53] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:00:32] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:00:39] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [12:01:21] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [12:01:37] (03update) 10raymond-ndibe: [envvars-cli] hide envvar if truncate true [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [12:01:43] (03open) 10aborrero: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/214 (https://phabricator.wikimedia.org/T380174) [12:01:45] (03update) 10raymond-ndibe: [envvars-cli] hide envvar if truncate true [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [12:02:03] (03update) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [12:02:29] (03approved) 10taavi: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/214 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [12:02:49] (03merge) 10aborrero: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/214 (https://phabricator.wikimedia.org/T380174) [12:02:53] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:02:58] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.369-20250423103208-3adcb40a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/750 (https://phabricator.wikimedia.org/T352989) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:03:01] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.369-20250423103208-3adcb40a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/750 (https://phabricator.wikimedia.org/T352989) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:03:11] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.369-20250423103208-3adcb40a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/750 (https://phabricator.wikimedia.org/T352989) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:03:32] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:03:39] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:04:06] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:06:26] (03update) 10aborrero: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215 (https://phabricator.wikimedia.org/T380174) [12:06:28] (03open) 10aborrero: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215 (https://phabricator.wikimedia.org/T380174) [12:06:58] (03approved) 10taavi: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [12:07:38] (03merge) 10aborrero: eqiad1: router_intercaces: introduce cloudinstances2b-gw-dualstack-v6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/215 (https://phabricator.wikimedia.org/T380174) [12:07:39] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:08:20] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:08:23] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:08:51] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:10:45] (03open) 10aborrero: eqiad1: dns: add records for new neutron router addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/216 (https://phabricator.wikimedia.org/T380174) [12:11:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [12:11:51] (03approved) 10taavi: eqiad1: dns: add records for new neutron router addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/216 (https://phabricator.wikimedia.org/T380174) (owner: 10aborrero) [12:11:56] 10Cloud-Services, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478 (10MoritzMuehlenhoff) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.or... [12:12:22] (03merge) 10aborrero: eqiad1: dns: add records for new neutron router addresses [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/216 (https://phabricator.wikimedia.org/T380174) [12:12:23] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:12:58] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:13:06] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#10760474 (10fnegri) [12:13:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10760475 (10fnegri) [12:14:19] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10760476 (10fnegri) [12:14:40] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10760477 (10fnegri) @Marostegui you're right, I corrected the task description. [12:14:57] 06cloud-services-team, 10Horizon, 10Striker, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#10760479 (10taavi) [12:15:21] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:15:49] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:15:50] (03approved) 10dcaro: components-api: bump to 0.0.102-20250423113430-2abb5ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/751 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:16:07] (03update) 10dcaro: components-api: bump to 0.0.102-20250423113430-2abb5ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/751 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:16:19] (03merge) 10dcaro: components-api: bump to 0.0.102-20250423113430-2abb5ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/751 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:19:20] (03open) 10aborrero: networktests: create dualstack infra in both deployments [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/16 (https://phabricator.wikimedia.org/T380174) [12:22:09] (03merge) 10aborrero: networktests: create dualstack infra in both deployments [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/16 (https://phabricator.wikimedia.org/T380174) [12:26:14] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10760504 (10aborrero) we discovered that when enabling https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/commit/1a85f66e47b615f16e9dc492d823c313c0bdf086 on the... [12:28:39] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] (add_runner_parameter) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [12:56:23] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [13:01:09] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [13:03:11] (03approved) 10dcaro: build: add runner parameter [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T380127) [13:03:15] (03merge) 10dcaro: build: add runner parameter [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T380127) [13:03:16] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [13:04:52] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [13:06:00] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10760644 (10ops-monitoring-bot) Host rebooted by aborrero@cumin1002 with reason: enable IPv6 [13:09:04] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.187-20250423130326-2e1c3b05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/752 (https://phabricator.wikimedia.org/T380127) [13:14:13] (03update) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [13:18:50] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [13:23:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:32:13] (03open) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127) [13:35:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [13:36:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:40:36] (03approved) 10dcaro: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) (owner: 10raymond-ndibe) [13:47:17] (03open) 10taavi: codfw1dev: Fix zone comments [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 [13:50:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:09:54] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004#10760911 (10fnegri) @Andrew any updates on this one? [14:12:32] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [14:13:30] (03approved) 10dcaro: builds-api: bump to 0.0.187-20250423130326-2e1c3b05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/752 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:13:34] (03merge) 10dcaro: builds-api: bump to 0.0.187-20250423130326-2e1c3b05 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/752 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:18:18] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [14:18:28] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004#10760943 (10Andrew) 05Stalled→03Invalid No longer needed, we're back to using codfw1dev. [14:21:58] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 [14:22:35] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 [14:24:34] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [14:25:37] (03update) 10aborrero: codfw1dev: dns: drop unused IPv6 reverse zones [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/206 [14:26:05] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10760977 (10fnegri) > I'm going to assume that @fnegri is seeing flavors that are hidden from mor... [14:26:58] (03merge) 10aborrero: codfw1dev: dns: drop unused IPv6 reverse zones [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/206 [14:27:03] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:27:38] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:27:44] aborrero@cloudcumin1001: Failed to log message to wiki. Somebody should check the error logs. [14:27:45] (03update) 10taavi: codfw1dev: Fix zone comments [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 [14:28:00] (03approved) 10aborrero: codfw1dev: Fix zone comments [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 (owner: 10taavi) [14:28:28] (03merge) 10taavi: codfw1dev: Fix zone comments [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/217 [14:28:43] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:29:18] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:31:42] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [14:45:16] (03open) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:45:24] (03update) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:46:07] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [14:48:12] (03update) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:48:44] (03update) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:49:56] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [14:56:18] (03update) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:56:37] (03update) 10raymond-ndibe: [envvars-cli] test hide envvars value by default [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/753 (https://phabricator.wikimedia.org/T363544) [14:57:57] (03update) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [14:58:03] (03update) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [14:58:05] (03approved) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [14:59:12] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [15:00:41] (03open) 10aborrero: tofu-provisioning: enable for tools [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/4 [15:01:48] (03update) 10aborrero: tofu-provisioning: enable for tools [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/4 [15:01:52] (03merge) 10raymond-ndibe: [envvars-cli] hide envvar by default [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/78 (https://phabricator.wikimedia.org/T363544) [15:03:14] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [15:05:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:19:35] (03merge) 10aborrero: tofu-provisioning: enable for tools [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/4 [15:22:21] 06cloud-services-team, 10Cloud-VPS: gitlab ci: validate secrets settings in pipeline for tofu integration - https://phabricator.wikimedia.org/T391467#10761333 (10aborrero) data point: the setup was also extended to cover tofu in here: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge... [15:22:43] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10761335 (10Andrew) Here's my mortal user adding a VM in zuul3. The user is a member and reader b... [15:24:26] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service - https://phabricator.wikimedia.org/T392506 (10taavi) 03NEW [15:24:34] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service - https://phabricator.wikimedia.org/T392506#10761364 (10taavi) [15:24:40] 06cloud-services-team, 10Cloud-VPS, 07Epic, 07IPv6: Enable IPv6 on CloudVPS - https://phabricator.wikimedia.org/T37947#10761365 (10taavi) [15:25:17] (03open) 10taavi: legacy_redirector: New module to provision VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/5 (https://phabricator.wikimedia.org/T392506) [15:26:18] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.13 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/79 (https://phabricator.wikimedia.org/T363544) [15:26:29] (03update) 10taavi: legacy_redirector: New module to provision VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/5 (https://phabricator.wikimedia.org/T392506) [15:27:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [15:28:00] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli [15:28:15] (03update) 10taavi: legacy_redirector: New module to provision VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/5 (https://phabricator.wikimedia.org/T392506) [15:30:48] (03approved) 10aborrero: legacy_redirector: New module to provision VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/5 (https://phabricator.wikimedia.org/T392506) (owner: 10taavi) [15:31:32] (03merge) 10taavi: legacy_redirector: New module to provision VMs [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/5 (https://phabricator.wikimedia.org/T392506) [15:31:45] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikifunctions, and 2 others: Make wikifunctionsclient_usage table available on cloud wiki replicas - https://phabricator.wikimedia.org/T392475#10761406 (10Jdforrester-WMF) [15:32:28] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli [15:32:48] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli [15:33:33] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [15:43:57] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [15:44:34] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikifunctions, and 2 others: Make wikifunctionsclient_usage table available on cloud wiki replicas - https://phabricator.wikimedia.org/T392475#10761495 (10Jdforrester-WMF) p:05Triage→03Low [15:55:09] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud [15:55:58] (03update) 10raymond-ndibe: d/changelog: bump to 0.0.13 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/79 (https://phabricator.wikimedia.org/T363544) [15:56:42] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud [15:59:49] (03open) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:00:57] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:01:39] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:02:55] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:04:00] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:06:02] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:06:39] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-3:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-3:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:11:32] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:14:48] 06cloud-services-team, 10Toolforge: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10761676 (10Raymond_Ndibe) > What's on view on it? I'm of the opinion that the priority is low. This is because yes, tests take a long time to r... [16:15:01] 06cloud-services-team, 10Toolforge: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10761677 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:15:11] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for Toolforge services - https://phabricator.wikimedia.org/T392509 (10taavi) 03NEW [16:15:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10761690 (10Raymond_Ndibe) [16:15:38] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for Toolforge services - https://phabricator.wikimedia.org/T392509#10761691 (10taavi) [16:15:43] 06cloud-services-team, 10Toolforge, 07IPv6, 13Patch-For-Review: Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service - https://phabricator.wikimedia.org/T392506#10761692 (10taavi) [16:15:50] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 on toolforge.org - https://phabricator.wikimedia.org/T211575#10761693 (10taavi) [16:15:57] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 on toolforge.org - https://phabricator.wikimedia.org/T211575#10761694 (10taavi) [16:16:07] 06cloud-services-team, 10Toolforge, 07IPv6, 13Patch-For-Review: Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service - https://phabricator.wikimedia.org/T392506#10761695 (10taavi) [16:16:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api,jobs-cli] Replace already completed one-off jobs when starting a new one - https://phabricator.wikimedia.org/T352989#10761705 (10Raymond_Ndibe) 05In progress→03Resolved [16:17:00] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 on the Toolforge bastion - https://phabricator.wikimedia.org/T392510 (10taavi) 03NEW [16:17:21] 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for Toolforge mail server - https://phabricator.wikimedia.org/T392511 (10taavi) 03NEW [16:17:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [16:17:57] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [16:18:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [16:18:34] 06cloud-services-team, 10Cloud-VPS, 07IPv6: dns: add PTR support for 2a02:ec80:a000:: - https://phabricator.wikimedia.org/T380746#10761750 (10taavi) 05Open→03Resolved a:03cmooney [16:20:19] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 07IPv6, 13Patch-For-Review: Enable IPv6 for tools.wmflabs.org / *.toolserver.org legacy redirector service - https://phabricator.wikimedia.org/T392506#10761764 (10taavi) [16:23:36] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:24:59] (03update) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127) [16:26:37] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [16:27:03] (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127) [16:27:42] (03update) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127) [16:27:50] (03update) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127) [16:29:02] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [16:29:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [16:29:27] (03update) 10taavi: legacy_redirector: Add IPv6 records for toolserver.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6 (https://phabricator.wikimedia.org/T392506) [16:29:31] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) [16:29:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [16:31:37] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) [16:32:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [16:32:25] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) [16:32:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [16:47:13] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [16:50:20] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [17:14:15] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [17:23:15] 10Tools, 10Wikidata, 07TestMe: ListeriaBot returns 502 Bad Gateway error - Requires probably a server restart - https://phabricator.wikimedia.org/T324882#10762126 (10matej_suchanek) [17:49:16] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [17:54:53] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 07IPv6: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10762365 (10Ottomata) @taavi thanks! What's the timeline for this? [18:02:52] 10Toolforge (Toolforge iteration 19): Better toolforge cli deployment flow - https://phabricator.wikimedia.org/T392524 (10Raymond_Ndibe) 03NEW [18:03:15] 10Toolforge (Toolforge iteration 19), 07Epic: Better toolforge cli deployment flow - https://phabricator.wikimedia.org/T392524#10762396 (10Raymond_Ndibe) [18:04:43] 10Toolforge (Toolforge iteration 19), 07Epic: DRAFT: Fix toolforge tests and deployment cicd pipelines - https://phabricator.wikimedia.org/T392524#10762399 (10Raymond_Ndibe) [18:29:27] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [18:29:58] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [18:46:27] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [18:48:20] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [18:58:19] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [19:03:33] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [19:03:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [19:04:46] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [19:05:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:15:33] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [19:15:56] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [19:16:06] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 [19:21:35] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 [19:37:04] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10762640 (10JJMC89) 05Resolved→03Open a:05bd808→03None >>! In T374993#10222631, @KFrancis wrote: > Hi all, I am confirming @JJMC89 has an NDA on file. Thanks! The referenced NDA d... [19:44:22] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [19:49:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:49:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:53:07] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10762722 (10bd808) p:05Triage→03High Raising priority as an indication of value to the Toolforge community and as correction for prior communication difficulties. [19:59:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:06:43] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [20:12:30] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [20:15:46] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10762810 (10KFrancis) Thanks all! I'll put the agreement together and send it out today for signatures. [20:22:59] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [21:19:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:21:14] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) [21:21:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:22:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [21:23:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:23:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [21:23:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:24:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [21:25:02] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:25:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [21:26:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:26:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [21:37:30] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [21:40:05] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [21:44:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [21:45:33] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [21:45:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [21:45:49] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) [21:51:12] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [21:59:31] 06cloud-services-team: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539 (10Andrew) 03NEW [22:39:49] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [23:34:01] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [23:36:45] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [23:37:48] FIRING: PuppetFailure: Puppet has failed on cloudlb2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:37:55] 06cloud-services-team: PuppetFailure Puppet has failed on cloudlb2004-dev:9100 - https://phabricator.wikimedia.org/T392543 (10phaultfinder) 03NEW [23:43:50] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 [23:47:12] (03update) 10chuckonwumelu: Draft: Started DNS imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3