[00:19:14] (03PS1) 10Andrew Bogott: Add secret octavia heartbeat keys [labs/private] - 10https://gerrit.wikimedia.org/r/1146784 [00:21:30] (03PS2) 10Andrew Bogott: Add secret octavia heartbeat keys [labs/private] - 10https://gerrit.wikimedia.org/r/1146784 [00:21:45] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Add secret octavia heartbeat keys [labs/private] - 10https://gerrit.wikimedia.org/r/1146784 (owner: 10Andrew Bogott) [00:27:02] (03PS1) 10Andrew Bogott: Correct spelling of 'heartbeat' [labs/private] - 10https://gerrit.wikimedia.org/r/1146785 [00:28:21] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Correct spelling of 'heartbeat' [labs/private] - 10https://gerrit.wikimedia.org/r/1146785 (owner: 10Andrew Bogott) [00:58:48] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Test (and implement?) Openstack Octavia lbaas - https://phabricator.wikimedia.org/T393783#10828524 (10Andrew) [x] fix health check traffic from amphora -> cloudcontrol [] move amphorae out of 'service' project into a properly named 'octavia' project []... [01:02:19] 06cloud-services-team, 10Toolforge, 10Toolhub: Require tools to host a valid toolinfo.json file (e.g. while upgrading from one Debian version to another) - https://phabricator.wikimedia.org/T271712#10828527 (10bd808) What does "require" mean? I have a lot of naive questions about how this would be expected t... [01:03:21] 06cloud-services-team, 10Toolforge: [build-service] Allow custom tagging of build service generated images - https://phabricator.wikimedia.org/T369192#10828530 (10bd808) [01:07:42] 06cloud-services-team, 10Toolforge: [build-service] Allow custom tagging of build service generated images - https://phabricator.wikimedia.org/T369192#10828535 (10bd808) I just did the first major update to a shared container that I personally maintain (https://gitlab.wikimedia.org/toolforge-repos/containers-b... [02:03:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:35:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:45:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:28:37] 06cloud-services-team, 10Data-Services, 07affects-Kiwix-and-openZIM: enwiki_p query returned empty results on May 14 from ~UTC 0:00 - 05:00 - https://phabricator.wikimedia.org/T394429#10828635 (10Kelson) [06:03:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [07:50:39] (03update) 10arthurtaylor: Add support for build completion notification [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/10 (https://phabricator.wikimedia.org/T392892) [07:51:17] (03update) 10arthurtaylor: Add support for build completion notification [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/10 (https://phabricator.wikimedia.org/T392892) [07:55:40] (03update) 10arthurtaylor: Add support for build completion notification [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/10 (https://phabricator.wikimedia.org/T392892) [08:09:14] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#10828807 (10dcaro) [08:10:50] 06cloud-services-team, 10Toolforge: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack - https://phabricator.wikimedia.org/T394466#10828811 (10dcaro) Thanks @bd808, I've added a preliminar things to test in the parent task, if you find/notice any issues can you add them... [08:15:19] 06cloud-services-team: Emails to cloudservices@wikimedia.org from root@beta.toolforge.org bouncing - https://phabricator.wikimedia.org/T394453#10828813 (10aborrero) a:03aborrero [08:58:10] 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10829003 (10Gehel) p:05Triage→03Medium [08:58:49] 10Data-Services, 06Data-Engineering, 06Data-Platform-SRE: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10829005 (10Gehel) @EBernhardson do you know enough about JsonConfig (and maybe charts) to validate Ben's analysis above? [09:01:37] 10Data-Services, 06Data-Engineering, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10829021 (10Gehel) [09:24:10] (03open) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:26:33] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:28:19] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:30:15] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:31:48] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:32:44] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:36:17] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:37:35] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:49:11] (03merge) 10aborrero: tofu-provisioning: drop imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/23 [09:50:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [09:50:14] (03update) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [09:51:21] (03approved) 10aborrero: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) (owner: 10taavi) [09:52:54] (03merge) 10taavi: Manage Prometheus-related security and server groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/29 (https://phabricator.wikimedia.org/T393697) [10:03:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:05:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [10:10:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [10:44:07] 06cloud-services-team: Emails to cloudservices@wikimedia.org from root@beta.toolforge.org bouncing - https://phabricator.wikimedia.org/T394453#10829392 (10aborrero) 05Open→03Resolved manually edited the email address via this: `lang=shell-session aborrero@mwmaint1002:~ $ sudo ldapvi -b ou=people,dc=wiki... [10:46:00] 06cloud-services-team: Emails to cloudservices@wikimedia.org from root@beta.toolforge.org bouncing - https://phabricator.wikimedia.org/T394453#10829396 (10aborrero) [10:50:43] 06cloud-services-team: Emails to cloudservices@wikimedia.org from root@beta.toolforge.org bouncing - https://phabricator.wikimedia.org/T394453#10829405 (10taavi) 05Resolved→03Open Re-opening, because that doesn't really solve the problem. We do not want to send Puppet failure emails to these service accounts... [11:35:21] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.openstack.quota_increase [11:35:29] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [11:37:35] (03update) 10aborrero: kubeconfig: support updating the file [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/68 (https://phabricator.wikimedia.org/T262562) [11:38:52] (03open) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:40:01] (03update) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:41:04] (03update) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:42:04] (03update) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:44:13] (03update) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:44:55] (03update) 10aborrero: kubeconfig: support updating the file [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/68 (https://phabricator.wikimedia.org/T262562) [11:45:43] (03update) 10taavi: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) [11:51:54] (03merge) 10aborrero: Add toolsbeta-prometheus-2 [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/30 (https://phabricator.wikimedia.org/T393697) (owner: 10taavi) [11:56:35] (03open) 10taavi: Rename toolsbeta-prometheus-2 volume [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/31 [11:57:24] (03approved) 10aborrero: Rename toolsbeta-prometheus-2 volume [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/31 (owner: 10taavi) [11:57:38] (03merge) 10taavi: Rename toolsbeta-prometheus-2 volume [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/31 [12:01:59] (03merge) 10aborrero: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/65 (owner: 10ghost) [12:38:44] (03open) 10taavi: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 [12:40:06] (03update) 10taavi: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 [12:41:13] (03update) 10taavi: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 [12:41:22] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: [DRAFT] Decision Request - Initial product approach to integrating Toolforge UI functionality with Toolsadmin - https://phabricator.wikimedia.org/T393010#10829763 (10Sarai-WMF) [12:42:24] (03update) 10taavi: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 [12:53:01] (03approved) 10taavi: kubeconfig: support updating the file [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/68 (https://phabricator.wikimedia.org/T262562) (owner: 10aborrero) [12:53:06] (03update) 10taavi: kubeconfig: support updating the file [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/68 (https://phabricator.wikimedia.org/T262562) (owner: 10aborrero) [12:57:49] (03approved) 10aborrero: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 (owner: 10taavi) [12:59:07] (03merge) 10taavi: Manage toolsbeta Prometheus web proxies [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/32 [13:05:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:15:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:15:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:20:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:25:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:30:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [13:50:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:05:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:05:30] (03Abandoned) 10LD: frwiki: Enable the NewUserMessage extension [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1146702 (https://phabricator.wikimedia.org/T382199) (owner: 10LD) [14:23:00] 06cloud-services-team, 10Cloud-VPS, 07Documentation, 07IPv6: Cloud VPS: prepare documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054#10830126 (10aborrero) 05Open→03Resolved a:03aborrero [14:23:43] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10830130 (10aborrero) 05Open→03Resolved I think we can consider this ticket as resolved. We won't be "migrating" VMs into the VXLAN setup, but rebuild... [14:25:03] FIRING: [3x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [14:25:29] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515 (10Nokib_Sarkar) 03NEW [14:27:04] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830151 (10Nokib_Sarkar) [14:28:55] 06cloud-services-team, 10Toolforge: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516 (10fnegri) 03NEW [14:31:11] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830175 (10Nokib_Sarkar) [14:31:13] 06cloud-services-team, 10Toolforge: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516#10830174 (10Nokib_Sarkar) [14:32:10] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830180 (10Nokib_Sarkar) [14:56:00] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests): Request increased quota for Toolforge tool - https://phabricator.wikimedia.org/T394520 (10Nokib_Sarkar) 03NEW [14:56:16] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests): Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520#10830272 (10Nokib_Sarkar) [14:57:10] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests): Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520#10830276 (10Nokib_Sarkar) [14:58:05] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830278 (10Nokib_Sarkar) [14:58:42] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests): Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520#10830281 (10Nokib_Sarkar) [14:58:43] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830282 (10Nokib_Sarkar) [14:58:45] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests): Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520#10830284 (10aborrero) looks good to me, +1 [15:05:19] (03open) 10aborrero: maintain-harbor: bump build quota for campwiz [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/787 (https://phabricator.wikimedia.org/T394520) [15:16:44] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10830365 (10fnegri) p:05Triage→03Medium a:03fnegri [15:18:13] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:18:31] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:20:13] (03open) 10nokibsarkar: Should fix T394516 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:24:58] (03update) 10fnegri: Should fix T394516 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (owner: 10nokibsarkar) [15:27:53] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:32:23] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:34:34] (03update) 10nokibsarkar: Should fix T394516 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:34:44] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:36:49] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:36:58] (03update) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:37:11] (03update) 10nokibsarkar: Should fix T394516 [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:37:48] (03update) 10aborrero: dns: use zonename_recordname as opentofu state key [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/22 [15:37:51] (03merge) 10tburmeister: Finish styling of results, add legend, and fix a bug [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/4 (https://phabricator.wikimedia.org/T390390) [15:40:15] (03update) 10nokibsarkar: [builds-api] Main branch is used even when a different "ref" is specified. [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:40:24] (03update) 10aborrero: dns: use zonename_recordname as opentofu state key [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/22 [15:44:55] (03update) 10nokibsarkar: [builds-api] Main branch is used even when a different "ref" is specified. [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:47:12] (03update) 10nokibsarkar: [builds-api] Main branch is used even when a different "ref" is specified. [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 [15:49:51] (03CR) 10Krinkle: [C:03+2] "Looks like by very rare exception, someone decided to change an existing irc.wm.o source channel name (outreach.wikipedia > outreach.wikim" [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1146724 (owner: 10AntiCompositeNumber) [15:52:46] 06cloud-services-team, 10Toolforge: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516#10830527 (10Nokib_Sarkar) a:03Nokib_Sarkar [15:55:28] (03update) 10nokibsarkar: [builds-api] Main branch is used even when a different "ref" is specified. [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [16:06:46] (03update) 10fnegri: [builds-api] Main branch is used even when a different "ref" is specified. [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [16:21:24] 10Data-Services, 06Data-Engineering, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#10830585 (10EBernhardson) I don't know a ton about this, but i took a look at it. A few thoughts: * `x1` doesn't really mean i... [16:23:35] (03approved) 10fnegri: maintain-harbor: bump build quota for campwiz [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/787 (https://phabricator.wikimedia.org/T394520) (owner: 10aborrero) [16:31:04] (03update) 10nokibsarkar: [builds-api] ref is now being used for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [16:32:18] 06cloud-services-team, 10Toolforge: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516#10830614 (10Nokib_Sarkar) 05Open→03In progress [16:38:37] (03update) 10nokibsarkar: [builds-api] ref is now being used for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [16:39:59] (03update) 10nokibsarkar: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [16:43:01] (03update) 10nokibsarkar: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [16:43:22] (03merge) 10andrew: maintain-harbor: bump build quota for campwiz [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/787 (https://phabricator.wikimedia.org/T394520) (owner: 10aborrero) [16:43:31] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:43:36] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [16:43:51] (03update) 10fnegri: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [16:44:07] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:44:12] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [16:44:42] (03approved) 10fnegri: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [16:46:11] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor (T394520) [16:46:15] T394520: Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520 [16:46:16] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor (T394520) [16:46:46] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component harbor (T394520) [16:46:50] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor (T394520) [16:48:59] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor (T394520) [16:49:05] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor (T394520) [16:51:40] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor (T394520) [16:51:44] T394520: Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520 [16:55:22] 10Tool-campwiz-nxt, 10Toolforge (Quota-requests), 13Patch-For-Review: Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520#10830674 (10Andrew) 05Open→03Resolved a:03Andrew [16:58:59] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 [17:02:33] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor (T394520) [17:02:39] T394520: Request increased quota for campwiz Toolforge tool - https://phabricator.wikimedia.org/T394520 [17:03:09] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [17:09:59] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 [17:10:03] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-67 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcess [17:10:10] (03update) 10nokibsarkar: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:11:00] (03update) 10nokibsarkar: use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:12:49] (03update) 10nokibsarkar: Use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:13:42] (03update) 10nokibsarkar: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:14:00] (03update) 10nokibsarkar: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:20:27] 10Wikibugs, 07Technical-Debt: Refactor irc message formatting so that there is a clear separation between event ingestion and message output stages - https://phabricator.wikimedia.org/T394532 (10bd808) 03NEW [17:20:32] 10Wikibugs: Slack interation for Wikibugs - https://phabricator.wikimedia.org/T394533 (10bd808) 03NEW [17:21:04] 10Wikibugs: Slack interation for Wikibugs - https://phabricator.wikimedia.org/T394533#10830747 (10bd808) [17:21:05] 10Wikibugs, 07Technical-Debt: Refactor irc message formatting so that there is a clear separation between event ingestion and message output stages - https://phabricator.wikimedia.org/T394532#10830748 (10bd808) [17:24:28] 10Wikibugs: Slack interation for Wikibugs - https://phabricator.wikimedia.org/T394533#10830758 (10bd808) Actually writing this feature request down was inspired by seeing @Slack-bot in action in a WMF Slack channel. That project by @DSantamaria is a bit more like @Stashbot's Phabricator integration than the #wik... [17:25:06] 10Wikibugs: Slack interation for Wikibugs - https://phabricator.wikimedia.org/T394533#10830762 (10bd808) [17:26:56] (03CR) 10Krinkle: [V:03+2 C:03+2] megatable: Refresh data via /bin/update-www-var [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1146724 (owner: 10AntiCompositeNumber) [17:35:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:35:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:37:25] (03update) 10nokibsarkar: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) [17:40:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-9 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:44:08] (03PS1) 10Krinkle: megatable: Upgrade jQuery from 2.1.0 to 3.7.1 [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1147038 [17:44:08] (03PS1) 10Krinkle: megatable: Fix "XML Parsing Error: not well-formed" warning [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1147039 [17:44:28] (03CR) 10Krinkle: [V:03+2 C:03+2] megatable: Upgrade jQuery from 2.1.0 to 3.7.1 [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1147038 (owner: 10Krinkle) [17:48:22] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10830829 (10Nokib_Sarkar) 05Open→03In progress [17:48:32] (03PS2) 10Krinkle: megatable: Fix "XML Parsing Error: not well-formed" warning [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1147039 [17:59:11] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [18:07:08] (03update) 10raymond-ndibe: Draft: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [18:07:14] (03update) 10raymond-ndibe: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [18:13:45] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [18:14:43] (03open) 10dcaro: Use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T394516) [18:16:44] (03update) 10dcaro: DONOTMERGE Use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T394516) [18:25:11] (03approved) 10dcaro: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [18:25:49] (03merge) 10dcaro: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [18:25:51] (03update) 10dcaro: Use the given ref as a pattern in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T394516) (owner: 10nokibsarkar) [18:26:14] (03close) 10dcaro: DONOTMERGE Use the given ref as the 4th argument for pattern matching in git ls-remote [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T394516) [18:30:21] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516#10831026 (10Nokib_Sarkar) 05In progress→03Resolved [18:30:52] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10831030 (10Nokib_Sarkar) [18:32:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.192-20250516182559-51f1f993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/788 (https://phabricator.wikimedia.org/T394516) [18:34:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [18:34:57] 10wikitech.wikimedia.org: Cookie “WMF-Last-Access-Global” has been rejected for invalid domain. - https://phabricator.wikimedia.org/T376384#10831046 (10Krinkle) →14Duplicate dup:03T261803 [18:42:44] (03PS1) 10Andrew Bogott: Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 [18:44:00] (03CR) 10CI reject: [V:04-1] Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 (owner: 10Andrew Bogott) [18:45:30] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [18:45:35] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [18:50:27] (03PS2) 10Andrew Bogott: Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 [18:50:54] (03CR) 10CI reject: [V:04-1] Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 (owner: 10Andrew Bogott) [18:51:49] (03PS3) 10Andrew Bogott: Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 [18:52:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [builds-api] Main branch is used even when a different "ref" is specified - https://phabricator.wikimedia.org/T394516#10831125 (10dcaro) [18:52:22] (03CR) 10Andrew Bogott: [C:03+2] Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 (owner: 10Andrew Bogott) [18:53:34] (03Merged) 10jenkins-bot: Include octavia VIPs as targets for proxies [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147044 (owner: 10Andrew Bogott) [18:58:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [18:58:33] (03approved) 10dcaro: builds-api: bump to 0.0.192-20250516182559-51f1f993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/788 (https://phabricator.wikimedia.org/T394516) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:58:36] (03merge) 10dcaro: builds-api: bump to 0.0.192-20250516182559-51f1f993 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/788 (https://phabricator.wikimedia.org/T394516) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [19:15:17] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10831181 (10Nokib_Sarkar) [19:16:07] 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10831182 (10Nokib_Sarkar) [19:33:51] (03PS1) 10Andrew Bogott: Further attempt to get a loadbalancer list from Octavia [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147052 [19:34:27] (03CR) 10Andrew Bogott: [C:03+2] Further attempt to get a loadbalancer list from Octavia [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147052 (owner: 10Andrew Bogott) [19:35:10] (03Merged) 10jenkins-bot: Further attempt to get a loadbalancer list from Octavia [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1147052 (owner: 10Andrew Bogott) [20:17:46] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [20:29:34] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [20:59:14] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [21:35:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:45:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:42:57] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10831672 (10Jhancock.wm)