[00:05:39] <wikibugs>	 10Tool-extjsonuploader: Validate Composer names against Packagist - https://phabricator.wikimedia.org/T370729#10013381 (10Samwilson) 05Open→03Resolved a:03Samwilson I've updated the tool, so its next run should remove the invalid composer names from Module:ExtensionJson.
[00:16:39] <wikibugs>	 10Tool-extjsonuploader: Validate Composer names against Packagist - https://phabricator.wikimedia.org/T370729#10013388 (10Samwilson) 05Resolved→03Open No, well that didn't go according to plan! https://www.mediawiki.org/w/index.php?title=Module%3AExtensionJson&diff=6666565&oldid=6666378  It's removed them all!
[00:21:22] <wikibugs>	 (03CR) 10Samwilson: Check for composer name before adding it to the output (031 comment) [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1056050 (https://phabricator.wikimedia.org/T370729) (owner: 10Samwilson)
[00:26:22] <wikibugs>	 (03PS1) 10Samwilson: Actually run the composer show command [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1056629 (https://phabricator.wikimedia.org/T370729)
[01:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:59:10] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[02:14:12] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[02:48:48] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10013569 (10tstarling)
[02:51:53] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10013570 (10tstarling) >>! In T370610#10010516, @dcaro wrote: > Or to tell it that the size limit is unlimited (it just sets it to 0), @tstarling an...
[02:56:05] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10013571 (10tstarling)
[03:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[03:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[04:37:23] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209)
[04:40:14] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209)
[04:51:43] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209)
[04:52:20] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209)
[04:53:08] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209)
[05:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:43:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-19 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[06:10:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:15:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:31:51] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10013672 (10Slst2020)
[06:32:38] <wikibugs>	 06cloud-services-team, 10Data-Services, 10VPS-Projects: Request to /data/project(/statanalyser) - https://phabricator.wikimedia.org/T326904#10013673 (10Leaderboard) Hi if possible that would be nice.
[06:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:48:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-19 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[06:55:32] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.backend] Transform the "/" API reply to json - https://phabricator.wikimedia.org/T370323#10013680 (10dcaro) 05Open→03In progress
[07:01:38] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.frontend] Show the backend status - https://phabricator.wikimedia.org/T370324#10013683 (10dcaro) 05Open→03In progress
[07:03:11] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.backend] Transform the "/" API reply to json - https://phabricator.wikimedia.org/T370323#10013687 (10dcaro) Done:  ` tools.sample-complex-app@tools-bastion-13:~$ toolforge jobs run --command "curl http://backend-api:8080/" --...
[07:05:21] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.backend] Transform the "/" API reply to json - https://phabricator.wikimedia.org/T370323#10013691 (10dcaro) 05In progress→03Resolved
[07:09:06] <wikibugs>	 (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069)
[07:14:11] <wikibugs>	 10Horizon: Horizon does not commit changes to cloud/instance-puppet git repo since June 24th 2024 - https://phabricator.wikimedia.org/T370136#10013699 (10hashar) Thanks!
[07:16:04] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[07:16:07] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[07:25:13] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[07:58:03] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[07:58:15] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[07:58:53] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:01:01] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:01:02] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[08:01:33] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:04:47] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
[08:05:00] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
[08:24:26] <wikibugs>	 (03open) 10aborrero: tofu-infra: refactor imports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25 (https://phabricator.wikimedia.org/T370037)
[08:25:02] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:25:08] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:27:25] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:27:27] <wikibugs>	 (03update) 10aborrero: tofu-infra: refactor imports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25 (https://phabricator.wikimedia.org/T370037)
[08:27:35] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:28:45] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:28:47] <wikibugs>	 (03update) 10aborrero: tofu-infra: refactor imports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25 (https://phabricator.wikimedia.org/T370037)
[08:29:06] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
[08:37:29] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
[08:37:42] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
[08:41:54] <wikibugs>	 (03approved) 10fnegri: tofu-infra: refactor imports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25 (https://phabricator.wikimedia.org/T370037) (owner: 10aborrero)
[08:42:15] <wikibugs>	 (03merge) 10aborrero: tofu-infra: refactor imports [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25 (https://phabricator.wikimedia.org/T370037)
[08:42:35] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[08:42:58] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[08:43:40] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services: Cloud VPS "packaging" project Buster deprecation - https://phabricator.wikimedia.org/T367544#10013825 (10hashar) //I am copy pasting below a comment I have made about `python-all` on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1049...
[08:48:11] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[08:50:43] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-07-23 - https://phabricator.wikimedia.org/T370760#10013832 (10fnegri) Unfortunately there was one wrong assumption in my calculations above: the number of `row lock(s)` reported by `SHOW ENGINE INNO...
[08:50:49] <wikibugs>	 (03merge) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[08:51:19] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[08:52:07] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:52:10] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[08:52:26] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:56:58] <wikibugs>	 (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069)
[08:57:19] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[08:57:31] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[08:57:53] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[09:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:30:43] <wikibugs>	 (03approved) 10fnegri: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037) (owner: 10aborrero)
[09:32:14] <wikibugs>	 (03merge) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[09:32:20] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[09:35:52] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[09:44:35] <wikibugs>	 (03update) 10dcaro: components: add components-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/173 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[09:51:29] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#10013983 (10kostajh) It's worth noting that even after temporary accounts are deployed to all wikis, we'll continue to have I...
[10:40:38] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge, 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool/<toolname>/deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10014134 (10dcaro)
[10:47:53] <wikibugs>	 (03open) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[10:47:55] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:48:14] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:48:52] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:48:54] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[10:49:07] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:56:47] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[10:56:52] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:57:03] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:57:22] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[10:57:23] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:57:38] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:58:22] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[10:58:26] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[10:58:54] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[11:00:32] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[11:01:05] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[11:01:19] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[11:01:24] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[11:02:11] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
[11:03:06] <wikibugs>	 (03approved) 10fnegri: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037) (owner: 10aborrero)
[11:03:44] <wikibugs>	 (03update) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[11:03:48] <wikibugs>	 (03merge) 10aborrero: data/: add routers for eqiad1 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26 (https://phabricator.wikimedia.org/T370037)
[11:03:51] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[11:04:34] <wikibugs>	 (03close) 10aborrero: Draft: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[11:05:09] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[11:58:16] <wikibugs>	 06cloud-services-team: maintain-dbusers: some accounts don't have access enve thought maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011 (10aborrero) 03NEW
[11:58:54] <wikibugs>	 06cloud-services-team: toolforge: some accounts don't have access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014308 (10aborrero)
[11:59:46] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014313 (10aborrero)
[12:00:22] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014316 (10aborrero) I have manually restarted maintain-dbusers in cloudcontrol1005 but it did not make much of a difference.
[12:01:24] <wikibugs>	 10Toolforge: BibTex Bug of MakeRef - https://phabricator.wikimedia.org/T371012 (10Molind214) 03NEW
[12:01:41] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014329 (10fnegri) I can see the user and grants in ToolsDB:  ` MariaDB [(none)]> SELECT user,host FROM mysql.user WHERE user = 's56035'; +-...
[12:16:35] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[12:18:52] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[12:20:15] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014354 (10fnegri) Connecting to ToolsDB with the password stored in `/data/project/lexica-tool/replica.my.cnf` failed with: ` ERROR 1045 (2...
[12:20:32] <wmcs-alerts>	 FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-3 is lagging behind the primary, the current lag is 215693 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh
[12:21:41] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014359 (10aborrero) The research by @dcaro and @fnegri points in this direction: * a maintain-dbusers run generated a password and a replic...
[12:23:19] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/17
[12:35:18] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014387 (10dcaro) The root cause is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056489  When the envvars api url change was made,...
[12:38:23] <wikibugs>	 (03open) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069)
[12:43:07] <wikibugs>	 06cloud-services-team: toolforge: an account don't have db access even though maintain-dbusers claims it has created everything - https://phabricator.wikimedia.org/T371011#10014396 (10aborrero) 05Open→03Resolved a:03aborrero thanks everyone for assisting with this.
[12:47:10] <wikibugs>	 (03update) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069)
[12:47:16] <wikibugs>	 06cloud-services-team: MetricsinfraAlertmanagerDown - https://phabricator.wikimedia.org/T368394#10014429 (10fnegri) 05Open→03Resolved a:03fnegri
[12:49:24] <wikibugs>	 (03approved) 10raymond-ndibe: [toolforge-weld]: add patch method to ToolforgeClient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209)
[12:49:31] <wikibugs>	 (03merge) 10raymond-ndibe: [toolforge-weld]: add patch method to ToolforgeClient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209)
[12:55:50] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[12:55:51] <wikibugs>	 (03open) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[12:55:57] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[12:56:20] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[12:56:22] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[12:56:26] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:00:56] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:00:57] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:01:05] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:08:23] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:08:23] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:08:29] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:08:48] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:08:49] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:08:54] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:09:49] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:09:49] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:09:58] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:10:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[13:11:42] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:11:44] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:11:53] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:12:27] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:12:29] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:12:35] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:15:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[13:15:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:15:37] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:15:44] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:16:43] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:16:45] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[13:17:05] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[13:18:31] <wikibugs>	 (03approved) 10dcaro: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[13:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:19:18] <wikibugs>	 (03approved) 10dcaro: components: add components-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/173 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[13:24:44] <wikibugs>	 (03approved) 10dcaro: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[13:25:31] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[13:27:38] <wikibugs>	 (03approved) 10dcaro: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[13:27:43] <wikibugs>	 (03update) 10dcaro: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069) (owner: 10sstefanova)
[13:30:17] <wikibugs>	 (03update) 10dcaro: auth: use the header passed by the api gateway [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/106 (https://phabricator.wikimedia.org/T367180)
[13:30:42] <wikibugs>	 (03merge) 10sstefanova: components: add components-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/437 (https://phabricator.wikimedia.org/T362069)
[13:33:21] <wikibugs>	 (03open) 10dcaro: toolforge_get_version: add back wrapper [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/175
[13:35:04] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[13:35:57] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[13:36:37] <wikibugs>	 (03update) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[13:36:44] <wikibugs>	 (03merge) 10sstefanova: initial boilerplate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T362069)
[13:53:25] <wikibugs>	 (03CR) 10Gergő Tisza: [C:03+2] Actually run the composer show command [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1056629 (https://phabricator.wikimedia.org/T370729) (owner: 10Samwilson)
[13:54:16] <wikibugs>	 (03Merged) 10jenkins-bot: Actually run the composer show command [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1056629 (https://phabricator.wikimedia.org/T370729) (owner: 10Samwilson)
[14:04:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948#10014828 (10sguebo_WMF) 05Stalled→03Declined
[14:08:46] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.3-20240725133653-f188d2d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/450
[14:09:02] <wikibugs>	 (03update) 10dcaro: components-api: bump to 0.0.3-20240725133653-f188d2d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/450 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:24:43] <wikibugs>	 (03open) 10dcaro: deployment: make api backends configurable per environment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/29
[14:34:25] <wikibugs>	 (03open) 10dcaro: api-gateway: use the new structure for api_maps config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/451
[14:38:03] <wikibugs>	 (03update) 10dcaro: api-gateway: use the new structure for api_maps config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/451
[14:41:50] <wikibugs>	 (03update) 10sstefanova: deployment: make api backends configurable per environment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/29 (owner: 10dcaro)
[14:41:53] <wikibugs>	 (03approved) 10sstefanova: deployment: make api backends configurable per environment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/29 (owner: 10dcaro)
[14:42:53] <wikibugs>	 (03update) 10sstefanova: api-gateway: use the new structure for api_maps config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/451 (owner: 10dcaro)
[14:42:54] <wikibugs>	 (03approved) 10sstefanova: api-gateway: use the new structure for api_maps config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/451 (owner: 10dcaro)
[14:42:58] <wikibugs>	 (03merge) 10dcaro: api-gateway: use the new structure for api_maps config [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/451
[14:43:30] <wikibugs>	 (03merge) 10dcaro: deployment: make api backends configurable per environment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/29
[14:45:03] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[14:45:04] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[14:45:12] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[14:45:38] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.30-20240725144339-331dbe65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/452
[14:46:10] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[14:46:12] <wikibugs>	 (03update) 10aborrero: routers: consolidate router_interfaces into the same module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27 (https://phabricator.wikimedia.org/T370037)
[14:46:42] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
[14:55:51] <wikibugs>	 10Toolforge (Toolforge iteration 13): [gateway-api] something is caching the openapi docs - https://phabricator.wikimedia.org/T371033 (10Slst2020) 03NEW
[14:56:53] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
[14:57:08] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
[14:57:08] <wikibugs>	 10Tools: BibTex Bug of MakeRef - https://phabricator.wikimedia.org/T371012#10015093 (10JJMC89)
[14:57:45] <wikibugs>	 (03update) 10dcaro: api-gateway: bump to 0.0.30-20240725144339-331dbe65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/452 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:59:51] <wikibugs>	 (03update) 10dcaro: api-gateway: bump to 0.0.30-20240725144339-331dbe65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/452 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:00:30] <wikibugs>	 (03update) 10sstefanova: backends: add components-api [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/28 (https://phabricator.wikimedia.org/T362069)
[15:04:39] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
[15:04:50] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
[15:11:39] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#10015192 (10jhathaway) >>! In T367547#10012271, @Dzahn wrote: > I am running into unexpected compilation failures (before and aft...
[15:19:44] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
[15:19:55] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
[15:21:36] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 13), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10015267 (10dcaro) a:03dcaro
[15:31:31] <wikibugs>	 (03approved) 10dcaro: api-gateway: bump to 0.0.30-20240725144339-331dbe65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/452 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:31:34] <wikibugs>	 (03merge) 10dcaro: api-gateway: bump to 0.0.30-20240725144339-331dbe65 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/452 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[15:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[15:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[16:09:38] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Tools, 07SecTeam Discussion, 07Security, 07Vuln-Infoleak: Database credentials for s51347 (fatg) publicly readable on Toolforge - https://phabricator.wikimedia.org/T326613#10015469 (10fnegri) > The code should also just read/parse from replica.my.cnf inste...
[16:10:37] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Tools, 07SecTeam Discussion, 07Security, 07Vuln-Infoleak: Database credentials for s51347 (fatg) publicly readable on Toolforge - https://phabricator.wikimedia.org/T326613#10015463 (10sbassett)
[16:35:39] <wikibugs>	 06cloud-services-team, 10Toolforge: Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015596 (10dcaro) a:03dcaro
[16:36:00] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015599 (10dcaro) p:05Triage→03High
[16:36:17] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015598 (10dcaro)
[16:36:26] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015603 (10dcaro) 05Open→03In progress
[16:41:18] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015614 (10dcaro) @ForzaGreen Hi! Just created your credentials for elasticsearch, they are available as envvars inside toolforge: ` toolforge envvars...
[16:41:41] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10015617 (10dcaro) 05In progress→03Resolved
[16:45:30] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10015630 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye
[16:57:54] <wikibugs>	 (03update) 10dcaro: toolforge_get_version: add back wrapper [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/175
[17:16:54] <wikibugs>	 (03open) 10dcaro: toolforge_deploy_mr: add toolforge-weld package [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/176
[17:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:19:36] <wikibugs>	 (03open) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453
[17:24:19] <wikibugs>	 (03update) 10dcaro: toolforge_get_versions: show component names [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/453
[17:33:02] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10015750 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye ex...
[17:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[17:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[18:06:03] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10015870 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye
[18:17:04] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add +16 cpu / +16 Gb ram to video project quota - https://phabricator.wikimedia.org/T371047 (10Don-vip) 03NEW
[18:38:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[18:43:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[19:03:58] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10video2commons: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711#10015975 (10Don-vip) @Andrew can I get support from WMCS team, I got something that doesn't work and I don't understand...
[19:12:23] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10016000 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host cloudcephmon1004.eqiad.wmnet with OS bullseye co...
[19:18:46] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/17 (owner: 10l10n-bot)
[19:18:50] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/17 (owner: 10l10n-bot)
[19:21:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465#10016016 (10Gehel) p:...
[19:24:44] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#10016019 (10Papaul) a:05Papaul→03Jclark-ctr @Jclark-ctr  looking at 1004 i realized that com2 was not set  `  System BIOS    System BIOS Settings > Se...
[21:18:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:49:34] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13): Elasticsearch credential request for wikitermbase - https://phabricator.wikimedia.org/T368376#10016456 (10ForzaGreen) Thank you very much @bd808 and @dcaro ! I will move forward with the provided information.  Have a nice day !
[22:40:49] <wikibugs>	 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10016625 (10Dwisehaupt) Migrated to the production VPS today. Hopefully we will be able to start the rebuild of the cloudVPS host ne...
[23:20:14] <wikibugs>	 10Striker: "Set source code repository in toolinfo to this repository" checkbox on repos/create screen is unexpectedly required - https://phabricator.wikimedia.org/T369395#10016686 (10bd808) 05Open→03Resolved a:03bd808
[23:49:58] <wikibugs>	 10Tool-extjsonuploader: Validate Composer names against Packagist - https://phabricator.wikimedia.org/T370729#10016745 (10Samwilson)
[23:50:19] <wikibugs>	 10Tool-extjsonuploader: Validate Composer names against Packagist - https://phabricator.wikimedia.org/T370729#10016744 (10Samwilson) Thanks @Tgr, sorry for the noise.  I've updated the tool, and the next run should update things correctly and remove the wrong composer names. I'm not closing this task this time t...