[00:11:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.86% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:15:29] <wmcs-alerts>	 FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:20:29] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:22:13] <wikibugs>	 (03CR) 10Afimaame: "The update func looks good to me." [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 (https://phabricator.wikimedia.org/T367397) (owner: 10Sebastian Berlin (WMSE))
[00:23:29] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[00:24:47] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[00:35:33] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209)
[00:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[00:40:43] <wikibugs>	 (03open) 10raymond-ndibe: ToolforgeClient: add patch method [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209)
[00:42:33] <wikibugs>	 (03update) 10raymond-ndibe: ToolforgeClient: add patch method [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209)
[00:43:30] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld]: add patch method to ToolforgeClient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209)
[00:44:29] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209)
[00:44:47] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209)
[00:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[00:46:29] <wikibugs>	 10Toolforge, 07Kubernetes: toolforge-jobs and packbuild images - https://phabricator.wikimedia.org/T369786#10009178 (10Hawkeye7) Yes.   The jobs.yaml file now reads: `# autocheck2 - name: autocheck2   command: launcher web -ff -n=1000   image: tool-milhistbot/autocheck:latest   schedule: "45 12 * * *"   emails...
[00:46:56] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.67% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:47:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.74% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:57:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.03% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[01:00:33] <wikibugs>	 (03CR) 10Ketulucas: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 (https://phabricator.wikimedia.org/T367397) (owner: 10Sebastian Berlin (WMSE))
[01:41:58] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#10009190 (10Andrew)
[02:23:02] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 05Goal: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291#10009227 (10Andrew) 05Open→03Resolved All Trove instances (well, really, all but one which is getting upgraded next week) are now updated, and ba...
[02:23:11] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Data-Services, 05Goal: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291#10009230 (10Andrew)
[02:34:01] <wikibugs>	 (03open) 10musikanimal: Add MobileFrontend [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/4
[02:36:45] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [Cloud VPS] Trove dbs do not restart after a hypervisor restart - https://phabricator.wikimedia.org/T329949#10009233 (10Andrew) 05Open→03Resolved a:03Andrew This seems to have been fixed by upstream (although not the way I was going to fix it.) DB instances briefly...
[02:56:36] <wikibugs>	 (03update) 10musikanimal: Add MobileFrontend [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/4 (https://phabricator.wikimedia.org/T370839)
[03:00:27] <wikibugs>	 (03merge) 10musikanimal: Add MobileFrontend [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/4 (https://phabricator.wikimedia.org/T370839)
[04:14:48] <wikibugs>	 (03CR) 10Ketulucas: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 (https://phabricator.wikimedia.org/T367397) (owner: 10Sebastian Berlin (WMSE))
[04:34:23] <icinga-wm>	 PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static
[04:36:19] <icinga-wm>	 RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29687 bytes in 5.926 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static
[04:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:59:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:41:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[05:46:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[06:25:23] <wikibugs>	 (03open) 10sstefanova: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[06:31:34] <wikibugs>	 (03update) 10sstefanova: ingress-admission: bump to 0.0.47-20240723124431-37ffef74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/445 (https://phabricator.wikimedia.org/T370046) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[06:45:03] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
[06:45:12] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
[06:49:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:56:40] <wikibugs>	 (03update) 10sstefanova: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[06:57:27] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
[06:59:57] <wmcs-alerts>	 FIRING: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown
[07:07:14] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
[07:14:02] <wikibugs>	 (03update) 10dsavuljesku: Draft: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2
[07:29:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:39:57] <wmcs-alerts>	 RESOLVED: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown
[07:48:56] <wmcs-alerts>	 FIRING: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown
[08:03:33] <wikibugs>	 (03update) 10dsavuljesku: Draft: Bugfixing [toolforge-repos/cr-grants-team-metasync] - 10https://gitlab.wikimedia.org/toolforge-repos/cr-grants-team-metasync/-/merge_requests/2
[08:08:56] <wmcs-alerts>	 RESOLVED: HarborComponentDown: No data about Harbor components found. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown
[08:11:26] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
[08:11:35] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
[08:18:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:28:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:36:44] <wikibugs>	 (03update) 10sstefanova: ingress-admission: bump to 0.0.47-20240723124431-37ffef74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/445 (https://phabricator.wikimedia.org/T370046) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[08:37:08] <wikibugs>	 (03approved) 10sstefanova: ingress-admission: bump to 0.0.47-20240723124431-37ffef74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/445 (https://phabricator.wikimedia.org/T370046) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[08:37:13] <wikibugs>	 (03merge) 10sstefanova: ingress-admission: bump to 0.0.47-20240723124431-37ffef74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/445 (https://phabricator.wikimedia.org/T370046) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[08:41:32] <wikibugs>	 (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046)
[08:46:38] <wikibugs>	 (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046)
[08:47:12] <wikibugs>	 (03update) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046)
[08:58:27] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
[08:58:38] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
[09:21:01] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
[09:21:13] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
[09:29:33] <wikibugs>	 (03merge) 10sstefanova: ingress-nginx: bump version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/433 (https://phabricator.wikimedia.org/T370046)
[09:43:07] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10009572 (10Slst2020)
[10:30:31] <wmcs-alerts>	 FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-3 is lagging behind the primary, the current lag is 122693 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh
[10:42:33] <wikibugs>	 (03open) 10aborrero: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:42:38] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:42:45] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:43:33] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:43:34] <wikibugs>	 (03update) 10aborrero: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:43:42] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:44:11] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:44:13] <wikibugs>	 (03update) 10aborrero: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:44:31] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:44:39] <wikibugs>	 (03open) 10sstefanova: tekton-pipelines: factor out tekton version [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/57
[10:45:11] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:45:12] <wikibugs>	 (03update) 10aborrero: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:45:37] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[10:46:38] <wikibugs>	 (03update) 10aborrero: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:46:48] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: put all networking data in the same yaml file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24 (https://phabricator.wikimedia.org/T370037)
[10:48:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[10:54:18] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[10:57:57] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: toolforge: upgrade all Kubernetes components to versions supporting Kubernetes 1.26 - https://phabricator.wikimedia.org/T370046#10009995 (10Slst2020) We might want to upgrade Tekton independently rather than having this k8s upgr...
[11:39:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[11:44:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[12:19:09] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[12:25:58] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[12:34:43] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[12:50:59] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10010264 (10dcaro) @tstarling I have a patch that implements the fix mentioned in the link above, do you have an image I can play with/test to make...
[12:53:25] <wikibugs>	 10Toolforge (Toolforge iteration 13): [builds-builder, builds-api] upgrade tekton version - https://phabricator.wikimedia.org/T370869 (10Slst2020) 03NEW
[12:53:29] <wikibugs>	 10Toolforge (Toolforge iteration 13): [builds-builder, builds-api] upgrade tekton version - https://phabricator.wikimedia.org/T370869#10010279 (10Slst2020) a:03Slst2020
[12:55:08] <wikibugs>	 (03update) 10sstefanova: tekton-pipelines: factor out tekton version [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/57 (https://phabricator.wikimedia.org/T370869)
[13:02:50] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[13:05:02] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10010305 (10tstarling) >>! In T370610#10010264, @dcaro wrote: > @tstarling I have a patch that implements the fix mentioned in the link above, do yo...
[13:05:20] <wikibugs>	 (03unapproved) 10aborrero: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046) (owner: 10sstefanova)
[13:12:46] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[13:17:17] <wikibugs>	 10Toolforge (Toolforge iteration 13): [harbor] 2024-07-24 Tools harbor db out of space - https://phabricator.wikimedia.org/T370843#10010370 (10dcaro)
[13:17:21] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: Trove does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728#10010371 (10dcaro)
[13:17:36] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: [trove] does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728#10010382 (10dcaro)
[13:21:47] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10010400 (10dcaro) Hmm, first try using the fix mentioned above does not seem to change much: ` heroku@c73de7ed0d69:/workspace$ echo $MAGICK_CONFIGU...
[13:22:58] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10010403 (10dcaro)  Wait, it does only if you overwrite it: ` heroku@c73de7ed0d69:~/.magick$ cat <<EOP >policy.xml <policymap>   <policy domain="del...
[13:23:39] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[13:27:50] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[13:33:58] <wikibugs>	 (03update) 10sstefanova: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[13:34:14] <wikibugs>	 (03update) 10sstefanova: tekton-pipelines: factor out tekton version [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/57 (https://phabricator.wikimedia.org/T370869)
[13:34:23] <wikibugs>	 (03close) 10sstefanova: tekton-pipelines: factor out tekton version [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/57 (https://phabricator.wikimedia.org/T370869)
[13:36:58] <wikibugs>	 10Toolforge, 13Patch-For-Review: Toolforge build service: Can't process an image larger than 128 Mpx using ImageMagick - https://phabricator.wikimedia.org/T370610#10010516 (10dcaro) I don't seem to find a way for it not to use the default one: ` heroku@c73de7ed0d69:~/.magick$ identify -list policy | grep ^Path...
[13:37:13] <wikibugs>	 (03approved) 10dcaro: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449 (owner: 10sstefanova)
[13:38:48] <wikibugs>	 (03update) 10sstefanova: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[13:38:58] <wikibugs>	 (03merge) 10sstefanova: functional-tests/webservice: fix webservice test [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/449
[13:43:08] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[13:49:28] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[13:55:08] <wikibugs>	 (03update) 10sstefanova: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046)
[14:05:52] <wikibugs>	 (03approved) 10aborrero: wmcs-k8s-metrics: bump kube-state-metrics version [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/434 (https://phabricator.wikimedia.org/T370046) (owner: 10sstefanova)
[14:10:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[14:11:09] <wikibugs>	 06cloud-services-team: Unexpected pages during rabbitmq outage - https://phabricator.wikimedia.org/T324208#10010687 (10Andrew) 05Open→03Declined Closing this pending future confusion :)
[14:13:35] <wikibugs>	 06cloud-services-team, 10Striker: 'email maintainers' link in striker - https://phabricator.wikimedia.org/T324163#10010713 (10aborrero) 05Open→03Declined
[14:14:12] <wikibugs>	 10Tools: 'cronout' dir in tb-dev - https://phabricator.wikimedia.org/T324172#10010715 (10Andrew) a:05Tb→03Andrew
[14:14:49] <wikibugs>	 10Tools: 'cronout' dir in tb-dev - https://phabricator.wikimedia.org/T324172#10010700 (10joanna_borun) p:05Triage→03Low a:03Tb
[14:15:19] <wikibugs>	 10Tools: 'cronout' dir in tb-dev - https://phabricator.wikimedia.org/T324172#10010696 (10Andrew) a:05Tb→03None Hello, @Tb, do you have an answer for this?
[14:15:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[14:15:27] <wikibugs>	 10Tool-toolviews: Statistics from toolviews are erratic for Scholia - https://phabricator.wikimedia.org/T320533#10010717 (10joanna_borun)
[14:16:00] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Support managing Cloud VPS project membership via Terraform - https://phabricator.wikimedia.org/T320750#10010719 (10joanna_borun) p:05Triage→03Low
[14:16:21] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Support managing Cloud VPS project membership via Terraform - https://phabricator.wikimedia.org/T320750#10010720 (10aborrero)
[14:16:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#10010721 (10aborrero)
[14:17:58] <wikibugs>	 10Toolforge, 13Patch-For-Review: [toolforge.infra] create fullstack tests - https://phabricator.wikimedia.org/T357977#10010727 (10dcaro)
[14:18:40] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): check toolforge alerting config to ensure total failure to start any new k8s pods does not go unnoticed - https://phabricator.wikimedia.org/T320480#10010725 (10dcaro) →14Duplicate dup:03T357977
[14:22:06] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: figure out deterministic way to tell if a rabbitmq cluster is paritioned - https://phabricator.wikimedia.org/T320266#10010748 (10fnegri)
[14:22:07] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: figure out deterministic way to tell if a rabbitmq cluster is paritioned - https://phabricator.wikimedia.org/T320266#10010749 (10joanna_borun) p:05Triage→03Low
[14:22:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Give openstack services their own rabbitmq users - https://phabricator.wikimedia.org/T320256#10010762 (10joanna_borun) p:05Triage→03Low
[14:23:13] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Give openstack services their own rabbitmq users - https://phabricator.wikimedia.org/T320256#10010764 (10Andrew) a:03Andrew I'll see if I can wrap this up
[14:24:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Technical-Debt: Clean up osm_host from firewall rules and puppet manifests - https://phabricator.wikimedia.org/T309201#10010771 (10joanna_borun) p:05Triage→03Low
[14:24:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Technical-Debt: Clean up osm_host from firewall rules and puppet manifests - https://phabricator.wikimedia.org/T309201#10010776 (10joanna_borun) 05Open→03Resolved
[14:25:44] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: Scrape Kubernetes controller-manager and apiserver metrics into Prometheus - https://phabricator.wikimedia.org/T308381#10010779 (10joanna_borun) p:05Triage→03Low
[14:26:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07LDAP, 07Technical-Debt: Remove obsolete LDAP schemas - https://phabricator.wikimedia.org/T306623#10010780 (10joanna_borun) p:05Triage→03Low
[14:28:41] <wikibugs>	 06cloud-services-team: Forward ceph/octopus packages to Bullseye repo - https://phabricator.wikimedia.org/T306565#10010786 (10fnegri) Upgrade to Ceph v16 is tracked in {T306820}
[14:28:43] <wikibugs>	 06cloud-services-team: Forward ceph/octopus packages to Bullseye repo - https://phabricator.wikimedia.org/T306565#10010788 (10Andrew) 05Open→03Invalid Seems moot at this point, we're running cloud-vps on Bookworm now
[14:30:07] <wikibugs>	 06cloud-services-team: consider eliminating labweb/cloudweb hardware servers - https://phabricator.wikimedia.org/T305233#10010796 (10joanna_borun) p:05Triage→03Low
[14:30:55] <wikibugs>	 06cloud-services-team: cloudvirt puppet config fails on new install - https://phabricator.wikimedia.org/T304038#10010800 (10joanna_borun) 05Open→03Resolved
[14:31:15] <wikibugs>	 06cloud-services-team: cloudmetrics: rsync sometimes fails - https://phabricator.wikimedia.org/T300138#10010808 (10joanna_borun) 05Open→03Invalid
[14:34:20] <wikibugs>	 06cloud-services-team: [NFS] Test the share migration process to an NFS VM - https://phabricator.wikimedia.org/T293801#10010820 (10Andrew) 05Open→03Resolved a:03Andrew Long since done and migrated
[14:34:56] <jinxer-wm>	 FIRING: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[14:36:42] <wikibugs>	 06cloud-services-team, 10Horizon, 13Patch-For-Review: Horizon should use a proxy to access cloud vps hosted apis - https://phabricator.wikimedia.org/T305453#10010840 (10aborrero) p:05Triage→03Low
[14:37:22] <wikibugs>	 06cloud-services-team, 10Horizon, 13Patch-For-Review: Horizon should use a proxy to access cloud vps hosted apis - https://phabricator.wikimedia.org/T305453#10010845 (10aborrero) p:05Low→03Triage
[14:38:52] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Epic: [Epic] Toolforge community hackathon for community issues - https://phabricator.wikimedia.org/T194335#10010856 (10joanna_borun) 05Open→03Declined
[14:39:29] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[14:39:54] <wikibugs>	 06cloud-services-team, 10Toolforge, 10urbanecmbot: urbanecmbot's continuous jobs are getting restarted too frequently - https://phabricator.wikimedia.org/T294228#10010862 (10joanna_borun) 05Open→03Invalid
[14:39:56] <jinxer-wm>	 RESOLVED: SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[14:41:30] <wikibugs>	 06cloud-services-team, 10Toolforge: [NFS] Add monitoring and alerting to the new NFS system - https://phabricator.wikimedia.org/T293804#10010870 (10fnegri)
[14:42:00] <wikibugs>	 06cloud-services-team, 10Toolforge, 10urbanecmbot: urbanecmbot's continuous jobs are getting restarted too frequently - https://phabricator.wikimedia.org/T294228#10010864 (10dcaro) Hi! We don't have a grid anymore, please open a new one if it happens on k8s
[14:42:01] <wikibugs>	 06cloud-services-team, 10Toolforge: [NFS] Add monitoring and alerting to the new NFS system - https://phabricator.wikimedia.org/T293804#10010871 (10joanna_borun) a:03dcaro
[14:43:27] <wikibugs>	 06cloud-services-team: Audit & standardize our use of ldap on metal servers - https://phabricator.wikimedia.org/T292208#10010872 (10joanna_borun) 05Open→03Resolved
[14:44:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[14:44:30] <wikibugs>	 06cloud-services-team: monitor ldap functionality from within cloud-vps - https://phabricator.wikimedia.org/T292205#10010874 (10joanna_borun) p:05Triage→03Medium
[14:46:02] <wikibugs>	 06cloud-services-team, 10Toolforge: Track package versions in Toolforge images - https://phabricator.wikimedia.org/T291908#10010890 (10joanna_borun) p:05Triage→03Low
[14:46:40] <wikibugs>	 06cloud-services-team, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Seen): Intermittent package download failures during jenkins/pipeline tests - https://phabricator.wikimedia.org/T291135#10010891 (10joanna_borun) 05Open→03Invalid
[14:47:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Review paging setup for WMCS with onboarding in alertmanager - https://phabricator.wikimedia.org/T289364#10010894 (10joanna_borun) 05Open→03Resolved
[14:47:45] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: TLS for Cloud VPS MariaDB (Galera) - https://phabricator.wikimedia.org/T297269#10010902 (10joanna_borun) p:05Triage→03Medium
[14:48:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Sustainability (Incident Followup): toolforge: Set up alerting based on Kubernetes API response times - https://phabricator.wikimedia.org/T296596#10010905 (10joanna_borun) p:05Triage→03Medium
[14:48:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:49:28] <wikibugs>	 06cloud-services-team, 10Tool-openstack-browser: Trove quota usage not available from the API - https://phabricator.wikimedia.org/T286128#10010912 (10joanna_borun) a:03Andrew
[14:49:46] <wikibugs>	 06cloud-services-team, 10Data-Services: toolsdb: What happens at Midnight? - https://phabricator.wikimedia.org/T324629#10010898 (10fnegri) 05Open→03Invalid p:05Triage→03Low This is not happening anymore.  Last 24 hours:  {F56646071}  Last 7 days:  {F56646073}
[14:49:59] <wikibugs>	 06cloud-services-team, 10Data-Services, 07Epic: [toolsdb] convert myisam tables to innodb - https://phabricator.wikimedia.org/T306455#10010920 (10fnegri)
[14:50:33] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Upstream: [trove] does not expose amount of disk space used - https://phabricator.wikimedia.org/T354728#10010927 (10dcaro) It's quite possible that it just can't call that endpoint {T286128}
[14:52:00] <wikibugs>	 06cloud-services-team, 10Tool-openstack-browser: Trove quota usage not available from the API - https://phabricator.wikimedia.org/T286128#10010924 (10aborrero) See also {T354728}
[14:52:22] <wikibugs>	 06cloud-services-team, 10Data-Services, 07Epic: toolsdb: convert myisam tables to innodb - https://phabricator.wikimedia.org/T306455#10010916 (10fnegri) p:05Triage→03Medium
[14:53:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: TLS for Cloud VPS RabbitMQ - https://phabricator.wikimedia.org/T297268#10010947 (10joanna_borun) p:05Triage→03Low
[14:53:36] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Support multi-domain OpenStack - https://phabricator.wikimedia.org/T321349#10010950 (10Andrew) 05Open→03Resolved a:03Andrew
[14:55:02] <wikibugs>	 06cloud-services-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#10010954 (10joanna_borun) p:05Triage→03Low a:03Andrew
[14:55:11] <wikibugs>	 06cloud-services-team, 07Documentation: Update operational documentation for NFS and wikireplicas - https://phabricator.wikimedia.org/T291092#10010958 (10fnegri) 05Open→03Invalid Wiki Replicas docs improvements are tracked in this newer task: {T365717}  NFS implementation has changed since this task, I...
[14:57:09] <wikibugs>	 06cloud-services-team, 10Data-Services, 10VPS-Projects: Request to /data/project(/statanalyser) - https://phabricator.wikimedia.org/T326904#10010965 (10Andrew) a:03Andrew Hi! I'm very sorry this got missed; would you still like /scratch added to your cloud-vps project?  Or did that get done already?
[15:02:20] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Cloud VPS: refresh openstack resources grafana dashboard - https://phabricator.wikimedia.org/T333975#10010985 (10aborrero) 05In progress→03Resolved a:03aborrero
[15:03:37] <wikibugs>	 10Toolforge (Toolforge iteration 13): [jobs-cli] enforce proper validation for load jobs before calculate_changes - https://phabricator.wikimedia.org/T366211#10010994 (10dcaro) p:05Triage→03Medium
[15:03:44] <wikibugs>	 10Toolforge (Toolforge iteration 13), 13Patch-For-Review: [jobs-api] move jobs load feature to the backend - https://phabricator.wikimedia.org/T366209#10010995 (10dcaro) p:05Triage→03Medium
[15:03:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.backend] Create worker and connect to redis - https://phabricator.wikimedia.org/T370321#10010998 (10dcaro) p:05Triage→03High
[15:03:58] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 13): [sct.backend] Create trove database - https://phabricator.wikimedia.org/T370317#10010999 (10dcaro) p:05Triage→03High
[15:04:07] <wikibugs>	 10Toolforge (Toolforge iteration 13): [cli] the generic cli swallows the `--` from other commands - https://phabricator.wikimedia.org/T370184#10011000 (10dcaro) p:05Triage→03Low
[15:04:15] <wikibugs>	 10Toolforge (Toolforge iteration 13): [lima-kilo] add ingress-admission - https://phabricator.wikimedia.org/T370774#10011001 (10dcaro) p:05Triage→03Medium
[15:04:23] <wikibugs>	 10Toolforge (Toolforge iteration 13): [harbor] Investigate how to deactivate wal from trove for postrges databasese - https://phabricator.wikimedia.org/T370845#10011002 (10dcaro) p:05Triage→03Medium
[15:04:30] <wikibugs>	 10Toolforge (Toolforge iteration 13): [builds-builder, builds-api] upgrade tekton version - https://phabricator.wikimedia.org/T370869#10011003 (10dcaro) p:05Triage→03High
[15:07:06] <wikibugs>	 06cloud-services-team: openstack magnum api response times increase - https://phabricator.wikimedia.org/T344172#10011008 (10dcaro) 05Open→03Invalid This is not valid anymore:  {F56646763}  I'm guessing it just needed more usage
[15:08:36] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: OpenStack API response time gets slower over time - https://phabricator.wikimedia.org/T345084#10011012 (10dcaro) 05Open→03Resolved a:03dcaro I think we can close this for now, we tweaked a few of the api response times, and currently it has not...
[15:12:09] <wikibugs>	 06cloud-services-team, 10Toolforge, 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Priority Backlog 📥): Move Toolforge PipelineLib repositories to GitLab - https://phabricator.wikimedia.org/T334399#10011021 (10dcaro) 05Open→03Resolved a:03dcaro This is done
[15:17:48] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI, 13Patch-For-Review: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#10011050 (10jhathaway) @Scott_French I believe the issue with pcc is now fixed. Please let me know if you c...
[15:18:28] <wikibugs>	 (03update) 10dcaro: builds: override default imagemagick config [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/56 (https://phabricator.wikimedia.org/T370610)
[15:28:29] <wikibugs>	 06cloud-services-team, 10Data-Services: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-07-23 - https://phabricator.wikimedia.org/T370760#10011092 (10fnegri) p:05Triage→03High Replication is still stuck processing the same DELETE transaction. From past experience, it usually takes a few days to comple...
[15:28:35] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:28:39] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:28:51] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:29:35] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:29:38] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:29:44] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:31:27] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:31:27] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:31:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:34:23] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:34:23] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:34:32] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:37:26] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:37:30] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:37:47] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:38:57] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:38:59] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:39:24] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:43:15] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:43:16] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:43:51] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:44:55] <wikibugs>	 (03approved) 10dcaro: [jobs-api] refactor before moving jobs load to backend [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/103 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe)
[15:45:47] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:45:49] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:46:11] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:49:01] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:49:06] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:49:27] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:56:27] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:56:27] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:56:57] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:57:28] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:57:30] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:57:50] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[15:59:36] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[15:59:36] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[15:59:54] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[16:01:17] <wikibugs>	 (03update) 10aborrero: Draft: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[16:01:22] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[16:01:23] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
[16:01:26] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[16:01:43] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
[16:04:15] <wikibugs>	 (03update) 10aborrero: tofu-infra: define neutron routers [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23 (https://phabricator.wikimedia.org/T370037)
[16:04:50] <wikibugs>	 06cloud-services-team, 10Data-Services: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-07-23 - https://phabricator.wikimedia.org/T370760#10011275 (10fnegri) Update: I think I found a way to estimate how long the transaction will take, thanks to [this brilliant StackOverflow post](https://stackoverflow.co...
[16:09:45] <wikibugs>	 (03approved) 10dcaro: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe)
[16:09:58] <wikibugs>	 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-07-23 - https://phabricator.wikimedia.org/T370760#10011308 (10fnegri)
[16:11:12] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: [toolsdb] Migrate mixnmatch db to Trove - https://phabricator.wikimedia.org/T350862#10011314 (10fnegri)
[16:11:14] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 05Goal: [toolsdb] Migrate quickstatements db to Trove - https://phabricator.wikimedia.org/T369177#10011312 (10fnegri)
[16:14:58] <wikibugs>	 06cloud-services-team, 10Data-Services, 07Epic: Migrate largest ToolsDB users to Trove - https://phabricator.wikimedia.org/T291782#10011329 (10fnegri) This is an #epic  that will require multiple quarters. I'm removing the milestone #wmcs-fy2024-2025-q1-q2 from this task, and adding that milestone to two sub...
[16:15:05] <wikibugs>	 06cloud-services-team, 10Data-Services, 07Epic: Migrate largest ToolsDB users to Trove - https://phabricator.wikimedia.org/T291782#10011333 (10fnegri)
[16:24:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#10011405 (10Southparkfan) 05Open→03Re...
[16:25:54] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#10011411 (10Southparkfan)
[16:57:38] <wikibugs>	 (03approved) 10dcaro: [toolforge-weld]: add patch method to ToolforgeClient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe)
[16:57:48] <wikibugs>	 (03update) 10dcaro: [toolforge-weld]: add patch method to ToolforgeClient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/49 (https://phabricator.wikimedia.org/T366209) (owner: 10raymond-ndibe)
[17:20:57] <wikibugs>	 10Toolforge: [toolforge-weld] support back python 3.7 - https://phabricator.wikimedia.org/T370932 (10dcaro) 03NEW
[17:21:41] <wikibugs>	 10Toolforge (Toolforge iteration 13): [toolforge-weld] support back python 3.7 - https://phabricator.wikimedia.org/T370932#10011957 (10dcaro) p:05Triage→03High
[17:22:18] <wikibugs>	 10Toolforge (Toolforge iteration 13): [toolforge-weld] support back python 3.7 - https://phabricator.wikimedia.org/T370932#10011952 (10dcaro) a:05Raymond→03Raymond_Ndibe
[17:22:32] <wikibugs>	 10Toolforge (Toolforge iteration 13): [toolforge-weld] support back python 3.7 - https://phabricator.wikimedia.org/T370932#10011956 (10dcaro)
[17:41:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[17:46:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[17:58:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:31:59] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 06Infrastructure-Foundations, 10Puppet CI: Cloud VPS "puppet-diffs" project Buster deprecation - https://phabricator.wikimedia.org/T367547#10012271 (10Dzahn) I am running into unexpected compilation failures (before and after a change I am trying to test) (that are no...
[18:46:43] <wikibugs>	 10VPS-project-Wikistats: Add cswikivoyage to wikistats - https://phabricator.wikimedia.org/T370917#10012388 (10Dzahn) 05Open→03In progress ` MariaDB [wikistats]> insert into wikivoyage (lang,loclang,prefix,method) values ("Czech","&#268;e&#353;tina","cs",8);  `
[19:10:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[19:15:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[19:38:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[19:43:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[19:50:58] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[19:51:24] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move jobs load to backend [repos/cloud/toolforge/jobs-api] (more_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T366209)
[19:57:18] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48
[19:57:55] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48
[20:23:14] <wikibugs>	 10Tool-replag: Report on tools-readonly.db.svc.wikimedia.cloud replag - https://phabricator.wikimedia.org/T370957 (10bd808) 03NEW
[20:27:41] <wikibugs>	 10Tool-replag: Report on tools-readonly.db.svc.wikimedia.cloud replag - https://phabricator.wikimedia.org/T370957#10012768 (10bd808) ` $ mariadb --defaults-file=$HOME/replica.my.cnf --host=tools-readonly.db.svc.wikimedia.cloud Welcome to the MariaDB monitor.  Commands end with ; or \g. Your MariaDB connection id...
[21:13:42] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10video2commons: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711#10012891 (10Don-vip) @Andrew in fact I need help. I see the old encoding instances use the `role::labs::lvm::srv` puppe...
[21:15:42] <wikibugs>	 10Toolforge, 06Privacy Engineering: Add Content-Security-Policy header enforcing 3rd party web interaction restrictions to proxy responses - https://phabricator.wikimedia.org/T130748#10012902 (10bd808) I finally pushed a [[https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/T...
[21:18:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:36:59] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10video2commons: Replace or remove Debian Buster VMs in 'video' cloud-vps project - https://phabricator.wikimedia.org/T360711#10012943 (10Don-vip) Update: OK I understand that NFS and role::labs::lvm::srv are not related:   ` encoding04:~$ df -h...
[21:42:57] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add 360Gb storage to video project quota - https://phabricator.wikimedia.org/T370964 (10Don-vip) 03NEW
[21:55:18] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Community-Tech (Darwin's Fox (July 15-26, 2024)): Cloud VPS "eventmetrics" project Buster deprecation - https://phabricator.wikimedia.org/T367530#10012979 (10MusikAnimal) 05Open→03Resolved The Debian Buster box is no more/
[22:38:46] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add 360Gb storage to video project quota - https://phabricator.wikimedia.org/T370964#10013171 (10Andrew) 05Open→03Resolved a:03Andrew done. thanks for doing this work!
[22:40:23] <jinxer-wm>	 FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[22:42:18] <wikibugs>	 10Cloud-VPS (Quota-requests), 10video2commons: Request: add 360Gb storage to video project quota - https://phabricator.wikimedia.org/T370964#10013189 (10Don-vip) Thank you Andrew!
[22:45:23] <jinxer-wm>	 RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM
[23:15:49] <wikibugs>	 10Data-Services, 06DBA: Prepare and check storage layer for cswikivoyage - https://phabricator.wikimedia.org/T370912#10013270 (10Zabe) wiki has been created
[23:39:23] <wikibugs>	 10VPS-project-Wikistats: Add cswikivoyage to wikistats - https://phabricator.wikimedia.org/T370917#10013297 (10Dzahn) 05In progress→03Resolved a:03Dzahn ` dzahn@wikistats-bookworm:~$ /usr/lib/wikistats/update.php wy prefix cs sent query: 'select * from wikivoyage where prefix="cs"'. A(1/26) - cs.wikivo...