[00:06:20] <wmcs-alerts>	 FIRING: [2x] PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon
[01:33:28] <wikibugs>	 (03PS1) 10Krinkle: phan: Add missing phan config file, and upgrade Phan [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1150162
[01:34:02] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] phan: Add missing phan config file, and upgrade Phan [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1150162 (owner: 10Krinkle)
[01:34:11] <wikibugs>	 (03PS2) 10AntiCompositeNumber: Switch from /usr/bin/timeout to toolforge-jobs timeout [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1149847
[01:34:35] <wikibugs>	 (03Merged) 10jenkins-bot: phan: Add missing phan config file, and upgrade Phan [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1150162 (owner: 10Krinkle)
[01:35:23] <wikibugs>	 (03PS3) 10Krinkle: Switch from /usr/bin/timeout to toolforge-jobs timeout [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1149847 (owner: 10AntiCompositeNumber)
[01:35:32] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Switch from /usr/bin/timeout to toolforge-jobs timeout [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1149847 (owner: 10AntiCompositeNumber)
[01:36:01] <wikibugs>	 (03Merged) 10jenkins-bot: Switch from /usr/bin/timeout to toolforge-jobs timeout [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1149847 (owner: 10AntiCompositeNumber)
[07:03:49] <wikibugs>	 10Tools: zoomviewer uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T395020#10855672 (10tstarling) It's now 1.3TB. And most of that is from the last 7 days:  ` tools.zoomviewer@tools-bastion-13:~/public_html/cache$ for d in $(seq 15); do echo -n "$d " ; find -mtime "$d" -print0 | du...
[07:59:29] <wikibugs>	 10Quarry: quarry: Set a timeout on Redis operations - https://phabricator.wikimedia.org/T395226 (10taavi) 03NEW
[08:01:46] <wikibugs>	 06cloud-services-team, 10Toolforge: Renew Prometheus K8s cert - https://phabricator.wikimedia.org/T395227 (10taavi) 03NEW p:05Triage→03High
[08:03:43] <wikibugs>	 10Quarry: Limit number of concurrent queries one user can run at a time - https://phabricator.wikimedia.org/T104316#10855770 (10taavi) →14Duplicate dup:03T225869
[08:03:44] <wikibugs>	 10Quarry, 13Patch-Needs-Improvement: Add rate limiting on queries execution - https://phabricator.wikimedia.org/T225869#10855772 (10taavi)
[08:15:08] <wikibugs>	 06cloud-services-team, 10Toolforge: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10855785 (10Aklapper)
[09:04:57] <notefromgithub>	 supertassu opened https://github.com/toolforge/quarry/pull/81
[09:18:46] <wikibugs>	 10Quarry: quarry: Set a timeout on Redis operations - https://phabricator.wikimedia.org/T395226#10856044 (10taavi) p:05Triage→03Medium a:03taavi https://github.com/toolforge/quarry/pull/81
[09:44:34] <wikibugs>	 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237 (10taavi) 03NEW
[09:46:20] <wikibugs>	 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10856126 (10taavi) Sample file in there: `lang=xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Properties xmlns="http://schemas.openxmlformats.org/officeDocument/2006/extended-properties" xmlns:vt="http://schemas.openx...
[09:47:23] <wikibugs>	 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10856129 (10taavi) 05Open→03Resolved Filed subtasks for the issues I've found. Closing.
[09:47:26] <wikibugs>	 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10856131 (10taavi) p:05Triage→03High
[09:49:31] <notefromgithub>	 supertassu closed https://github.com/toolforge/quarry/pull/81
[09:49:42] <wikibugs>	 10Quarry: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238 (10taavi) 03NEW
[09:49:43] <wikibugs>	 10Quarry: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238#10856148 (10taavi) p:05Triage→03High
[09:50:57] <wikibugs>	 10Quarry: quarry: Set a timeout on Redis operations - https://phabricator.wikimedia.org/T395226#10856152 (10taavi) 05Open→03Resolved
[09:55:40] <wikibugs>	 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10856163 (10taavi) a:03taavi
[10:00:03] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856165 (10fnegri) > You can specify the table instead, ores_model and ores_classification  Yes good point. I would still like to do a full run to make sure ever...
[10:00:29] <notefromgithub>	 supertassu opened https://github.com/toolforge/quarry/pull/82
[11:06:33] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856294 (10fnegri) p:05Triage→03Medium
[11:09:14] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856303 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by fnegri: Started updating wiki replica views
[11:14:57] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856318 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by fnegri completed: - an-redacteddb1001.eqiad.wmnet (**PASS**)   - Ran P...
[11:16:04] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856321 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by fnegri: Started updating wiki replica views
[11:17:41] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:18:41] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:20:21] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856328 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by fnegri completed: - an-redacteddb1001.eqiad.wmnet (**PASS**)   - Ran P...
[11:21:29] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:22:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:27:31] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:30:53] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: Run maintain-views to create new ORES tables - https://phabricator.wikimedia.org/T395122#10856350 (10fnegri) 05In progress→03Resolved Using `--table` worked quite fast, this task is Resolved.
[11:33:47] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[11:35:14] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Make cloudvirt.safe_reboot stable enough to use for the entire cluster - https://phabricator.wikimedia.org/T395244 (10taavi) 03NEW
[11:48:10] <wikibugs>	 (03PS1) 10Majavah: openstack: cloudvirt: safe_reboot: Ask what to do if drain fails [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150635 (https://phabricator.wikimedia.org/T395244)
[11:49:36] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::codfw1dev::nova::compute::service}'
[12:26:43] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/21
[12:26:44] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/38
[12:48:25] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::codfw1dev::control}'
[12:49:18] <wikibugs>	 (03PS1) 10Majavah: openstack: cloudnet: Remove dead code [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150667
[12:49:42] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.roll_reboot_cloudnets
[12:49:51] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
[12:52:59] <wikibugs>	 (03PS1) 10Majavah: openstack: neutron: Only list down agents in NetworkUnhealthy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150668
[12:53:04] <wikibugs>	 (03CR) 10Majavah: [C:03+2] toolforge: k8s: Wait for host to come back up after hard reboot [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1145088 (owner: 10Majavah)
[12:54:27] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.roll_reboot_cloudnets
[12:54:31] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.roll_reboot_cloudnets (exit_code=99)
[12:54:36] <wikibugs>	 (03CR) 10FNegri: [C:03+1] "This might make the cookbook less "fire it and forget it", but it's probably a good thing so that you look into why the drain fails instea" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150635 (https://phabricator.wikimedia.org/T395244) (owner: 10Majavah)
[12:56:42] <wikibugs>	 (03Merged) 10jenkins-bot: toolforge: k8s: Wait for host to come back up after hard reboot [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1145088 (owner: 10Majavah)
[12:56:53] <wikibugs>	 (03PS2) 10Majavah: openstack: cloudvirt: safe_reboot: Ask what to do if drain fails [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150635 (https://phabricator.wikimedia.org/T395244)
[12:56:53] <wikibugs>	 (03PS2) 10Majavah: openstack: cloudnet: Remove dead code [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150667
[12:56:53] <wikibugs>	 (03PS2) 10Majavah: openstack: neutron: Only list down agents in NetworkUnhealthy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150668
[12:57:31] <wikibugs>	 10Wikibugs, 10Phabricator: Wikibugs reports color of milestones wrong - https://phabricator.wikimedia.org/T395250#10856681 (10A_smart_kitten) I wonder if this might be an upstream #phabricator issue - the Conduit API also says that the color of that tag (`PHID-PROJ-fx3a7xtrqi6qkrvoddik`) is blue.
[13:03:43] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2005-dev.codfw.wmnet
[13:06:24] <wikibugs>	 (03CR) 10Majavah: [C:03+2] "Right now the cookbook crashes if this fails, and you need to start all over again. So for eqiad1 this is more like moving it from "unusab" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150635 (https://phabricator.wikimedia.org/T395244) (owner: 10Majavah)
[13:06:55] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2005-dev.codfw.wmnet
[13:09:53] <wikibugs>	 (03PS1) 10Majavah: openstack: cloudnet: show: s/linuxbridge/OVS/g [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150678
[13:09:54] <wikibugs>	 (03PS1) 10Majavah: openstack: common: Drop Linuxbridge alert support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679
[13:09:57] <wikibugs>	 (03Merged) 10jenkins-bot: openstack: cloudvirt: safe_reboot: Ask what to do if drain fails [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150635 (https://phabricator.wikimedia.org/T395244) (owner: 10Majavah)
[13:10:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudnet.reboot_node for host cloudnet2006-dev.codfw.wmnet
[13:13:01] <wikibugs>	 10Wikibugs, 10Phabricator: Wikibugs reports color of milestones wrong - https://phabricator.wikimedia.org/T395250#10856721 (10taavi) Hmm. That seems likely. I note that {https://phabricator.wikimedia.org/source/phabricator/browse/wmf%252Fstable/src/applications/project/storage/PhabricatorProject.php} has separ...
[13:13:22] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) for host cloudnet2006-dev.codfw.wmnet
[13:16:07] <wikibugs>	 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api,buildsa-api] When building and deploying, if none of the settings changed, the jobs are not restarted - https://phabricator.wikimedia.org/T389044#10856723 (10Raymond_Ndibe) >>! In T389044#10851295, @dcaro wrote: >>>! In T389044#10851153,...
[13:23:24] <wikibugs>	 (03CR) 10FNegri: [C:03+1] openstack: cloudnet: Remove dead code [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150667 (owner: 10Majavah)
[13:23:28] <wikibugs>	 (03CR) 10FNegri: [C:03+1] openstack: neutron: Only list down agents in NetworkUnhealthy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150668 (owner: 10Majavah)
[13:23:34] <wikibugs>	 (03CR) 10FNegri: [C:03+1] openstack: cloudnet: show: s/linuxbridge/OVS/g [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150678 (owner: 10Majavah)
[13:23:39] <wikibugs>	 (03CR) 10FNegri: [C:03+1] openstack: common: Drop Linuxbridge alert support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679 (owner: 10Majavah)
[13:24:14] <wikibugs>	 (03CR) 10Majavah: [C:03+2] openstack: cloudnet: Remove dead code [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150667 (owner: 10Majavah)
[13:24:20] <wikibugs>	 (03CR) 10Majavah: [C:03+2] openstack: neutron: Only list down agents in NetworkUnhealthy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150668 (owner: 10Majavah)
[13:25:35] <wikibugs>	 (03CR) 10Majavah: [C:03+2] openstack: cloudnet: show: s/linuxbridge/OVS/g [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150678 (owner: 10Majavah)
[13:25:43] <wikibugs>	 (03CR) 10Majavah: [C:03+2] openstack: common: Drop Linuxbridge alert support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679 (owner: 10Majavah)
[13:27:16] <wikibugs>	 (03CR) 10Majavah: openstack: common: Drop Linuxbridge alert support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679 (owner: 10Majavah)
[13:27:24] <wikibugs>	 (03PS2) 10Majavah: openstack: common: Drop Linuxbridge agent support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679
[13:27:29] <wikibugs>	 (03CR) 10Majavah: [C:03+2] openstack: common: Drop Linuxbridge agent support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679 (owner: 10Majavah)
[13:28:20] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: codfw1dev has seen neutron metadata agents down since epoxy upgrade - https://phabricator.wikimedia.org/T395255 (10taavi) 03NEW
[13:28:25] <wikibugs>	 (03Merged) 10jenkins-bot: openstack: cloudnet: Remove dead code [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150667 (owner: 10Majavah)
[13:28:25] <wikibugs>	 (03Merged) 10jenkins-bot: openstack: neutron: Only list down agents in NetworkUnhealthy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150668 (owner: 10Majavah)
[13:28:48] <wikibugs>	 (03Merged) 10jenkins-bot: openstack: cloudnet: show: s/linuxbridge/OVS/g [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150678 (owner: 10Majavah)
[13:33:21] <wikibugs>	 (03Merged) 10jenkins-bot: openstack: common: Drop Linuxbridge agent support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1150679 (owner: 10Majavah)
[14:02:12] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Make cloudvirt.safe_reboot stable enough to use for the entire cluster - https://phabricator.wikimedia.org/T395244#10856846 (10taavi) p:05Triage→03Medium
[14:02:27] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Make cloudvirt.safe_reboot stable enough to use for the entire cluster - https://phabricator.wikimedia.org/T395244#10856847 (10taavi) 05Open→03Resolved Let's call that done for now, and re-open if/when new issues surface.
[14:02:34] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: codfw1dev has seen neutron metadata agents down since epoxy upgrade - https://phabricator.wikimedia.org/T395255#10856849 (10taavi) p:05Triage→03Medium
[14:03:20] <wikibugs>	 06cloud-services-team: HighIOWaitStalling High iowait detected on clouddumps1002:9100. - https://phabricator.wikimedia.org/T394613#10856851 (10taavi) 05Open→03Resolved These haven't been seen since 1001 was put back into service.
[14:05:21] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06Data-Platform-SRE, 13Patch-For-Review: PuppetConstantChange on clouddumps100[12] - https://phabricator.wikimedia.org/T394921#10856865 (10taavi) 05Open→03Resolved a:03BTullis
[14:08:57] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Persistence, 10Data-Platform, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): an-redacteddb1001: upgrade MariaDB to 10.11 - https://phabricator.wikimedia.org/T394930#10856873 (10taavi)
[14:23:30] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 20): [infra] Toolforge bastion sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10856908 (10taavi) 05Open→03Resolved Let's call this resolved for now. There's been a few fixes applied here and in T394283 and we haven't go...
[14:35:30] <wikibugs>	 06cloud-services-team: SystemdUnitDown The systemd unit backup_glance_images.service on node cloudbackup1003 has been failing for more than two hours. - https://phabricator.wikimedia.org/T395133#10856977 (10taavi) 05Open→03Resolved This alert has cleared by itself
[14:36:47] <wikibugs>	 10Data-Services, 06Data-Persistence, 10Data-Platform, 10Data-Platform-SRE (2025.05.24 - 2025.06.13): an-redacteddb1001: upgrade MariaDB to 10.11 - https://phabricator.wikimedia.org/T394930#10856989 (10taavi)
[14:57:12] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] skip build if refs are same [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044)
[15:03:45] <wikibugs>	 (03update) 10addshore: Draft: Components [repos/cloud/toolforge/toolforge-gen-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli/-/merge_requests/2
[15:04:37] <wikibugs>	 06cloud-services-team, 10Toolforge: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10857093 (10Ladsgroup) I might be very wrong but historically speaking Sentry has been considered not open source and as result the code for its integrati...
[15:14:16] <wikibugs>	 06cloud-services-team, 10Toolforge: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10857111 (10Nokib_Sarkar) @Ladsgroup As per their license, as long as we are not their commercial compeititor, the license is exactly like Open-source Apa...
[15:15:24] <wikibugs>	 10Tool-campwiz-nxt: Migration of CampWiz NXT to toolforge - https://phabricator.wikimedia.org/T394515#10857112 (10Nokib_Sarkar)
[15:19:07] <wikibugs>	 10Tool-campwiz-nxt: Implement Reverse proxy and Failover server into campwiz nxt - https://phabricator.wikimedia.org/T394730#10857116 (10Nokib_Sarkar) Now fixing this issue solved another issue called bandwidth issue for the users. I am very grateful to you for figuring out the issue @dcaro . But We figured out...
[16:16:15] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Refactor maintenance scripts to allow local testing - https://phabricator.wikimedia.org/T395266 (10fnegri) 03NEW
[16:17:40] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#10857286 (10fnegri) I split the refactoring work I've started into a subtask: {T395266}  This task will retain its original scope of "addi...
[16:17:54] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Refactor maintenance scripts to allow local testing - https://phabricator.wikimedia.org/T395266#10857288 (10fnegri) p:05Triage→03Medium
[16:21:17] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#10857302 (10fnegri)
[16:24:05] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#10857305 (10fnegri)
[16:24:37] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 13Patch-For-Review: [wikireplicas] Refactor maintenance scripts to allow local testing - https://phabricator.wikimedia.org/T395266#10857306 (10fnegri) 05Open→03In progress
[16:51:57] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/21 (owner: 10l10n-bot)
[16:52:01] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/21 (owner: 10l10n-bot)
[16:53:36] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/38 (owner: 10l10n-bot)
[16:53:39] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/38 (owner: 10l10n-bot)
[17:00:28] <wikibugs>	 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api,buildsa-api] When building and deploying, if none of the settings changed, the jobs are not restarted - https://phabricator.wikimedia.org/T389044#10857378 (10dcaro) Hey, if you put it in the config, there's no way to control per-deployme...
[17:02:03] <wikibugs>	 (03open) 10raymond-ndibe: [components-api] add forcebuild and forcerun query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044)
[17:21:15] <notefromgithub>	 supertassu closed https://github.com/toolforge/quarry/pull/82
[17:55:29] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] setup: Autodiscover the /etc/php/X.Y directory [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1149744 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[18:05:06] <wikibugs>	 10Cloud-VPS (Quota-requests): Temporary quota increase for 'cvn' - https://phabricator.wikimedia.org/T395274 (10Krinkle) 03NEW
[18:09:42] <wikibugs>	 10Cloud-VPS (Quota-requests): Temporary quota increase for 'cvn' - https://phabricator.wikimedia.org/T395274#10857558 (10taavi) Just to confirm: do you explicitely need public v4 addresses? New VMs get public v6 addresses by default which might be enough depending on your use case.
[18:15:47] <wikibugs>	 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10857560 (10taavi) 05Open→03Resolved
[18:18:50] <wikibugs>	 (03PS1) 10Krinkle: setup: Add `sudo mkdir -p /etc/cron.hourly` [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150745 (https://phabricator.wikimedia.org/T395164)
[18:21:21] <wikibugs>	 (03CR) 10Majavah: setup: Add `sudo mkdir -p /etc/cron.hourly` (031 comment) [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150745 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[18:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[18:39:21] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] skip build if refs are same [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044)
[18:42:23] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] add forcebuild and forcerun query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044)
[18:46:52] <wikibugs>	 (03PS2) 10Krinkle: setup: Install `cron` by default [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150745 (https://phabricator.wikimedia.org/T395164)
[18:49:49] <wikibugs>	 (03CR) 10Krinkle: setup: Install `cron` by default (031 comment) [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150745 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[18:49:58] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] setup: Install `cron` by default [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150745 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[19:41:59] <wikibugs>	 (03CR) 10Krinkle: Build: Update build system (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1143806 (https://phabricator.wikimedia.org/T395036) (owner: 10Slyngshede)
[19:43:03] <wikibugs>	 (03PS1) 10Krinkle: setup: Remove redundant ca-certificates-mono workaround [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150753 (https://phabricator.wikimedia.org/T395164)
[19:47:12] <wikibugs>	 10Cloud-VPS (Quota-requests): Temporary quota increase for 'cvn' - https://phabricator.wikimedia.org/T395274#10857694 (10Krinkle) That depends on: * what Libera Chat supports/does (does it support connecting over IPv6, and if yes, do they grant IPv6 addys the same concurrent client allowance?)  * what the DNS an...
[19:55:44] <wikibugs>	 (03PS1) 10Krinkle: mysql_cvnclerkbot.sql: Dump fresh backup [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150756 (https://phabricator.wikimedia.org/T395164)
[19:56:19] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] mysql_cvnclerkbot.sql: Dump fresh backup [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1150756 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[19:57:24] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] megatable: Fix "XML Parsing Error: not well-formed" warning [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1147039 (owner: 10Krinkle)
[20:33:14] <wikibugs>	 (03PS1) 10Krinkle: Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164)
[20:33:31] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[20:36:19] <wikibugs>	 (03PS2) 10Krinkle: Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164)
[20:36:30] <icinga-wm>	 RECOVERY - MD RAID on cloudcephmon1004 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering
[20:36:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[20:42:20] <wikibugs>	 (03PS3) 10Krinkle: Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164)
[20:43:06] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[20:43:38] <wikibugs>	 (03Merged) 10jenkins-bot: Commit ulrichsg/getopt-php 3.4.0 to fix fatal "Array and string offset access syntax with curly" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150764 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[20:47:27] <wikibugs>	 (03PS1) 10Krinkle: Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150768 (https://phabricator.wikimedia.org/T395164)
[20:47:48] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150768 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[20:48:18] <wikibugs>	 (03Merged) 10jenkins-bot: Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1150768 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[21:26:19] <wikibugs>	 (03open) 10raymond-ndibe: [components.deployment.create] add force-build and force-restart option [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T389044)
[21:31:18] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] add forcebuild and forcerun query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044)
[21:32:02] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] add forcebuild and forcerun query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044)
[21:32:21] <wikibugs>	 (03update) 10raymond-ndibe: [components-api] add force-build and force-run query params [repos/cloud/toolforge/components-api] (skip_build_if_refs_are_same) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/80 (https://phabricator.wikimedia.org/T389044)
[21:32:33] <wikibugs>	 10Tools: zoomviewer uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T395020#10857873 (10dschwen) Hm, only the pyramids should need to be retained. This amount of growth indicates new images being accessed. A look at the logs might be in order. Is this coming from a small number of IPs?
[21:34:35] <wikibugs>	 06cloud-services-team, 10Toolforge: Is Using sentry for error monitoring against wikimedia cloud privacy policy? - https://phabricator.wikimedia.org/T394577#10857874 (10Ladsgroup) Yeah, this is [[https://techcrunch.com/2023/11/20/with-functional-source-license-sentry-wants-to-grant-developers-freedom-without-h...
[22:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:47:48] <wikibugs>	 10Tools: zoomviewer uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T395020#10857987 (10tstarling) >>! In T395020#10857873, @dschwen wrote: > Hm, only the pyramids should need to be retained.   Right now, it depends on originals being retained.  > This amount of growth indicates new...