[00:12:59] FIRING: CloudVPSDesignateLeaks: Detected 15 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:14:56] 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12173266 (10AntiCompositeNumber) [01:20:57] 10Tool-refill: Watchlist labels are reset for article when ReFill is used on it - https://phabricator.wikimedia.org/T432197#12173275 (10StefenTower) [04:12:59] FIRING: CloudVPSDesignateLeaks: Detected 15 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:14:08] (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5 [04:14:11] (03open) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5 [04:46:06] (03merge) 10countcount: Mirror and expose per-wiki global-block local-status history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [04:48:47] 10Tool-toolinfo-scraper: Prevent tool pages duplication while importing the scripts list from enwiki in Toolhub - https://phabricator.wikimedia.org/T433659 (10SerDIDG) 03NEW [06:34:49] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173440 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048... [06:44:40] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173452 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt... [07:36:54] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12173499 (10dcaro) a:05TLepage-WMF→03None [08:12:59] FIRING: CloudVPSDesignateLeaks: Detected 15 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:14:30] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12173560 (10fgiunchedi) I am working on getting cloudvirt1048 up an running in C8 after the move. The current situation... [08:33:07] (03update) 10tlepage: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [08:37:30] (03update) 10tlepage: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [08:37:59] (03merge) 10tlepage: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) [08:52:53] 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-Project-wikicommunityhealth: Change mount propagation for /mnt/nfs in backend.wikicommunityhealth - https://phabricator.wikimedia.org/T433667 (10fgiunchedi) 03NEW [09:00:23] 10Cloud-VPS, 06tools-infrastructure-team, 10Research-collaborations: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api - https://phabricator.wikimedia.org/T433668 (10fgiunchedi) 03NEW [09:10:42] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671 (10fgiunchedi) 03NEW [09:12:19] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12173710 (10fnegri) Things are not getting better, history length is now over 100 mil. There are currently 52 queries from the `mixnmatch` user (`s51434... [09:14:41] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12173724 (10fnegri) I didn't get any reply from @magnusmanske either on this task or via email, I will now stop some of the mixnmatch processes, startin... [09:15:42] 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-Project-wikicommunityhealth: Change mount propagation for /mnt/nfs in backend.wikicommunityhealth - https://phabricator.wikimedia.org/T433667#12173727 (10fgiunchedi) [09:15:49] 10Cloud-VPS, 06tools-infrastructure-team, 10Research-collaborations: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api - https://phabricator.wikimedia.org/T433668#12173728 (10fgiunchedi) [09:15:59] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12173729 (10fgiunchedi) [09:20:48] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12173738 (10fnegri) `lang=shell-session tools.mix-n-match@tools-bastion-15:~$ kubectl scale --replicas=0 deployment.apps/rustbot tools.mix-n-match@tool... [09:23:48] FIRING: PuppetFailure: Puppet has failed on cloudvirt1048:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:31:10] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12173747 (10fnegri) I killed all queries with `time > 3600`. We're down to 27 threads for `s51434`, most of them in `Sleep`. The remaining `maintenance_... [09:34:22] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12173756 (10fnegri) This is the current situation, I will let it run for a while and check back later: `lang=mysql MariaDB [(none)]> SELECT id, time, c... [09:56:04] RESOLVED: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 24d 11h 38m 21s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [10:15:45] (03update) 10renovatebot: Update dependency brace-expansion to v5.0.9 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/28 [10:15:49] (03open) 10renovatebot: Update dependency brace-expansion to v5.0.9 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/28 [10:26:48] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [11:18:17] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12173976 (10VRiley-WMF) 05Open→03In progress [11:18:38] (03open) 10tlepage: Draft: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [11:19:38] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [11:26:05] PROBLEM - Host cloudcephosd1044 is DOWN: PING CRITICAL - Packet loss = 100% [11:27:11] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [11:28:20] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [11:29:47] FIRING: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [11:36:10] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [11:40:37] (03update) 10tlepage: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [11:41:23] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) [11:41:37] (03merge) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) [11:44:38] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [11:44:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.39-20260731114147-1af2003d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1353 [11:44:43] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.39-20260731114147-1af2003d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1353 [11:46:35] RECOVERY - Host cloudcephosd1044 is UP: PING OK - Packet loss = 0%, RTA = 0.16 ms [11:49:47] RESOLVED: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [11:50:11] 10Tool-wmf-openapi-linter, 06Content-Platform-Team (Work In Progress), 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12174075 (10daniel) Proposal to unblock: * make a... [11:53:20] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [11:53:37] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [11:53:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:01:03] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [12:03:15] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [12:04:06] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12174092 (10aputhin) >>! In T432860#12171702, @fnegri wrote: >> I'd strongly recommend not to direct users to karma/prometheus > > Fair enough, tooling might change and we... [12:06:47] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12174095 (10VRiley-WMF) Updated firmware on iDRAC and BIOS. No change of yet. Grabbing TSR and will be opening up a ticket with Dell [12:12:59] FIRING: CloudVPSDesignateLeaks: Detected 15 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:25:26] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690 (10aputhin) 03NEW [12:25:50] 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12174177 (10aputhin) [12:30:53] (03close) 10tlepage: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [12:31:03] (03reopen) 10tlepage: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [12:31:29] (03approved) 10tlepage: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [12:31:57] (03merge) 10tlepage: d/changelog: bump to 16.2.0 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/168 [12:32:19] (03merge) 10tlepage: logs-api: bump to 0.0.39-20260731114147-1af2003d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1353 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:31] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12174230 (10fnegri) History length has started going down: {F96644058} Disk metrics are also much healthier now: {F96644067} [12:33:47] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12174234 (10TLepage-WMF) 05In progress→03Resolved a:03TLepage-WMF All MRs merged, and verified that `toolforge jobs logs` is working as expected on tool... [12:36:32] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12174253 (10TLepage-WMF) @DamianZaremba thank you for reporting this, please let us know if you notice anything wrong! [[ https://wikitech.wikimedia.org/w/ind... [12:37:22] 10Toolforge, 06tools-platform-team: [logs-api] Using --latest N gives the first N lines since the last hour, not the last - https://phabricator.wikimedia.org/T419863#12174259 (10TLepage-WMF) 05Open→03Resolved a:03TLepage-WMF Fixed via T428253. [12:56:16] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12174345 (10Chippyy) maps-warper4 instance is now paused. monitoring continues [13:06:15] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [13:21:55] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:24:26] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [13:30:38] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174510 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048... [13:40:37] 10Cloud-VPS, 06tools-infrastructure-team, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12174538 (10brouberol) [13:40:45] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 5 others: codfw: rack B7 maintenance - Tuesday July 21st 14:00 UTC - https://phabricator.wikimedia.org/T430928#12174539 (10brouberol) [13:44:33] (03approved) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [13:44:43] (03merge) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [13:44:50] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755) [13:56:19] RESOLVED: PuppetFailure: Puppet has failed on cloudvirt1048:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [14:07:24] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12174649 (10fnegri) Some stats on the proposed alerts. You can quickly recompute this yourself by pasting the expressions below at https://prom... [14:10:32] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174654 (10fgiunchedi) Ok baby steps: reimage wasn't completing because puppet wasn't able to complete a successful run... [14:13:23] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, and 3 others: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267#12174656 (10VRiley-WMF) opened ticket 229798904 with dell [14:15:07] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174658 (10fgiunchedi) pinging hosts on `cloud-private` though still doesn't work: ` root@cloudvirt1048:~# ping -c2 -4... [14:37:47] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174803 (10taavi) The switch port was missing the required tagged VLANs (cloud-instances and cloud-private), I fixed th... [14:41:09] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174829 (10fgiunchedi) Thank you @taavi, appreciate it! I don't understand what this line in the reimage cookbook outp... [14:51:35] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12174859 (10taavi) Good question. Looking at the Netbox changelog that change was first made in [[ https://netbox.wikime... [14:56:14] 10Cloud-VPS, 06tools-infrastructure-team, 10VPS-Project-wikicommunityhealth: Change mount propagation for /mnt/nfs in backend.wikicommunityhealth - https://phabricator.wikimedia.org/T433667#12174868 (10fgiunchedi) [14:56:20] 10Cloud-VPS, 06tools-infrastructure-team, 10Research-collaborations: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api - https://phabricator.wikimedia.org/T433668#12174869 (10fgiunchedi) [14:56:29] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12174870 (10fgiunchedi) [15:26:30] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12174946 (10mikez-WMF) @TheDJ exactly, thank you for putting it into better practical words than me. This sounds like a much more realistic short-term step to p... [15:28:19] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714 (10Bawolff) 03NEW [16:11:03] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12175075 (10fgiunchedi) In the meantime I have restore nova-compute and brought it up: ` | 79d5cff2-1f5b-423d-bd9a-6119... [16:11:57] (03open) 10krinkle: channels: update mediawiki-core-bots following July 2026 renames [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/78 [16:13:00] FIRING: CloudVPSDesignateLeaks: Detected 15 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:20:04] (03merge) 10krinkle: channels: update mediawiki-core-bots following July 2026 renames [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/78 [16:24:38] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175108 (10Eevans) I missed the email sent on Jun 17 (though I'm not sure it would have changed things much); I only became aware of this after Wednesday's (Jul 2... [16:35:56] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175143 (10bd808) >>! In T401804#12175107, @Eevans wrote: > I missed the email sent on Jun 17 (though I'm not sure it would have changed things much); I only beca... [16:36:00] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12175147 (10bd808) >>! In T401804#12175107, @Eevans wrote: > I... [16:37:47] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175150 (10bd808) [16:56:00] (03open) 10taavi: Use Toolforge CDNjs mirror for loading jQuery [toolforge-repos/sigma] - 10https://gitlab.wikimedia.org/toolforge-repos/sigma/-/merge_requests/1 [17:00:30] (03update) 10taavi: Use Toolforge CDNjs mirror for loading jQuery [toolforge-repos/sigma] - 10https://gitlab.wikimedia.org/toolforge-repos/sigma/-/merge_requests/1 [17:09:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [17:29:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [17:38:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12175321 (10bd808) >>! In T401804#12175107, @Eevans wrote: > Mo... [17:41:13] (03open) 10countcount: Show local global-block disable history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/18 [17:48:12] (03update) 10countcount: Show local global-block disable history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/18 [17:48:20] (03update) 10countcount: Show local global-block disable history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/18 [17:53:25] (03update) 10countcount: Show local global-block disable history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/18 [17:53:35] (03update) 10countcount: Show local global-block disable history [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/18 [18:26:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:29:23] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12175382 (10Eevans) >>! In T401839#12175319, @bd808 wrote: > [... [18:31:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:31:54] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175529 (10Cyberpower678) All Cyberbot VMs are now upgraded to Trixie [19:53:44] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12175555 (10taavi) This task is vague enough that I can't tell whether this is a request for help (which we usually prefer on the mailing list or on IR... [19:57:02] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175557 (10taavi) >>! In T401804#12175529, @Cyberpower678 wrote: > All Cyberbot VMs are now upgraded to Trixie It seems like an in-place upgrade was done there i... [20:02:06] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12175566 (10Cyberpower678) >>! In T401804#12175557, @taavi wrote: >>>! In T401804#12175529, @Cyberpower678 wrote: >> All Cyberbot VMs are now upgraded to Trixie >... [20:12:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:13:00] FIRING: CloudVPSDesignateLeaks: Detected 17 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:16:33] (03CR) 10Majavah: [C:04-1] "-1 on the `docker compose`/`docker-compose` thing, otherwise this all LGTM. I've tested locally as well (although I did not fully run thro" [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) (owner: 10Perryprog) [20:17:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:54:06] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12175622 (10Bawolff) >>rsync (port 883) > rsync by default uses port 873/tcp, is this an intentional change? That was a mistake. 873 is correct. [20:54:36] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12175623 (10Bawolff) [20:55:06] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12175624 (10Bawolff) [20:55:19] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12175625 (10HCoplin-WMF) [20:58:15] 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API, 07OKR-Work: Include contributor_count in Attribution API responses - https://phabricator.wikimedia.org/T429834#12175628 (10HCoplin-WMF) [21:01:46] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [21:12:50] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#12175668 (10BLiviero-WMF) In a different phabricator ticket it was pointed out that an upgrade-in-place may not be ideal... [21:22:58] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API: Update OpenAPI Linter to limit spec descriptions - https://phabricator.wikimedia.org/T433741 (10HCoplin-WMF) 03NEW [21:28:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [21:38:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [22:59:38] 10Cloud-VPS (Debian Bullseye Deprecation): Extension request for account-creation-assistance - https://phabricator.wikimedia.org/T433749 (10stwalkerster) 03NEW [23:06:01] 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12175900 (10ZhaoFJx) Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to seasoned contributors, so please take a mome... [23:06:48] 10Tool-wikinewsie, 07good first task: Wikinewsie i18n - https://phabricator.wikimedia.org/T432938#12175902 (10ZhaoFJx) Thanks @Yaron_Koren ! [23:35:46] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859#12175954 (10ZhaoFJx) All criterias are satisfied [23:35:48] 10Tool-wikinewsie: Wikinewsie Clickable Blue Links - https://phabricator.wikimedia.org/T432859#12175956 (10ZhaoFJx) 05In progress→03Resolved [23:39:54] (03CR) 10Perryprog: Update dev environment and its documentation (034 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) (owner: 10Perryprog) [23:40:04] (03PS2) 10Perryprog: Update dev environment and its documentation [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525)