[00:04:58] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s6 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/739 [00:04:58] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump redis from 8.6.7 to 8.6.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/741 [00:04:59] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump mariadb from 11.4 to 11.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/740 [00:05:01] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump redis from 8.6.7 to 8.6.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/741 [00:05:17] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/dev/bookworm-php83-fpm from 1.0.0-s7 to 1.0.0-s9 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/742 [00:05:17] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia from v1.33.0 to v1.35.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/743 [00:05:22] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/dev/bookworm-php83-fpm from 1.0.0-s7 to 1.0.0-s9 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/742 [00:05:23] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia from v1.33.0 to v1.35.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/743 [01:39:15] 10GitLab: Unable to push to gitlab - https://phabricator.wikimedia.org/T442790#12487637 (10Bugreporter) [03:00:17] Project mwcore-phpunit-coverage-master build #5722: 04FAILURE in 17 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5722/ [03:02:31] FIRING: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:02:36] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T442798 (10phaultfinder) 03NEW [03:07:31] RESOLVED: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:49:56] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: NDA Request for a Yubikey - https://phabricator.wikimedia.org/T442799#12487736 (10Aklapper) 05Invalidβ†’03Resolved a:03Aklapper Ah, I misunderstood now that I read that entire ticket. Done now. [04:18:51] 10GitLab: Unable to push to gitlab - https://phabricator.wikimedia.org/T442790#12487754 (10Aklapper) > Check out a project using the ssh form Which a project? [04:38:22] 10GitLab: Unable to push to gitlab - https://phabricator.wikimedia.org/T442790#12487771 (10Hawkeye7) milhistbot [04:38:53] 10GitLab: Unable to push to gitlab - https://phabricator.wikimedia.org/T442790#12487772 (10Hawkeye7) Milhistbot. Looks like a firewall issue. [05:27:48] 10GitLab (Support): Unable to push to gitlab - https://phabricator.wikimedia.org/T442790#12487794 (10brennen) 05Openβ†’03Resolved a:03brennen GitLab has moved behind the CDN - {T425441}. The change to `gitlab-ssh.wikimedia.org` for SSH remotes was announced back in June: wikitech-l: [[ https://lists.wik... [05:32:30] 10GitLab (Account Approval), 06Release-Engineering-Team, 10Tool-wikinewsie, 07Essential-Work: Requesting GitLab account activation for Wikinewsie Columbia students - https://phabricator.wikimedia.org/T442748#12487805 (10brennen) 05Openβ†’03Resolved a:03brennen Approved - happy hacking to all. [07:51:06] (03CR) 10Hashar: "I actually manually created symlinks in my checkout in order to verify the behavior. **done** by adding symlinks for each node versions w" [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [07:52:11] (03PS5) 10Hashar: Consolidate fresh-node* scripts to a single file [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) [08:06:24] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12488080 (10Volans) >>! In T442482#12486969, @Southparkfan wrote: >>>! In T442482#12486204, @Volans wrote: >> [...] >> The other thing that could b... [08:36:52] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12488130 (10Volans) Giving it another pass, will report shortly [08:38:35] 10Gerrit, 13Patch-For-Review: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#12488134 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d06abd8e-903e-4e34-ae5f-81cfdef620e5) set by jelto@cumin1004 for 0:15:00 on 1 host(s) and their services with reason: Java... [08:39:06] 10Gerrit, 13Patch-For-Review: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#12488148 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a1bbe82f-0e2b-4b1d-b824-14a320482a80) set by jelto@cumin1004 for 0:15:00 on 14 host(s) and their services with reason: Jav... [08:41:34] 10Phabricator, 06Collaboration-Services, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12488154 (10Addshore) Had a quick look today, and I think that all looks good to me (infact, to be honest, eve... [08:44:48] 10Gerrit: Check Gerrit mmap setting under Java 21 - https://phabricator.wikimedia.org/T442811 (10hashar) 03NEW [08:53:33] 10Gerrit, 13Patch-For-Review: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#12488189 (10hashar) 05Openβ†’03Resolved Resolved! Paired with @Jelto to roll the patch with the assistance of @MoritzMuehlenhoff for the Java side. πŸŽ‰ [08:58:24] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services: Check Gerrit mmap setting under Java 21 - https://phabricator.wikimedia.org/T442811#12488227 (10hashar) [09:01:33] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12488239 (10Volans) @brennen I gave it another pass and that's what's reported. Apart from `k3s-envDB` that doesn't hav... [09:02:54] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services: Check Gerrit mmap setting under Java 21 - https://phabricator.wikimedia.org/T442811#12488241 (10hashar) 05Openβ†’03Resolved a:03hashar From https://jira.xwiki.org/browse/XWIKI-22012: Vincent Massol wrote: > Note that this is not a probl... [09:04:57] 10Scap, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12488245 (10taavi) The [[ https://gitlab.wikimedia.org/repos/mediawiki/services/chart-renderer | chart-renderer ]] service seems to be mis... [09:14:44] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12488254 (10taavi) One last thing: what about the Magnum cluster? Do we know if it is not running any user-exposed vulnerable PHP/etc services? (I... [09:29:11] Yippee, build fixed! [09:29:11] Project mwcore-phpunit-coverage-master build #5723: 09FIXED in 29 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/5723/ [09:39:30] (03CR) 10Sanskardubey: "Thank you both for adding me to the allowlist!" [integration/config] - 10https://gerrit.wikimedia.org/r/1359312 (owner: 10Urbanecm) [09:52:11] (03PS1) 10Hashar: Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) [10:01:16] 10Gerrit, 06Release-Engineering-Team, 07Browser-Support-Google-Chrome, 07Browser-Support-Microsoft-Edge: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12488435 (10Xqt) Additional information: I always get this error when opening the Gerrit UI, unde... [10:05:21] 10Gerrit, 06Release-Engineering-Team, 07Browser-Support-Google-Chrome, 07Browser-Support-Microsoft-Edge: gerrit: Checks information and changed files failed to load. - https://phabricator.wikimedia.org/T432548#12488460 (10hashar) 05Openβ†’03Resolved a:03hashar The original issue was most certainly... [10:31:11] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - gerrit1003 - https://phabricator.wikimedia.org/T442798#12488542 (10LSobanski) [12:34:42] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12489330 (10hashar) [12:47:13] 10GitLab (Support): Unable to push to gitlab (due to T425441: gitlab behind CDN) - https://phabricator.wikimedia.org/T442790#12489385 (10Aklapper) [13:08:59] 06Release-Engineering-Team, 10Ceph, 06Infrastructure-Foundations, 10SRE-swift-storage, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12489452 (10elukey) Next steps: * Review and merge https://gerrit.wikimedia.org/r/c/operations... [13:20:03] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request to unlink MediaWiki/Wikimedia account from old/broken Phabricator account - https://phabricator.wikimedia.org/T442421#12489515 (10Aklapper) a:03Aklapper Unlinking is not possible but the Phab account seems unused so it could be deleted, so you can... [13:21:27] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request to unlink MediaWiki/Wikimedia account from old/broken Phabricator account - https://phabricator.wikimedia.org/T442421#12489528 (10Aklapper) I now just wonder how to verify that you are the same person who also owns `@JanusdeVikidia`. Maybe you could... [13:51:38] (03PS3) 10Hashar: Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) [13:51:38] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1360417" [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [13:52:06] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12489678 (10hashar) a:03hashar [14:03:05] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: Remove unneeded plugins from CI Jenkins - https://phabricator.wikimedia.org/T442871 (10hashar) 03NEW [14:20:26] (03PS1) 10Jforrester: Docker: [php83] Update PHP from 8.3.36 to 8.3.36-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1360422 [14:20:26] (03PS1) 10Jforrester: jjb: Update PHP83 jobs to 8.3.36-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1360423 [14:20:53] (03CR) 10Jforrester: [C:03+2] Docker: [php83] Update PHP from 8.3.36 to 8.3.36-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1360422 (owner: 10Jforrester) [14:22:19] (03Merged) 10jenkins-bot: Docker: [php83] Update PHP from 8.3.36 to 8.3.36-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1360422 (owner: 10Jforrester) [14:23:24] !log Docker: [php83] Update PHP from 8.3.36 to 8.3.36-1 [14:23:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:35:36] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1360423 (owner: 10Jforrester) [14:36:52] (03Merged) 10jenkins-bot: jjb: Update PHP83 jobs to 8.3.36-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1360423 (owner: 10Jforrester) [14:49:27] 10Scap, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12490114 (10dancy) [14:53:23] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12490130 (10Sfaci) [14:53:24] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470#12490133 (10dancy) 05Openβ†’03Resolved a:03dancy [14:53:37] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12490138 (10hashar) [15:12:26] (03open) 10dancy: Release 4.309.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1394 [15:12:28] (03update) 10dancy: Release 4.309.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1394 [15:15:06] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12490352 (10KReid-WMF) [15:16:27] (03merge) 10dancy: Release 4.309.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1394 [15:23:48] (03PS1) 10Eamedina: Zuul: [mediawiki/extensions/ArticleGuidance] Add VisualEditor to Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1360445 (https://phabricator.wikimedia.org/T439785) [15:25:23] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/ArticleGuidance] Add VisualEditor to Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1360445 (https://phabricator.wikimedia.org/T439785) (owner: 10Eamedina) [15:29:38] 10Scap: Weird scap install-world failure for aphlict2001.codfw.wmnet - https://phabricator.wikimedia.org/T382271#12490474 (10dancy) 05Openβ†’03Resolved a:03dancy [15:32:09] (03open) 10dancy: install_local_version.sh: Do not retry a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1395 (https://phabricator.wikimedia.org/T337394 https://phabricator.wikimedia.org/T382271) [15:32:24] (03update) 10dancy: install_local_version.sh: Do not retry a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1395 (https://phabricator.wikimedia.org/T337394 https://phabricator.wikimedia.org/T382271) [15:33:46] (03PS2) 10Eamedina: Zuul: [mediawiki/extensions/ArticleGuidance] Add VE to Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1360445 (https://phabricator.wikimedia.org/T439785) [15:33:52] (03update) 10dancy: make-container-image: Build image source files owned by 'app', not www-data [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/251 (https://phabricator.wikimedia.org/T411063) [15:35:26] (03merge) 10dancy: install_local_version.sh: Do not retry a failed install [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1395 (https://phabricator.wikimedia.org/T337394 https://phabricator.wikimedia.org/T382271) [15:42:29] CirrusSearch is dropping north of 200 req/s, we just don't have the capacity in codfw for all the current load. I'd like to scap out a patch that will re-route some traffic back to eqiad and hopefully fix this [15:51:07] 10WikimediaDebug, 10Wikimedia-Site-requests, 07Datacenter-Switchover, 06MediaWiki-Core-Platform-Team (Radar), 13Patch-For-Review: Have noc's debug.json server list automatically sort primary DC servers first - https://phabricator.wikimedia.org/T289745#12490634 (10ssingh) @SLyngshede-WMF / @jasmine_ : It... [15:56:07] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12490695 (10MLechvien-WMF) Proposal looks good to me with those safeguards. [16:15:32] (03PS5) 10Jforrester: jjb: Drop Node 24 jobs, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343071 (https://phabricator.wikimedia.org/T438085) [16:15:32] (03PS5) 10Jforrester: Docker: Drop Node 24 images, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/1343077 (https://phabricator.wikimedia.org/T438085) [16:19:03] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12490862 (10taavi) Thank you! `catalyst` looks good to me, I've unblocked it. `catalyst-qte` and `catalyst-dev` have n... [16:55:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Release Pipeline, 13Patch-For-Review: Upgrade integration-agent-docker instances to trixie - https://phabricator.wikimedia.org/T440670#12491067 (10Dzahn) The containerd config from 1356455 is live on the integration agents. O... [17:00:48] 10Scap, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12491079 (10Jdforrester-WMF) [17:03:43] 10Scap, 10ChangeProp, 06Data-Engineering, 10EventStreams, and 14 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12491112 (10Jdforrester-WMF) >>! In T393434#12488245, @taavi wrote: > The [[ https://gitlab.wikimedia.org/repos/mediawiki/services/chart-r... [17:28:56] 10WikimediaDebug, 10Wikimedia-Site-requests, 07Datacenter-Switchover, 06MediaWiki-Core-Platform-Team (Radar), 13Patch-For-Review: Have noc's debug.json server list automatically sort primary DC servers first - https://phabricator.wikimedia.org/T289745#12491277 (10SLyngshede-WMF) I'm working on an updated... [17:39:06] FIRING: WidespreadInstanceDown: Widespread instances down in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown [18:07:08] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491504 (10Southparkfan) T442482 would help us to upgrade containers running EOL versions of nodejs. [18:22:23] !log Shut down deployment-wikikube-v127 nodes - T442482 [18:22:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:22:27] T442482: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482 [18:23:54] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491613 (10Southparkfan) >>! In T442482#12488254, @taavi wrote: > One last thing: what about the Magnum cluster? Do we know if it is not running a... [18:25:04] hi releng folks [18:25:17] heh, came here too [18:25:41] phabricator just showed me this https://usercontent.irccloud-cdn.com/file/5zBEsOte/image.png [18:26:00] I hit a weird error this morning that went away (it was something like "database not connected " or some such, actually not what jack just posted) [18:26:20] hope all is OK over here and those are just temp blips while upgrading/whatever [18:26:24] back now [18:30:07] https://phabricator.wikimedia.org/T442909 [18:30:22] I am looking if there is something we can block but I can't react to IRC pings right now. [18:30:39] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491644 (10Southparkfan) [18:30:44] 10Phabricator: Phab unstable (2026-10-29) - https://phabricator.wikimedia.org/T442909#12491646 (10Wargo) β†’14Duplicate dup:03T442499 [18:30:48] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12491648 (10Wargo) [18:39:06] RESOLVED: WidespreadInstanceDown: Widespread instances down in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown [18:41:45] ^ we're back :) [18:45:58] Yay! [18:47:12] nice work! [18:49:36] \o/ [18:54:15] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491766 (10Southparkfan) [18:55:12] thanks :) [18:56:09] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12491779 (10Jdforrester-WMF) [18:56:16] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12491787 (10Jdforrester-WMF) a:05tchinβ†’03None [18:56:46] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491793 (10Southparkfan) Thanks Taavi! In the meantime, I discovered three images have been rebuilt yesterday. I have deployed the new images. At... [18:57:40] 10Beta-Cluster-Infrastructure: deployment-prep is down - https://phabricator.wikimedia.org/T442156#12491799 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan Network access has been restored in T442482. [19:05:00] Southparkfan: Keyholder needs to be rearmed on deployment-deploy06 each time it restarts. Do you want to do the honors? [19:07:09] Sure thing [19:07:10] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12491876 (10Andrew) >> Per T437500 if it is really serving no traffic one strategy would be to just get rid of that cluster and then re-build a CA... [19:07:25] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12491880 (10Dzahn) Several users pinged about getting the intermittent DB error again. I looked and found most time was spent on requests from the `GoogleOther` user agent... [19:07:42] I guess the Cumin host needs maintenance on its keyholder too... [19:12:01] References: https://wikitech.wikimedia.org/wiki/Keyholder particularly https://wikitech.wikimedia.org/wiki/Keyholder#WMCS_projects_passphrases [19:12:52] !log Armed keyholders on deployment and Cumin host after reboots - T442482 [19:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:56] T442482: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482 [19:13:25] Done! [19:14:41] Thanks! [19:15:12] testing scap sync-world [19:17:09] Perfect [19:17:19] jgleeson: We just made a change to ratel-limiting on phab, hopefully goes away now. [19:17:20] (03PS1) 10Jforrester: Docker: [java*] Re-build to get latest openjdks with October 2028 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 [19:17:20] (03PS1) 10Jforrester: jjb: Bump Java-based jobs to latest openjdks with October 2028 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360520 [19:17:30] greg-g: ^ same for you [19:17:37] +1 [19:17:38] Cool. Thanks for testing!! [19:18:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-jobrunner06 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:18:12] jgleeson: https://phabricator.wikimedia.org/T442499 [19:18:48] (03CR) 10CI reject: [V:04-1] Docker: [java*] Re-build to get latest openjdks with October 2028 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 (owner: 10Jforrester) [19:18:57] Error: Could not set 'present' on ensure: No such file or directory - A directory component in /srv/mwpresync/.gitconfig20261029-3949-efsmgu.lock does not exist or is a dangling symbolic link (file: /srv/puppet_code/environments/production/modules/git/manifests/userconfig.pp, line: 25) [19:21:05] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/docker/cli from 27.5.1+incompatible to 29.2.0+incompatible [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/744 [19:21:06] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/docker/cli from 27.5.1+incompatible to 29.2.0+incompatible [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/744 [19:21:27] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/glamour from 0.10.0 to 1.0.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/745 [19:21:48] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/txn2/txeh from 1.5.4 to 1.8.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/746 [19:22:18] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/x/ansi from 0.11.7 to 0.11.9 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/747 [19:22:42] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/term from 0.44.0 to 0.47.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/748 [19:22:44] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/term from 0.44.0 to 0.47.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/748 [19:23:08] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/rodaine/table from 1.3.1 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/749 [19:23:10] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/rodaine/table from 1.3.1 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/749 [19:23:37] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-runewidth from 0.0.23 to 0.0.31 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/750 [19:23:44] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-runewidth from 0.0.23 to 0.0.31 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/750 [19:23:45] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-runewidth from 0.0.23 to 0.0.31 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/750 [19:23:52] dancy: /srv is gone on the MediaWiki hosts, and I need to fix that. Do you need to deploy something in Beta? [19:23:56] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/sirupsen/logrus from 1.9.4 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/751 [19:24:07] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/sirupsen/logrus from 1.9.4 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/751 [19:24:11] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/sirupsen/logrus from 1.9.4 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/751 [19:24:16] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-isatty from 0.0.22 to 0.0.24 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/752 [19:24:21] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump golang.org/x/image from 0.20.0 to 0.41.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/753 [19:24:24] Southparkfan: Nothing beyond the usual automated wmf-beta-update-all [19:24:26] For some reason, the Cinder volume has not been mounted... [19:24:26] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-isatty from 0.0.22 to 0.0.24 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/752 [19:24:28] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/mattn/go-isatty from 0.0.22 to 0.0.24 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/752 [19:24:41] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump golang.org/x/image from 0.20.0 to 0.41.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/753 [19:24:49] Southparkfan: Is it listed in /etc/fstab ? [19:24:51] Right. Locking scap to fix this... [19:24:53] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump golang.org/x/image from 0.20.0 to 0.41.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/753 [19:24:53] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump golang.org/x/image from 0.20.0 to 0.41.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/753 [19:25:10] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/PuerkitoBio/goquery from 1.12.0 to 1.13.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/754 [19:25:16] (03merge) 10addshore: Bump redis from 8.6.7 to 8.6.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/741 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [19:25:22] There is something in fstab for /srv, but it's missing a proper partuuid [19:25:25] Ah, I see there's a weird entry in /etc/fstab for it: `UUID=None /srv None discard,nofail,x-systemd.device-timeout=2s 0 2` [19:25:27] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/PuerkitoBio/goquery from 1.12.0 to 1.13.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/754 [19:25:27] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/PuerkitoBio/goquery from 1.12.0 to 1.13.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/754 [19:25:30] That'll do it. [19:25:38] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbletea from 1.2.4 to 1.3.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/755 [19:25:47] (03merge) 10addshore: Bump docker-registry.wikimedia.org/dev/bookworm-php83-fpm from 1.0.0-s7 to 1.0.0-s9 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/742 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [19:26:09] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/andygrunwald/go-gerrit from 1.1.1 to 1.2.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/756 [19:26:12] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbletea from 1.2.4 to 1.3.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/755 [19:26:12] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbletea from 1.2.4 to 1.3.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/755 [19:26:31] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/andygrunwald/go-gerrit from 1.1.1 to 1.2.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/756 [19:26:32] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/andygrunwald/go-gerrit from 1.1.1 to 1.2.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/756 [19:27:11] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/gophercloud/gophercloud/v2 from 2.13.0 to 2.15.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/757 [19:27:26] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/gophercloud/gophercloud/v2 from 2.13.0 to 2.15.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/757 [19:27:26] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/gophercloud/gophercloud/v2 from 2.13.0 to 2.15.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/757 [19:28:39] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump gitlab.com/gitlab-org/api/client-go from 0.130.1 to 1.46.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/758 [19:28:46] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump gitlab.com/gitlab-org/api/client-go from 0.130.1 to 1.46.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/758 [19:29:06] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-ms-be10 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:29:06] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/crypto from 0.52.0 to 0.58.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/759 [19:29:09] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/crypto from 0.52.0 to 0.58.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/759 [19:29:13] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/crypto from 0.52.0 to 0.58.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/759 [19:29:14] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-ms-be10 in project deployment-prep - https://phabricator.wikimedia.org/T442932 (10wmcs-alerts) 03NEW [19:29:30] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/760 [19:29:39] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/760 [19:29:43] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/760 [19:29:55] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbles from 0.20.0 to 1.0.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/761 [19:30:12] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbles from 0.20.0 to 1.0.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/761 [19:30:12] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/bubbles from 0.20.0 to 1.0.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/761 [19:34:29] !log Ensure /srv Cinder volume is in /etc/fstab on MediaWiki hosts, deployment-jobrunner06 was broken after a reboot [19:34:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:34:42] (03CR) 10Ahmon Dancy: [C:03+1] "Looks good in local testing. Leaving to you to merge." [fresh] - 10https://gerrit.wikimedia.org/r/1347594 (https://phabricator.wikimedia.org/T439828) (owner: 10Hashar) [19:36:49] (03open) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s7 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/762 [19:36:52] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s7 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/762 [19:40:37] 06Release-Engineering-Team, 10Citoid, 06Collaboration-Services, 06Product Safety and Integrity, and 4 others: unattended upgrades are not upgrading libbrotli properly on some VMs - https://phabricator.wikimedia.org/T442936 (10Andrew) 03NEW [19:42:10] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12492083 (10Dzahn) Changed the logging_only rule to rate-limiting per IP. But there were many IPs so the busiest of them was not that busy. Then changed to deny outright.... [19:45:21] 06Release-Engineering-Team, 10Citoid, 06Collaboration-Services, 06Product Safety and Integrity, and 4 others: unattended upgrades are not upgrading libbrotli properly on some VMs - https://phabricator.wikimedia.org/T442936#12492089 (10Andrew) Here's an example: ` root@deployment-urldownloader04:~# apt pol... [19:48:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-jobrunner06 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:48:12] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-jobrunner06 in project deployment-prep - https://phabricator.wikimedia.org/T442937 (10wmcs-alerts) 03NEW [19:49:06] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-ms-be10 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:49:24] 06Release-Engineering-Team, 10Citoid, 06Collaboration-Services, 06Product Safety and Integrity, and 4 others: unattended upgrades are not upgrading libbrotli properly on some VMs - https://phabricator.wikimedia.org/T442936#12492108 (10Andrew) Running this: ` sudo cumin --force O{*} "apt install -y --only... [19:50:10] Southparkfan: The database on the primary database server also needs read-only mode turned off manually after a reboot. [19:50:43] (03merge) 10addshore: build(deps): bump github.com/andygrunwald/go-gerrit from 1.1.1 to 1.2.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/756 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:51:06] (03merge) 10addshore: build(deps): bump github.com/mattn/go-isatty from 0.0.22 to 0.0.24 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/752 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:51:28] (03merge) 10addshore: build(deps): bump golang.org/x/term from 0.44.0 to 0.47.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/748 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:52:15] (03merge) 10addshore: build(deps): bump github.com/sirupsen/logrus from 1.9.4 to 1.10.2 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/751 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:52:48] (03merge) 10addshore: build(deps): bump github.com/charmbracelet/x/ansi from 0.11.7 to 0.11.9 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/747 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:53:06] RESOLVED: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-jobrunner06 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:54:49] dancy: Thanks, works now. [19:55:02] !log Set read_only=OFF on deployment-db15 after project-wide reboots [19:55:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:06:21] 06Release-Engineering-Team, 10Catalyst, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Restore ingress to patchdemo / catalyst - https://phabricator.wikimedia.org/T442664#12492179 (10brennen) > catalyst-qte and catalyst-dev have no PHP according to the audit script. That is a bit surprisin... [20:08:22] (03close) 10addshore: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s6 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/739 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:08:23] (03update) 10addshore: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s6 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/739 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:20:42] 06Release-Engineering-Team, 10Citoid, 06Collaboration-Services, 06Product Safety and Integrity, and 4 others: unattended upgrades are not upgrading libbrotli properly on some VMs - https://phabricator.wikimedia.org/T442936#12492249 (10sbassett) a:05sbassettβ†’03None [20:21:32] (03update) 10addshore: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s7 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/762 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:21:39] (03update) 10addshore: Bump docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia from v1.33.0 to v1.35.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/743 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:21:44] (03update) 10addshore: Bump mariadb from 11.4 to 11.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/740 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:27:03] (03merge) 10addshore: build(deps): bump github.com/stretchr/testify from 1.11.1 to 1.12.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/760 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [20:44:06] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-ms-be09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:44:12] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-ms-be09 in project deployment-prep - https://phabricator.wikimedia.org/T442945 (10wmcs-alerts) 03NEW [20:46:00] (03merge) 10addshore: Bump docker-registry.wikimedia.org/releng/quibble-bookworm-php83 from 1.22.0-s5 to 1.22.0-s7 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/762 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:46:22] (03merge) 10addshore: Bump docker-registry.wikimedia.org/repos/data-engineering/eventgate-wikimedia from v1.33.0 to v1.35.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/743 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [20:48:59] (03merge) 10addshore: build(deps): bump github.com/charmbracelet/bubbletea from 1.2.4 to 1.3.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/755 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [21:03:06] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:03:18] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://phabricator.wikimedia.org/T442946 (10wmcs-alerts) 03NEW [21:04:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-ms-be09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:48:15] (03PS2) 10Jforrester: Docker: [java*] Re-build to get latest openjdks with Oct 2028 security updates [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 [21:48:24] (03CR) 10Jforrester: "@hashar@free.fr, what do you think?" [integration/config] - 10https://gerrit.wikimedia.org/r/1360519 (owner: 10Jforrester) [22:07:29] (03open) 10addshore: Emit events from sandboxes (such as agents) more often. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/763 [22:08:09] (03update) 10addshore: Emit events from sandboxes (such as agents) more often. [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/763 [22:10:36] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476#12492670 (10Southparkfan) On the backend hosts, I am experiencing timeouts during creation of the... [22:11:32] Jeez. [22:11:51] Wanted to install OpenStack Swift on a Beta Cluster VM, started deep-diving into Debian Packaging and LDAP users instead [22:12:38] (03update) 10addshore: Draft: Expand existing platform checks for Docker contexts and self-update [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/719 (https://phabricator.wikimedia.org/T384914 https://phabricator.wikimedia.org/T403412) [22:12:57] TL;DR the existence of https://ldap.toolforge.org/user/swift horribly broke the installation of Swift packages, because attempts a `usermod` on an LDAP user [22:29:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-ms-be09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:29:13] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-ms-be10 in project deployment-prep - https://phabricator.wikimedia.org/T442957 (10wmcs-alerts) 03NEW [22:30:19] !log Add new ms-be hosts to profile::swift::storagehosts - T436476 [22:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:30:23] T436476: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 [22:31:08] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-ms-be10 in project deployment-prep - https://phabricator.wikimedia.org/T442957#12492750 (10Southparkfan) 05Openβ†’03Resolved a:03Southparkfan T436476 [22:35:06] !log Disable puppet on deployment-ms-fe04 (Swift ring manager) to allow manual changes to /etc/swift/hosts.yaml to be distributed w/o Puppet overwriting them - T436476 [22:35:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:38:40] 10Scap (SpiderPig πŸ•ΈοΈ): Allow deployment of single releases with Spiderpig deploy-service - https://phabricator.wikimedia.org/T442586#12492779 (10dancy) @Jelto Just confirming that we want to limit this to single releases, not allowing the user to select multiple releases? [22:48:36] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-ms-be10 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [22:50:26] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12492892 (10Dzahn) {F111291826} [23:01:27] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12492925 (10dduvall) [23:01:39] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12492926 (10dduvall) 05Openβ†’03Resolved