[03:12:24] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12181588 (10tstarling) >>! In T431845#12148826, @Osnard wrote: > Bumping dozens or hundreds of versions i... [08:29:25] 06Project-Admins: Create project tag for MediaWiki-Core-Parsoid - https://phabricator.wikimedia.org/T433925 (10MSantos) 03NEW [08:38:06] 10Gerrit, 06collaboration-services: Backup Gerrit with LVM snapshots and Bacula pre/post job hooks - https://phabricator.wikimedia.org/T433926 (10ABran-WMF) 03NEW [10:13:49] 06Project-Admins: Create project tag for WikiChanges - https://phabricator.wikimedia.org/T433949 (10ItsNyoty) 03NEW [10:31:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:31:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T433951 (10phaultfinder) 03NEW [10:36:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:05:48] (03CR) 10Hashar: [C:03+2] build: Updating brace-expansion to 1.1.18, 2.1.4 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1320244 (owner: 10Libraryupgrader) [11:28:31] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:28:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T433951#12182825 (10phaultfinder) [11:33:31] RESOLVED: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:32:36] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2026-08-04 - https://phabricator.wikimedia.org/T433981 (10brennen) 03NEW [14:33:10] (03open) 10brennen: update submodules for 2026-08-04 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/114 (https://phabricator.wikimedia.org/T433981) [14:37:21] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-08-04 - https://phabricator.wikimedia.org/T433981#12183701 (10Aklapper) a:03brennen This deployment includes: * Phabricator: Added debug spew in https://gitlab.wik... [14:38:50] (03merge) 10brennen: update submodules for 2026-08-04 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/114 (https://phabricator.wikimedia.org/T433981) [14:50:52] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: Update gitlab cookbooks and processes - https://phabricator.wikimedia.org/T430655#12183771 (10ABran-WMF) a:03ABran-WMF [14:51:02] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: Update gitlab cookbooks and processes - https://phabricator.wikimedia.org/T430655#12183773 (10ABran-WMF) 05Open→03In progress [15:08:54] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.32.2 released - https://phabricator.wikimedia.org/T433985 (10dancy) 03NEW [15:11:31] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:11:41] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T433986 (10phaultfinder) 03NEW [15:16:31] RESOLVED: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:17:35] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12183884 (10pwangai) @Dzahn This will not involve sudo, I will be running a few scripts under my username to summarize the log data we need. [15:25:28] 06Release-Engineering-Team, 06collaboration-services: Deploying Phab/Phorge in devtools to new Trixie instance does not restart PHP 8.4 - https://phabricator.wikimedia.org/T433988 (10brennen) 03NEW [15:25:50] 06Release-Engineering-Team, 06collaboration-services: Deploying Phab/Phorge in devtools to new Trixie instance does not restart PHP 8.4 - https://phabricator.wikimedia.org/T433988#12183918 (10Aklapper) Puppet has `./modules/phabricator/templates/phab_deploy_finalize.sh.erb: $systemctl restart php7.4-fpm.service` [15:47:14] 10Phabricator (2026-08-04), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2026-08-04 - https://phabricator.wikimedia.org/T433981#12184007 (10Aklapper) 05Open→03Resolved [15:56:26] (03update) 10dancy: Add scap deploy-service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1237 (https://phabricator.wikimedia.org/T412941) [15:58:25] (03update) 10dancy: Add scap deploy-service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1237 (https://phabricator.wikimedia.org/T412941) [16:19:02] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998 (10bd808) 03NEW [16:28:32] (03PS1) 10Jforrester: Docker: [php83] Upgrade PHP to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321008 [16:30:23] (03CR) 10Jforrester: [C:03+2] Docker: [php83] Upgrade PHP to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321008 (owner: 10Jforrester) [16:32:09] (03Merged) 10jenkins-bot: Docker: [php83] Upgrade PHP to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321008 (owner: 10Jforrester) [16:39:19] 10Gerrit, 06Release-Engineering-Team (Seen), 06collaboration-services, 06SRE, 07Security: gerrit.wikimedia.org:29418 uses 1024-bit RSA key only - https://phabricator.wikimedia.org/T240266#12184308 (10Dzahn) >>! In T240266#12152251, @Bawolff wrote: > The announcement email kind of directs users here.... [16:43:31] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12184321 (10Dzahn) @pwangai Cool! in that case I think the simplest path forward is we just add people to "contint-users" which gives unprivileged shell ac... [16:43:42] (03open) 10jforrester: releases: Bump @wdio/* to 9.30.1 (9.30.0 was unpublished for junit), drop devtools-service [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/142 [16:47:51] (03approved) 10taavi: releases: Bump @wdio/* to 9.30.1 (9.30.0 was unpublished for junit), drop devtools-service [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/142 (owner: 10jforrester) [16:48:46] !log Docker: [php83] Upgrade PHP to 8.3.33 [16:48:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:57] (03merge) 10jforrester: releases: Bump @wdio/* to 9.30.1 (9.30.0 was unpublished for junit), drop devtools-service [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/142 [17:02:52] (03PS1) 10Jforrester: jjb: Update PHP83 jobs to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321019 [17:07:16] (03CR) 10Jforrester: [C:03+2] jjb: Update PHP83 jobs to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321019 (owner: 10Jforrester) [17:09:22] (03Merged) 10jenkins-bot: jjb: Update PHP83 jobs to 8.3.33 [integration/config] - 10https://gerrit.wikimedia.org/r/1321019 (owner: 10Jforrester) [17:21:55] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12184511 (10bd808) I would like us to try adopting the {T394316} idea as a core part of the migration.... [17:26:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:26:09] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T434005 (10wmcs-alerts) 03NEW [17:26:18] 10Gerrit, 06Release-Engineering-Team (Seen), 06collaboration-services, 06SRE, 07Security: gerrit.wikimedia.org:29418 uses 1024-bit RSA key only - https://phabricator.wikimedia.org/T240266#12184535 (10dancy) @dzahn Congratulations! [17:26:25] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Cloud-VPS (Debian Bullseye Deprecation): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12184537 (10bd808) 05Open→03In progress @Southparkfan reached out to me yesterday to ask if they mi... [18:10:50] 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team (Kanban), 07OKR-Work: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12184735 (10MusikAnimal) 05Open→03Resolved [18:13:50] Is the gerrit index out-of-sync? E.g. https://gerrit.wikimedia.org/r/q/hashtag:%22n%E2%80%94grunt%3D1.6.3%22+is:open+-age:5h lists https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PageExchange/+/1320979 as C+2 V0, but visiting that page I see C+2 V+2 [18:15:49] (03PS1) 10Pwangai: castor: Log load failures and rsync stats [integration/config] - 10https://gerrit.wikimedia.org/r/1321038 (https://phabricator.wikimedia.org/T433915) [18:16:03] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:19:47] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-upload08 in project deployment-prep - https://phabricator.wikimedia.org/T434005#12184780 (10BCornwall) 05Open→03Resolved a:03BCornwall Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/13... [18:26:55] (03PS1) 10Pwangai: castor: Log load failures and rsync stats [integration/config] - 10https://gerrit.wikimedia.org/r/1321038 (https://phabricator.wikimedia.org/T433915) [18:47:32] (03PS1) 10Pwangai: castor: Add rsync stats to the cache save [integration/config] - 10https://gerrit.wikimedia.org/r/1321049 (https://phabricator.wikimedia.org/T433915) [18:57:04] (03update) 10dancy: Bump buildkitd to v0.32.2 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/625 (https://phabricator.wikimedia.org/T433985) [18:57:13] (03open) 10dancy: Bump buildkitd to v0.32.2 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/625 (https://phabricator.wikimedia.org/T433985) [18:58:13] (03merge) 10dancy: Bump buildkitd to v0.32.2 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/625 (https://phabricator.wikimedia.org/T433985) [18:59:03] FIRING: InstanceDown: Project deployment-prep instance deployment-eventstreams-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:59:12] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-eventstreams-3 is down - https://phabricator.wikimedia.org/T434022 (10wmcs-alerts) 03NEW [19:00:30] James_F: fixed. i think [19:01:18] (ssh -p 29418 gerrit.wikimedia.org gerrit index changes 1320979) [19:06:23] !log Updating buildkitd to v0.32.2 on gitlab-cloud-runners (staging and production) (T433985) [19:06:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:06:25] T433985: Buildkit v0.32.2 released - https://phabricator.wikimedia.org/T433985 [19:06:42] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.32.2 released - https://phabricator.wikimedia.org/T433985#12184962 (10dancy) 05Open→03In progress p:05Triage→03High a:03dancy [19:14:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-eventstreams-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:26:03] 10Phabricator Translations Extension, 06translatewiki.net: Phabricator translations needs i18n check reviewed again - https://phabricator.wikimedia.org/T434026 (10Pppery) 03NEW [19:41:54] 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12185094 (10Lars) [19:53:28] (03open) 10dancy: Record the staged checkouts in deployment-info.json [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1238 [19:58:44] (03update) 10dancy: Record the staged checkouts in deployment-info.json [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1238 [20:34:05] mutante: Thanks! [21:03:39] thx mutante [21:51:07] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.32.2 released - https://phabricator.wikimedia.org/T433985#12185401 (10dancy) 05In progress→03Resolved [23:13:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12185567 (10greg) [23:53:32] (03update) 10swfrench: Add scap deploy-service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1237 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [23:53:36] (03approved) 10swfrench: Add scap deploy-service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1237 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy)