[01:05:51] (03open) 10denisse: Add repos/sre/grafana-plugins [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/185 (https://phabricator.wikimedia.org/T436056) [01:26:23] PROBLEM - jenkins_service_running on releases1003 is CRITICAL: PROCS CRITICAL: 3 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [01:27:23] RECOVERY - jenkins_service_running on releases1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [04:40:50] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/Disambiguator] Add MobileFrontend for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1337652 (owner: 10Umherirrender) [04:42:37] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/Disambiguator] Add MobileFrontend for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1337652 (owner: 10Umherirrender) [04:59:45] PROBLEM - zuul_service_running on contint1002 is CRITICAL: PROCS CRITICAL: 4 processes with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [05:00:45] RECOVERY - zuul_service_running on contint1002 is OK: PROCS OK: 2 processes with regex args bin/zuul-server https://www.mediawiki.org/wiki/Continuous_integration/Zuul [06:17:39] 10Continuous-Integration-Infrastructure, 10Castor, 07Spike, 06Test Platform (Arusha 30): Test out the https://github.com/git-pkgs/proxy/ for CI - https://phabricator.wikimedia.org/T437005#12296199 (10Peter) I did some quick test on an agent and got //429 Too Many Requests// trying to use it in Quibble. I'... [06:23:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:28:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:54:58] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12296257 (10Osnard) Thank you all for your explanation. I was on the wrong track, my apologies. I am not... [07:08:12] (03approved) 10jelto: Add repos/sre/grafana-plugins [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/185 (https://phabricator.wikimedia.org/T436056) (owner: 10denisse) [07:08:20] (03merge) 10jelto: Add repos/sre/grafana-plugins [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/185 (https://phabricator.wikimedia.org/T436056) (owner: 10denisse) [08:17:40] (03PS1) 10Robert Vogel: Review access change [extensions/BlueSpiceInsertTemplate] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337849 [08:18:03] (03CR) 10Robert Vogel: [V:03+2 C:03+2] Review access change [extensions/BlueSpiceInsertTemplate] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1337849 (owner: 10Robert Vogel) [09:53:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12296727 (10hashar) Bah, it is expired?? `counterexample root@integration-agent-docker-1079:~# puppet agent -tv Error: The certificate 'CN=Puppet CA: inte... [10:01:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12296752 (10hashar) On agent-docker-1079 I went to move away the old copy of the `ca.pem` using: ` mv /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/ssl... [10:35:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:38:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Integration Puppetserver CA about to expire - https://phabricator.wikimedia.org/T435319#12296884 (10hashar) 05Openβ†’03Resolved I have used cumin to move the `ca.pem`, ran the Puppet agent twice and it seems all fine now. https://grafa... [10:45:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:28:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:33:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:02:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:14:26] (03CR) 10Phedenskog: "Ive been testing this out on the 9999 agent and it been working for me. Let me know @hashar@free.fr what we need to do to test it out mor" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1335750 (https://phabricator.wikimedia.org/T436376) (owner: 10Phedenskog) [12:17:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:37:25] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12297205 (10ABran-WMF) [12:41:31] (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extension/OAuth] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1337913 (https://phabricator.wikimedia.org/T395817) [12:45:48] (03CR) 10Neriah: [C:03+1] "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1337913 (https://phabricator.wikimedia.org/T395817) (owner: 10Dreamy Jazz) [12:52:48] 10Phabricator-Bot-Requests, 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284 (10dcaro) 03NEW [12:54:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:09:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:22:12] 10Phabricator-Bot-Requests, 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297345 (10Aklapper) > An email address (which can be invalid but must be unique): dcaro@wikmimedia.org If the typo is not intentional (?), then this won't work //if/... [13:24:40] 10Phabricator-Bot-Requests, 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297353 (10dcaro) >>! In T437284#12297345, @Aklapper wrote: >> An email address (which can be invalid but must be unique): dcaro@wikmimedia.org > > If the typo is not... [13:40:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:47:35] (03CR) 10Phedenskog: [C:03+1] Require Python 3.11 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334901 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [13:55:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:03:00] (03CR) 10Hashar: [C:03+2] Run the next npm install during the browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1335750 (https://phabricator.wikimedia.org/T436376) (owner: 10Phedenskog) [14:03:16] (03CR) 10Hashar: [C:03+2] "Thank you Peter!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334901 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:03:28] (03CR) 10Hashar: [C:03+2] Use str.removesuffix() for clarity [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334924 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:03:40] (03CR) 10Hashar: [C:03+2] Use Parenthesized Context Managers [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334902 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:07:35] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12297531 (10thcipriani) >>! In T431845#12296257, @Osnard wrote: > Thank you all for your explanation. I w... [14:15:33] (03Merged) 10jenkins-bot: Run the next npm install during the browser tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/1335750 (https://phabricator.wikimedia.org/T436376) (owner: 10Phedenskog) [14:15:36] (03Merged) 10jenkins-bot: Require Python 3.11 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334901 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:15:38] (03Merged) 10jenkins-bot: Use str.removesuffix() for clarity [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334924 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:19:15] 10Phabricator (Upstream), 07Upstream: Phantom unread Phabricator notification - https://phabricator.wikimedia.org/T399599#12297652 (10matmarex) I have a phantom unread notification again as well. [14:20:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:20:13] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297654 (10Aklapper) a:05dcaroβ†’03Aklapper Created https://phabricator.wikimedia.org/p/ToolsPlatformBot/ Its API Token is in... [14:22:02] (03Merged) 10jenkins-bot: Use Parenthesized Context Managers [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334902 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:23:38] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297665 (10ToolsPlatformBot) Test from the bot account [14:24:08] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 06tools-platform-team: Bot account for tools-platform team task flows - https://phabricator.wikimedia.org/T437284#12297666 (10dcaro) 05Openβ†’03Resolved That worked :), closing, thanks a lot! [14:30:37] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Upstream: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319#12297699 (10Michael) I hadn't seen this error in a while, but today it happened again: ` 00:03:25.626 Finished in 11.85s 00:03:25.990 [... [14:36:17] (03PS1) 10Hashar: jjb: skip Quibble selenium tests under sqlite [integration/config] - 10https://gerrit.wikimedia.org/r/1337938 [14:36:34] (03CR) 10Hashar: [C:03+2] jjb: skip Quibble selenium tests under sqlite [integration/config] - 10https://gerrit.wikimedia.org/r/1337938 (owner: 10Hashar) [14:38:18] (03Merged) 10jenkins-bot: jjb: skip Quibble selenium tests under sqlite [integration/config] - 10https://gerrit.wikimedia.org/r/1337938 (owner: 10Hashar) [14:45:03] 06Release-Engineering-Team, 06Infrastructure-Foundations: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297 (10elukey) 03NEW [14:45:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:46:55] (03PS1) 10Hashar: release: Quibble 1.21.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337943 (https://phabricator.wikimedia.org/T429715) [14:47:05] (03PS1) 10Hashar: release: start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337944 [14:48:05] (03CR) 10Hashar: [C:03+2] release: start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337944 (owner: 10Hashar) [14:48:10] (03CR) 10Hashar: [C:03+2] release: Quibble 1.21.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337943 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [14:54:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:02:39] (03CR) 10Ottomata: [C:04-1] "> TLDR: I don't think EventBus should be coupled to Wikibase" [integration/config] - 10https://gerrit.wikimedia.org/r/1335904 (https://phabricator.wikimedia.org/T428176) (owner: 10Ottomata) [15:04:43] (03Merged) 10jenkins-bot: release: Quibble 1.21.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337943 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [15:05:57] (03Merged) 10jenkins-bot: release: start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1337944 (owner: 10Hashar) [15:06:16] !log Tag Quibble @ b692707e5213abf5e16894b1fb21cea41cd72c9e # T429715 T303270 T437134 T300727 [15:06:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:06:25] T429715: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715 [15:06:26] T303270: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 [15:06:26] T437134: Run only the PHPUnit Database tests in the sqlite and postgres core jobs to speed up the gated pipeline - https://phabricator.wikimedia.org/T437134 [15:06:26] T300727: Rename cmd.py to distinguish from Command - https://phabricator.wikimedia.org/T300727 [15:08:10] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1337948 [15:08:25] 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715#12297951 (10hashar) 05Openβ†’03Resolved a:03hashar Quibble 1.21.0 now requires Python 3.11. I did not go with the new dict merge/update, pattern matching or... [15:09:14] (03PS1) 10Hashar: jjb: update Quibble jobs for 1.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1337949 (https://phabricator.wikimedia.org/T429715) [15:09:30] (03CR) 10Hashar: [C:03+2] dockerfiles: update Quibble to 1.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1337948 (owner: 10Hashar) [15:11:21] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1337948 (owner: 10Hashar) [15:12:08] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:12:19] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12297979 (10dancy) >>! In T437069#12294166, @MoritzMuehlenhoff wrote: > bullseye-security was restored on the Debian archive s... [15:21:34] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12298009 (10MoritzMuehlenhoff) This is now reported as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1147093 [15:24:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:30:19] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Please un-archive the "New Private Task (WMF-FR)" Phabricator form - https://phabricator.wikimedia.org/T437067#12298070 (10Ejegg) Thank you @Aklapper! [15:32:15] (03CR) 10Hashar: [C:04-1] Zuul: Run WikibaseClient job for EventBus (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1335904 (https://phabricator.wikimedia.org/T428176) (owner: 10Ottomata) [15:40:13] 10Phabricator, 06Collaboration-Services: Querying Phabricator DB via phab1005 fails with TLS errors - https://phabricator.wikimedia.org/T437242#12298192 (10Urbanecm) 05Openβ†’03Resolved Now it works: ` [urbanecm@phab1005 ~]$ mysql Reading table information for completion of table and column names You ca... [15:42:23] !log Updating Quibble jobs to 1.21.0 # T429715 T303270 T437134 T300727 [15:42:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:30] T429715: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715 [15:42:31] T303270: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270 [15:42:31] T437134: Run only the PHPUnit Database tests in the sqlite and postgres core jobs to speed up the gated pipeline - https://phabricator.wikimedia.org/T437134 [15:42:32] T300727: Rename cmd.py to distinguish from Command - https://phabricator.wikimedia.org/T300727 [15:44:06] (03merge) 10dancy: digitalocean: create dedicated vpc and nat_gateway for each cluster [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/629 (https://phabricator.wikimedia.org/T425441) (owner: 10jelto) [15:45:01] 10Phabricator, 06Collaboration-Services: Querying Phabricator DB via phab1005 fails with TLS errors - https://phabricator.wikimedia.org/T437242#12298251 (10Urbanecm) Curiosity question...why have we decided to skip certificate verification altogether? Wouldn't it make sense to make the cert trusted, so tha... [15:45:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:47:09] (03open) 10dancy: Revert "digitalocean: create dedicated vpc and nat_gateway for each cluster" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/630 [15:48:15] (03merge) 10dancy: Revert "digitalocean: create dedicated vpc and nat_gateway for each cluster" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/630 [15:50:34] (03CR) 10Hashar: [C:03+2] "INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 223" [integration/config] - 10https://gerrit.wikimedia.org/r/1337949 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [15:53:11] (03Merged) 10jenkins-bot: jjb: update Quibble jobs for 1.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1337949 (https://phabricator.wikimedia.org/T429715) (owner: 10Hashar) [15:55:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:57:49] (03PS2) 10Hashar: jjb: Run only database tests in the sqlite and postgres core jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [15:59:54] (03CR) 10Hashar: [C:03+1] "I have updated the Quibble jobs to Quibble 1.21.0 which has the `phpunit-database` stage." [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [16:13:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:15:17] 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Failing branch cut job - https://phabricator.wikimedia.org/T437184#12298548 (10dancy) Thanks @jnuche! I was wondering what was going on there. [16:21:51] !log hard reboot deployment-prep:deployment-cache-text08 horizon logs showing OOMs [16:21:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:31:22] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Calendar monthly view makes event names unreadable - https://phabricator.wikimedia.org/T187019#12298717 (10Aklapper) [16:31:24] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12298718 (10Aklapper) [16:32:04] (03update) 10dancy: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419) [16:32:07] (03update) 10dancy: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419) [16:32:33] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Calendar monthly view makes event names unreadable - https://phabricator.wikimedia.org/T187019#12298730 (10Aklapper) 05Openβ†’03Stalled [16:38:23] (03PS1) 10Hashar: jjb: remove dupe --reporting-url from Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337963 [16:40:59] (03PS2) 10Hashar: jjb: remove dupe --reporting-url from Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337963 (https://phabricator.wikimedia.org/T323750) [16:41:01] (03PS3) 10Hashar: jjb: Run only database tests in the sqlite and postgres core jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [16:43:54] (03CR) 10Hashar: "Your original patch had:" [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [16:44:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:59:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:13:03] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extension/OAuth] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1337913 (https://phabricator.wikimedia.org/T395817) (owner: 10Dreamy Jazz) [17:13:38] (03open) 10jforrester: releases: Bump eslint-config-wikimedia to 0.32.6 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/145 [17:14:46] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extension/OAuth] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1337913 (https://phabricator.wikimedia.org/T395817) (owner: 10Dreamy Jazz) [17:14:50] (03merge) 10jforrester: releases: Bump eslint-config-wikimedia to 0.32.6 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/145 [17:22:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:30:10] (03update) 10swfrench: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419) (owner: 10dancy) [17:42:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:42:22] 10Phabricator, 06Collaboration-Services: Querying Phabricator DB via phab1005 fails with TLS errors - https://phabricator.wikimedia.org/T437242#12299207 (10Arnoldokoth) My assumption would be that since the connection between the `phab` host and `db` host happens on a trusted private network there is a low... [17:57:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:05:43] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 06Machine-Learning-Team, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12299322 (10sbassett) p:05Mediumβ†’03Low @ABran-WMF et al - Thanks for filing... [18:06:31] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: The PHPUnit config override does not appear to be auto-generated - https://phabricator.wikimedia.org/T419107#12299325 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Daimona All done, n... [18:08:26] (03CR) 10Jforrester: [C:03+1] "Makes sense." [integration/config] - 10https://gerrit.wikimedia.org/r/1335807 (https://phabricator.wikimedia.org/T436461) (owner: 10Phedenskog) [18:11:02] 10WikimediaDebug, 10Wikimedia-Site-requests, 07Datacenter-Switchover, 06MediaWiki-Core-Platform-Team (Radar): Have noc's debug.json server list automatically sort primary DC servers first - https://phabricator.wikimedia.org/T289745#12299358 (10Krinkle) [18:17:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:39:18] (03open) 10dancy: spiderpig: Keep the navigation bar in step with the page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1316 (https://phabricator.wikimedia.org/T433287) [18:40:28] 06Release-Engineering-Team, 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: Spiderpig tabs get out of sync when using browser back button - https://phabricator.wikimedia.org/T433287#12299519 (10dancy) Thanks for the very clear bug report @taavi! [19:27:14] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 06Machine-Learning-Team, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12299797 (10Dzahn) This is one of those things that people mean when talking ab... [19:31:55] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#12299825 (10Dzahn) phab1005 and phab1006 are on trixie phab2003 is on bookworm [19:32:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Collaboration-Services: Upgrade phab (phorge) hosts to bookworm - https://phabricator.wikimedia.org/T372619#12299827 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn [19:33:23] (03PS1) 10Phedenskog: jjb: Overlap npm install with the gated browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/1337439 (https://phabricator.wikimedia.org/T436376) [19:47:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Back up Zuul encryption keys - https://phabricator.wikimedia.org/T436131#12299910 (10Dzahn) a:03Dzahn [19:52:21] dancy: Hello! Regarding https://phabricator.wikimedia.org/T432989#12298473; do you need all of the MediaWiki hosts (maintenance, appservers, deployment) to be running on PHP 8.5 before proceeding with the next steps? [19:54:01] I assume a change to switch the `mediawiki_runtime_age` in Scap to a PHP 8.5-based one is part of the overall PHP upgrade process, but I am not sure if this should be done before(!) or after(!) switching the rest of Beta Cluster to PHP 8.5. [19:54:13] err, `mediawiki_runtime_image` [20:10:06] Southparkfan: I think it should be safe to update `mediawiki_runtime_image` (which is only used on the deploy server) before other targets are migrated. [20:11:00] or after, for that matter. [20:15:28] 06Release-Engineering-Team, 06Collaboration-Services, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Matomo K8s migration: Create OCI (Docker) image - https://phabricator.wikimedia.org/T436005#12300096 (10bking) 05In progressβ†’03Resolved We have a working image that includes Matomo and the GPL-license... [20:20:50] Southparkfan: I prepared https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338027. I'll cherry-pick it to beta's puppetserver and see how it goes. [20:27:28] dancy: Thanks. Haven't tested, but +1 for the approach. [20:27:45] Testing is in progress (via the wmf-beta-update-all timer) [20:28:47] Run can run `tail-beta-update-logs` on `deployment-deploy04` [20:29:50] The parts `scap sync-world` that use that image have completed, so it looks good. [20:29:54] *parts of [20:30:06] Yay. [20:30:10] Cool! [20:30:40] https://phabricator.wikimedia.org/T437354 is the only new error I have spotted after temporarily running both appservers on PHP 8.5. [20:31:29] I'm tempted to announce the PHP 8.3 -> PHP 8.5 switch on wikitech-l and switch the maintenance hosts, jobrunner and both appservers right away. Feel free to endorse or reject this proposal :-) [20:33:45] WCPGW? [20:34:33] Southparkfan: As long as you're willing to hang out for a bit to monitor and switch back if something goes way off the rails, I endorse the proposal. [20:41:37] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592#12300231 (10komla) @Jdforrester-WMF any luck with progress on this? [20:44:20] (03update) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) [20:44:35] (03update) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) [20:58:13] Alright, in that case I'll reserve some time in my calendar and we'll "be bold" whenever I'm ready :-) [21:15:40] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12300326 (10Dzahn) please track decom in dedicated tasks - created from template for decoms [21:15:49] 06Release-Engineering-Team, 06Collaboration-Services, 07Essential-Work: Phabricator Switchover - from phab1004 (Bullseye, PHP 7.4) to phab1005 (Trixie, PHP 8.4) - https://phabricator.wikimedia.org/T435087#12300328 (10Dzahn) congrats on the switchover :) [21:28:52] (03open) 10denisse: Add repos/sre/klaxon [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/186 (https://phabricator.wikimedia.org/T437001) [21:46:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services, 13Patch-For-Review: Addition of zuul1004 broke zuul-eqiad zookeeper - https://phabricator.wikimedia.org/T435186#12300481 (10Dzahn) back from PTO - ready to help get this fixed [22:52:15] (03update) 10swfrench: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [22:52:17] (03approved) 10swfrench: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) (owner: 10dancy) [23:37:46] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Arusha 30): Keep npm debug logs out of the castor cache - https://phabricator.wikimedia.org/T437376 (10Mhurd) 03NEW [23:37:54] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Arusha 30): Run npm cache verify once a day per cache instead of on every build - https://phabricator.wikimedia.org/T437377 (10Mhurd) 03NEW [23:41:38] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Arusha 30): Run npm cache verify once a day per cache instead of on every build - https://phabricator.wikimedia.org/T437377#12300915 (10Mhurd) [23:41:42] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Arusha 30): Keep npm debug logs out of the castor cache - https://phabricator.wikimedia.org/T437376#12300917 (10Mhurd)