[00:04:48] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox image group to 2026-11-17-150318 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/823 [00:04:51] (03update) 10project_16_bot_841eade40a22c56a6272172b8cb67384: Bump docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox image group to 2026-11-17-150318 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/823 [00:14:22] (03merge) 10addshore: build(deps): bump github.com/PuerkitoBio/goquery from 1.13.0 to 1.14.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/824 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [00:18:51] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [00:22:38] (03open) 10dduvall: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) [00:22:40] (03update) 10dduvall: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) [00:25:25] (03update) 10dduvall: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) [00:29:21] (03update) 10dduvall: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) [00:40:54] 10Beta-Cluster-Infrastructure: unblock.py crashes when network picked to unblock is superset of blocked network - https://phabricator.wikimedia.org/T445124#12579328 (10bd808) 05In progress→03Resolved [00:47:49] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf31 in project deployment-prep - https://phabricator.wikimedia.org/T443914#12579331 (10bd808) 05Open→03Resolved a:03bd808 disk was full because of `navtiming` not being able to find a kafka cluster. Will m... [00:47:59] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12579342 (10SecurityPatchBot) [00:48:08] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12579353 (10SecurityPatchBot) [00:48:37] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-webperf31 on project deployment-prep - https://phabricator.wikimedia.org/T443255#12579355 (10bd808) 05Open→03Resolved a:03bd808 [00:54:36] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12579364 (10Jdforrester-WMF) p:05Triage→03Medium [00:58:25] 06Project-Admins, 10database-orchestrator, 06DBA: Re-label "#orchestrator" to be less generic so it's less confusing - https://phabricator.wikimedia.org/T444624#12579384 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF >>! In T444624#12565283, @Marostegui wrote: > I think it is fine to dro... [01:05:02] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-webperf32 is down - https://phabricator.wikimedia.org/T436707#12579417 (10bd808) 05Open→03Invalid instance deleted [01:06:51] 10Beta-Cluster-Infrastructure: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12579435 (10bd808) p:05Triage→03High Disk on deployment-webperf31 is filling up because of the non-stop log spam from `navtiming`. [01:07:07] 10Beta-Cluster-Infrastructure: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12579437 (10bd808) [01:37:25] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12579599 (10thcipriani) >>! In T442224#12579187, @Ladsgroup wrote: > ##### Risky Patch! 🚂🔥 [snip] > * **Summary**: > ** We have re... [01:40:53] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579616 (10bd808) `lang=shell-session,counterexample bd808@deployment-cp-upload09:~$ sudo run-puppet-agent Info: Using environment '... [01:46:13] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579626 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/95a730d04a511a77945c03306b8a05c8504061ad%5E... [01:46:29] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579627 (10bd808) p:05Triage→03High [01:50:42] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579670 (10ssingh) @Fabfur: can you look into this please? I think we just need to do this but I may be wrong. ` <%- if @use_kapow_... [01:51:21] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12579671 (10bd808) The `before => [Service['haproxy'], $site_resource],` constraint on `file { $kapow_score_file:` seems to be the... [06:03:44] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12580218 (10KartikMistry) [08:05:58] 06Release-Engineering-Team, 06Collaboration-Services: Update sync-gitlab-group-with-ldap to handle syncing users who have no email address - https://phabricator.wikimedia.org/T414428#12580379 (10ABran-WMF) [08:13:44] 10Gerrit, 06Collaboration-Services: Investigate missing commit from old changes - https://phabricator.wikimedia.org/T388507#12580392 (10ABran-WMF) a:03ABran-WMF [08:41:55] We are doing Gerrit maintenance between 08:00–09:00 UTC [09:17:06] (03CR) 10Hashar: [C:03+2] Gerrit 3.11.x rebuild plugins and update TypeScript API [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360417 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [09:17:35] (03Merged) 10jenkins-bot: Gerrit 3.11.x rebuild plugins and update TypeScript API [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360417 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [10:09:30] 10MediaWiki-Releasing, 07Security: Release MediaWiki 1.43.10/1.45.5/1.46.1 - https://phabricator.wikimedia.org/T429197#12580787 (10Reedy) 05Open→03Resolved a:03Reedy [10:10:29] (03update) 10kbach: feat(REST): Add Spectral rules for API module summary length [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/19 (https://phabricator.wikimedia.org/T434253) (owner: 10kineticpelagic) [10:10:54] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12580792 (10hashar) 05Open→03Resolved We have rolled the Gerrit 3.11 upgrade with @ABran-WMF @jelto @jnuche. Se... [10:28:46] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06Collaboration-Services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#12580854 (10ABran-WMF) [10:50:26] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12581012 (10Fabfur) a:03Fabfur [11:10:11] (03merge) 10addshore: Bump docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox image group to 2026-11-17-150318 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/823 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [11:50:22] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12581429 (10Lucas_Werkmeister_WMDE) Still happens on Gerrit 3.11; perhaps some config needs to be up... [11:53:14] 10Gerrit (Gerrit 3.11), 07Upstream: Syntax highlightning of Embedded Puppet (EPP) files in Gerrit - https://phabricator.wikimedia.org/T339096#12581432 (10hashar) 05Open→03Resolved a:03Paladox We have upgraded to Gerrit 3.11: > [[ https://gerrit-review.googlesource.com/c/gerrit/+/426018 | Change 4260... [11:56:17] 10Gerrit (Gerrit 3.11): Add support for Vue language in Gerrit code editor - https://phabricator.wikimedia.org/T380668#12581462 (10hashar) 05Open→03Resolved a:03Paladox We have upgraded to Gerrit 3.11: > [[ https://gerrit-review.googlesource.com/c/gerrit/+/438424 | Change 438424 ]]: Add support for Vu... [12:38:00] 10Gerrit, 10Phabricator: Have Gerrit show an hovercard for Phabricator tasks links - https://phabricator.wikimedia.org/T445516 (10hashar) 03NEW [12:41:28] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12581823 (10Paladox) Oh I think this might be fixed with https://github.com/apache/mina-sshd/commit/... [12:42:46] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12581832 (10Paladox) Gerrit master works for me locally with ssh without triggering that warning. [12:46:54] 10Gerrit, 10Phabricator: Have Gerrit show an hovercard for Phabricator tasks links - https://phabricator.wikimedia.org/T445516#12581878 (10hashar) [12:50:48] 10Gerrit, 10Phabricator: Have Gerrit show an hovercard for Phabricator tasks links - https://phabricator.wikimedia.org/T445516#12581890 (10hashar) [12:50:55] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 32): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12581892 (10JVanderhoop-WMF) What I'm hearing from you @DLynch and @Jdlrobson-WMF is not a "don't make... [12:50:57] https://phabricator.wikimedia.org/F115982884 :b [12:51:50] filed https://phabricator.wikimedia.org/T445516 for Gerrit to look at whether we can have a card to show up when being over a link to a Phabricator task [12:55:42] 10Gerrit: Unable to remove reviewer - https://phabricator.wikimedia.org/T413309#12581981 (10Paladox) UI seems to have been fixed in 3.11. See https://gerrit-review.googlesource.com/c/gerrit/+/420064 [12:58:58] 10Gerrit: Enforce project configuration changes for code review - https://phabricator.wikimedia.org/T445067#12582019 (10hashar) [13:00:02] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Update gerrit replication plugin with new metrics - https://phabricator.wikimedia.org/T418215#12582041 (10hashar) We have upgraded to Gerrit 3.11, so this is back to the general queue. [13:00:40] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12582043 (10Jdforrester-WMF) [13:09:49] (03update) 10reedy: deployment-calendar: Send a descriptive user-agent to Phabricator [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/264 (owner: 10bd808) [13:10:23] (03update) 10reedy: deployment-calendar: Send a descriptive user-agent to Phabricator [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/264 (owner: 10bd808) [13:11:49] (03merge) 10reedy: deployment-calendar: Send a descriptive user-agent to Phabricator [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/264 (owner: 10bd808) [13:14:21] hashar: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1371818 [13:14:52] (03update) 10reedy: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) (owner: 10mstyles) [13:15:17] (03update) 10reedy: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) (owner: 10mstyles) [13:15:37] (03update) 10reedy: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) (owner: 10mstyles) [13:16:02] (03merge) 10reedy: Add scripts for reserving CVE IDs and listing the tasks of a security release [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/285 (https://phabricator.wikimedia.org/T443837) (owner: 10mstyles) [13:18:53] (03merge) 10addshore: Bump mariadb from 11.4 to 11.8 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/740 (owner: 10project_16_bot_841eade40a22c56a6272172b8cb67384) [13:20:10] (03update) 10reedy: Add a script for checking which security patches can't be backported [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/282 (owner: 10catrope) [13:22:21] (03PS1) 10Hashar: wm-pcc: add a Rerun callback on CheckRun [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1369875 [13:23:02] (03CR) 10Hashar: [C:03+2] wm-pcc: add a Rerun callback on CheckRun [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1369875 (owner: 10Hashar) [13:23:30] (03Merged) 10jenkins-bot: wm-pcc: add a Rerun callback on CheckRun [software/gerrit] (deploy/wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1369875 (owner: 10Hashar) [13:35:36] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Aktau 32): Let CI jobs fetch npm packages through the cache proxy, with Castor fallback - https://phabricator.wikimedia.org/T444894#12582226 (10AMarkossyan-WMF) [13:35:38] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Aktau 32): Let CI jobs fetch Composer packages through the cache proxy, with Castor fallback - https://phabricator.wikimedia.org/T444738#12582228 (10AMarkossyan-WMF) [13:37:45] 10Gerrit, 07Upstream: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#12582260 (10Paladox) >>! In T395324#10859384, @hashar wrote: > At a quick glance, it does not like the semicolon in the string inside an anonymous function. That makes the bug very s... [13:38:18] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (San Miguel de Allende 31): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12582268 (10AMarkossyan-WMF) 05Open→03Resolved [13:38:30] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#12582272 (10Lucas_Werkmeister_WMDE) [13:38:34] 10Gerrit, 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12582271 (10Lucas_Werkmeister_WMDE) [13:42:29] 10Gerrit, 07Upstream: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#12582292 (10Paladox) @hashar your fix https://github.com/highlightjs/highlight.js/commit/d86474408778cbb8f0c0a37404512bdb070c181d is included in 11.11.2 and 11.12. But gerrit 3.15 u... [14:06:34] 10Gerrit, 07Upstream: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#12582481 (10Paladox) Oh this does seem to work at least on https://gerrit-review.googlesource.com/ {F115991154} [14:06:35] 10Gerrit (Gerrit 3.12), 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12582483 (10hashar) [14:27:17] someone on the cloud list was asking about development-metrics still being down, I directed it to https://phabricator.wikimedia.org/T443279 since it seems to have at least been noticed by you [14:32:40] phabricator seems to be having a moment [14:32:56] https://usercontent.irccloud-cdn.com/file/CLq4VgIH/image.png [14:41:47] phab should be back [14:53:30] Project mediawiki-core-doxygen build #24436: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24436/ [14:56:37] ty taavi [14:57:22] !log Updating development images on contint primary for PHP updates [14:57:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:06:11] (03PS1) 10Jforrester: Docker: [php83, php85] Update PHP to 8.3.37 / 8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371857 [15:13:18] 10Beta-Cluster-Infrastructure, 06Content-Platform-Team, 10MediaWiki-extensions-PdfHandler: Migrate deployment-prep PdfHandler to shellbox - https://phabricator.wikimedia.org/T445330#12582813 (10OSleger-WMF) a:03TheDJ [15:16:52] (03open) 10jnuche: php images: rebuild to pick up latest changes [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/138 [15:17:11] (03merge) 10jnuche: php images: rebuild to pick up latest changes [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/138 [15:17:50] !log Updating development images on contint primary for PHP updates (2nd attempt) [15:17:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:21:31] (03CR) 10Jforrester: [C:03+2] Docker: [php83, php85] Update PHP to 8.3.37 / 8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371857 (owner: 10Jforrester) [15:28:32] (03Merged) 10jenkins-bot: Docker: [php83, php85] Update PHP to 8.3.37 / 8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371857 (owner: 10Jforrester) [15:28:54] !log Docker: [php83, php85] Update PHP to 8.3.37 / 8.5.13 [15:28:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:36:58] Project mediawiki-core-phpmetrics build #923: 04FAILURE in 2 min 57 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/923/ [15:39:39] (03PS1) 10Jforrester: jjb: Upgrade to PHP 8.3.37/8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371868 [15:40:16] (03open) 10dancy: add-project.py: Reject repo names with a trailing slash [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/201 [15:40:17] (03update) 10dancy: add-project.py: Reject repo names with a trailing slash [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/201 [15:41:25] (03merge) 10dancy: add-project.py: Reject repo names with a trailing slash [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/201 [15:42:12] (03update) 10dancy: add sbom-scanner [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/200 (owner: 10lizf) [15:44:54] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#12583119 (10zeljkofilipin) 05Open→03In progress p:05Triage→03High a:03zeljkofilipin [15:46:50] 10Gerrit: Unchanged file with unresolved comment doesn't show up in file list - https://phabricator.wikimedia.org/T362269#12583144 (10Paladox) Does this work for you? Gerrit has been upgraded to 3.11. [15:49:14] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#12583172 (10vaughnwalters) [15:49:18] 10Gerrit, 07Mobile, 07Upstream: Missing sign in links on mobile - https://phabricator.wikimedia.org/T344891#12583174 (10Paladox) Gerrit has been upgraded to 3.11 bringing a more mobile optimised navigation bar: {F116001929} [15:49:33] 10Gerrit, 07Mobile, 07Upstream: Missing sign in links on mobile - https://phabricator.wikimedia.org/T344891#12583176 (10Paladox) 05Open→03Resolved a:03Paladox [15:50:44] (03update) 10dancy: add sbom-scanner [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/200 (owner: 10lizf) [15:50:48] (03update) 10dancy: add sbom-scanner [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/200 (owner: 10lizf) [15:50:56] (03update) 10dancy: add sbom-scanner [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/200 (owner: 10lizf) [15:51:12] (03merge) 10dancy: add sbom-scanner [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/200 (owner: 10lizf) [15:58:34] (03update) 10dancy: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) (owner: 10dduvall) [15:58:35] (03approved) 10dancy: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) (owner: 10dduvall) [16:01:23] 10Gerrit, 07Upstream: Horizontal scrollbars are missing when needed in Gerrit - https://phabricator.wikimedia.org/T257705#12583340 (10Paladox) 05Open→03Resolved Looks like this is resolved? please reopen if it's still a problem. [16:11:12] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 33): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12583432 (10KReid-WMF) [16:22:50] (03CR) 10Jforrester: [C:03+2] jjb: Upgrade to PHP 8.3.37/8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371868 (owner: 10Jforrester) [16:23:54] (03Merged) 10jenkins-bot: jjb: Upgrade to PHP 8.3.37/8.5.13 [integration/config] - 10https://gerrit.wikimedia.org/r/1371868 (owner: 10Jforrester) [16:32:22] (03update) 10dancy: deploy-service: Deploy one release of a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1435 (https://phabricator.wikimedia.org/T442586) [16:32:44] Yippee, build fixed! [16:32:44] Project mediawiki-core-doxygen build #24437: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24437/ [16:35:45] (03update) 10dancy: deploy-service: Deploy one release of a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1435 (https://phabricator.wikimedia.org/T442586) [17:00:14] gitlab question. can someone see what I'm missing with https://gitlab.wikimedia.org/repos/cloud/toolforge/kube-container-updater/-/merge_requests/2 not triggering any CI jobs? [17:02:57] bah, I am an idiot [17:04:19] What'd it turn out to be? [17:05:24] i was including the wrong file and the file i was including declared no pipelines [17:07:34] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/go-sql-driver/mysql from 1.8.1 to 1.10.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/825 [17:07:34] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/go-sql-driver/mysql from 1.8.1 to 1.10.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/825 [17:08:12] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump gopkg.in/ini.v1 from 1.67.3 to 1.68.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/819 [17:21:20] 10Beta-Cluster-Infrastructure, 10CFSSL-PKI, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12583864 (10dr0ptp4kt) @Southparkfan I am terribly sorry, I missed the request to review and merge the pa... [17:33:01] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 20 others: Migrate Event Platform to node26 - https://phabricator.wikimedia.org/T445574 (10tchin) 03NEW [17:34:56] (03merge) 10dduvall: image: Support building Zuul 14.3.0 on trixie [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/21 (https://phabricator.wikimedia.org/T445248) [17:43:32] 06Release-Engineering-Team, 06ServiceOps, 05FY2025-26 KR 5.1, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#12584022 (10AGhirelli-WMF) [17:50:58] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Build and deploy Zuul 14.3.0 - https://phabricator.wikimedia.org/T445248#12584050 (10dduvall) [17:51:52] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12584059 (10Jdforrester-WMF) [17:52:20] 06Release-Engineering-Team (Radar), 10Scap, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12584064 (10Jdforrester-WMF) [18:14:58] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584178 (10bd808) `lang=shell-session root@nginx:~# docker images i Info →... [18:51:55] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12584272 (10ssingh) This should now be resolved. Thanks to @Fabfur for working on it and @Vgutierrez for review... [18:54:06] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:55:51] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Puppet agent failure detected on instance deployment-cp-upload09 in project deployment-prep - https://phabricator.wikimedia.org/T445275#12584280 (10ssingh) 05Open→03Resolved [19:03:47] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584319 (10bd808) @thcipriani pointed me at https://github.com/bitergia-analytics/documentation-handover and control-node.development-... [19:16:14] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584365 (10bd808) `lang=shell-sesison zhquan@control-node:~/bap-deployment-toolkit/ansible$ ansible-playbook -i environments/wikimedia... [19:19:06] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:22:43] 10Gerrit (Gerrit 3.12), 06Release-Engineering-Team, 06Collaboration-Services, 07Upstream: Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10 - https://phabricator.wikimedia.org/T407844#12584401 (10Paladox) Confirmed that gerrit 3.12 fixed it for me. I tested 3.11 and 3.1... [19:26:47] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584409 (10bd808) `lang=shell-session zhquan@control-node:~/bap-deployment-toolkit/ansible$ grep 'environments/wikimedia/ playbooks/'... [19:26:48] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584 (10pwangai) 03NEW [20:32:14] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, and 2 others: Slow connections from Cloud VPS to cdn.cypress.io make CI builds take up to 50 minutes - https://phabricator.wikimedia.org/T445584#12584573 (10pwangai) 05Open→03Invalid CI seems to be healing... [20:38:10] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review: Buildkit v0.35.1 released - https://phabricator.wikimedia.org/T445370#12584588 (10dancy) [20:39:18] (03open) 10dancy: Count only filter-matching volumes toward volume usage [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/27 (https://phabricator.wikimedia.org/T443833) [20:39:35] (03update) 10dancy: Count only filter-matching volumes toward volume usage [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/27 (https://phabricator.wikimedia.org/T443833) [20:40:25] (03merge) 10dancy: Count only filter-matching volumes toward volume usage [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/27 (https://phabricator.wikimedia.org/T443833) [20:43:07] (03PS1) 10Jforrester: Docker: [php-scratch] Bump php-ast to v1.1.4 for PHP 8.6 support [integration/config] - 10https://gerrit.wikimedia.org/r/1372009 [20:45:22] (03open) 10dancy: Release 1.5.0 [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/28 [20:45:23] (03update) 10dancy: Release 1.5.0 [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/28 [20:46:25] (03merge) 10dancy: Release 1.5.0 [repos/releng/docker-gc] - 10https://gitlab.wikimedia.org/repos/releng/docker-gc/-/merge_requests/28 [20:48:09] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07PHP 8.6 support: Add the ability to test code on PHP 8.6 in Gerrit CI - https://phabricator.wikimedia.org/T445365#12584615 (10Jdforrester-WMF) Note that right now, as normal for brand-new x.0 PHP releases, we can't build our base image because fi... [20:49:47] (03CR) 10Jforrester: [C:03+2] Docker: [php-scratch] Bump php-ast to v1.1.4 for PHP 8.6 support [integration/config] - 10https://gerrit.wikimedia.org/r/1372009 (owner: 10Jforrester) [20:51:36] (03Merged) 10jenkins-bot: Docker: [php-scratch] Bump php-ast to v1.1.4 for PHP 8.6 support [integration/config] - 10https://gerrit.wikimedia.org/r/1372009 (owner: 10Jforrester) [20:53:35] !log Docker: [php-scratch] Bump php-ast to v1.1.4 for PHP 8.6 support, for T445365. [20:53:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:53:37] T445365: Add the ability to test code on PHP 8.6 in Gerrit CI - https://phabricator.wikimedia.org/T445365 [21:00:34] !log Updated gitlab-cloud-runners buildkitd to v0.35.1 (T445370) [21:00:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:36] T445370: Buildkit v0.35.1 released - https://phabricator.wikimedia.org/T445370 [21:00:58] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review: Buildkit v0.35.1 released - https://phabricator.wikimedia.org/T445370#12584670 (10dancy) [21:01:40] (03open) 10dancy: Bump buildkitd to v0.35.1 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/667 (https://phabricator.wikimedia.org/T445370) [21:01:51] (03update) 10dancy: Bump buildkitd to v0.35.1 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/667 (https://phabricator.wikimedia.org/T445370) [21:02:54] (03merge) 10dancy: Bump buildkitd to v0.35.1 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/667 (https://phabricator.wikimedia.org/T445370) [21:38:20] (03open) 10dduvall: image: Implement `PUBLISH_SBOM` for single image builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/173 (https://phabricator.wikimedia.org/T445371) [21:38:39] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584836 (10bd808) `lang=shell-session bd808@mordred:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND... [21:41:51] 06Release-Engineering-Team (Priority Backlog 📥), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584844 (10bd808) `lang=shell-session bd808@sortinghat:~$ sudo docker ps -a CONTAINER ID IMAGE... [21:42:31] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584851 (10bd808) 05Open→03In progress a:03bd808 [21:42:44] (03merge) 10dduvall: image: Implement `PUBLISH_SBOM` for single image builds [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/173 (https://phabricator.wikimedia.org/T445371) [21:44:22] 10Beta-Cluster-Infrastructure: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12584867 (10Southparkfan) 05Open→03In progress [21:44:31] 10Beta-Cluster-Infrastructure: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011#12584868 (10Southparkfan) a:03Southparkfan [21:49:16] (03open) 10dduvall: version: 2.19.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/174 [21:49:20] 10Beta-Cluster-Infrastructure, 10CFSSL-PKI, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12584898 (10Southparkfan) >>! In T436562#12583863, @dr0ptp4kt wrote: > @Southparkfan I am terribly so... [21:49:23] 10Beta-Cluster-Infrastructure, 10CFSSL-PKI, 06Infrastructure-Foundations: profile::pki::client does not hash symlinks and does not populate ca-certificates.crt - https://phabricator.wikimedia.org/T436562#12584899 (10Southparkfan) 05Open→03Resolved a:03Southparkfan [21:50:37] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12584905 (10bd808) `lang=shell-session bd808@sortinghat-worker:~$ sudo docker ps -a CONTAINER ID IMAGE... [21:52:27] (03merge) 10dduvall: version: 2.19.0 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/174 [22:10:52] (03PS1) 10Reedy: zuul: Remove PersonalDashboard as a dependancy on REL1_43/REL1_45 [integration/config] - 10https://gerrit.wikimedia.org/r/1372135 [22:13:52] (03CR) 10Reedy: [C:03+2] zuul: Remove PersonalDashboard as a dependancy on REL1_43/REL1_45 [integration/config] - 10https://gerrit.wikimedia.org/r/1372135 (owner: 10Reedy) [22:15:12] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585020 (10bd808) `lang=shell-session bd808@valley:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CRE... [22:15:41] (03Merged) 10jenkins-bot: zuul: Remove PersonalDashboard as a dependancy on REL1_43/REL1_45 [integration/config] - 10https://gerrit.wikimedia.org/r/1372135 (owner: 10Reedy) [22:18:57] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585031 (10bd808) I'm pretty sure sortinghat's workers are dependent on the redis/valley service, so now that it is up let's try starting the wor... [22:20:55] (03open) 10dduvall: image: Fix buildctl SBOM attestation option [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/175 [22:23:30] (03merge) 10dduvall: image: Fix buildctl SBOM attestation option [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/175 [22:26:07] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585066 (10bd808) `lang=shell-session bd808@mariadb:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED... [22:28:30] (03open) 10dduvall: version: 2.19.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/176 [22:28:30] (03update) 10dduvall: version: 2.19.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/176 [22:28:38] (03update) 10dduvall: version: 2.19.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/176 [22:30:37] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#12585079 (10matmarex) I'm here because I accidentally pushed a low-s... [22:30:55] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#12585084 (10matmarex) >>! In T388247#12438375, @sbassett wrote: >>>!... [22:31:46] (03merge) 10dduvall: version: 2.19.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/176 [22:34:03] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1372135 [22:34:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:49:07] !log Remove /var/log/syslog on deployment-webperf31 - T445011 [22:49:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:49:10] T445011: navtiming is broken on the Beta Cluster because kafka-jumbo has been removed - https://phabricator.wikimedia.org/T445011 [23:16:40] (03merge) 10addshore: build(deps): bump gopkg.in/ini.v1 from 1.67.3 to 1.68.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/819 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [23:18:22] (03update) 10addshore: Draft: Agents, guardrails, and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [23:18:51] (03update) 10addshore: Draft: Agents, guardrails, MCP and command metadata [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/821 [23:35:34] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics, 07Essential-Work: development-metrics.wmcloud.org is down - https://phabricator.wikimedia.org/T443279#12585370 (10bd808) @thcipriani and @Aklapper: I think I have gotten all of the containers running again. I have not dug through docs to try and fi... [23:50:51] Project mediawiki-core-doxygen build #24454: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/24454/