[02:07:15] (03PS1) 10Clare Ming: Add Test Kitchen experiment overrides to the header and popup [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) [02:11:31] (03CR) 10Clare Ming: "tested locally by disabling the current WikimediaDebug extension and loading (unpacked) the feature branch and enabling on a production wi" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) (owner: 10Clare Ming) [02:13:11] (03CR) 10Clare Ming: "Note: this patch needs to be tested end-to-end with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1344116" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) (owner: 10Clare Ming) [03:32:31] (03CR) 10Pwangai: api-testing: Do not install core's npm dependencies twice (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1344086 (https://phabricator.wikimedia.org/T438906) (owner: 10Pwangai) [03:34:00] (03PS2) 10Pwangai: api-testing: Do not install core's npm dependencies twice [integration/quibble] - 10https://gerrit.wikimedia.org/r/1344086 (https://phabricator.wikimedia.org/T438906) [05:00:47] (03CR) 10Pwangai: "PS2 drops the mocha wording since it's not what this change is about. The reason I targeted the second install rather than the first is T4" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1344086 (https://phabricator.wikimedia.org/T438906) (owner: 10Pwangai) [07:18:53] 06Release-Engineering-Team (Doing 😎), 10QS-Test-Automation, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Investigate retry mechanism for catalyst provisioning error - https://phabricator.wikimedia.org/T437955#12353579 (10jnuche) >>! In T437955#12352309, @jeena wrote: > @jnuche I recall that fail... [07:36:01] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 13Patch-For-Review: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829#12353617 (10MoritzMuehlenhoff) >>! In T437829#12320689, @tstarling wrote: > I filed an issue and a PR https://github.com/debuerreotype/deb... [07:44:40] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster jobrunner opcache is not updated during automated deployments - https://phabricator.wikimedia.org/T438656#12353628 (10JMonton-WMF) Thanks for the explanation! [08:22:17] (03PS2) 10Hashar: Zuul: Move repos to the generic tox job except where they need something special [integration/config] - 10https://gerrit.wikimedia.org/r/1343095 (owner: 10Jforrester) [08:22:17] (03PS2) 10Hashar: jjb: Drop bespoke tox jobs that added nothing except a queue, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343096 (owner: 10Jforrester) [08:24:59] (03CR) 10Hashar: [C:03+2] Zuul: Move repos to the generic tox job except where they need something special [integration/config] - 10https://gerrit.wikimedia.org/r/1343095 (owner: 10Jforrester) [08:25:07] (03CR) 10Hashar: [C:03+2] jjb: Drop bespoke tox jobs that added nothing except a queue, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343096 (owner: 10Jforrester) [08:25:51] (03PS2) 10Jforrester: jjb: Provide tox-mysqld job to share between repos [integration/config] - 10https://gerrit.wikimedia.org/r/1343103 [08:26:17] (03CR) 10Hashar: [C:03+2] jjb: Provide tox-mysqld job to share between repos [integration/config] - 10https://gerrit.wikimedia.org/r/1343103 (owner: 10Jforrester) [08:26:32] (03Merged) 10jenkins-bot: Zuul: Move repos to the generic tox job except where they need something special [integration/config] - 10https://gerrit.wikimedia.org/r/1343095 (owner: 10Jforrester) [08:26:54] (03Merged) 10jenkins-bot: jjb: Drop bespoke tox jobs that added nothing except a queue, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343096 (owner: 10Jforrester) [08:27:22] (03CR) 10Hashar: [C:03+2] Zuul: Use tox-mysqld over bespokes [integration/config] - 10https://gerrit.wikimedia.org/r/1343104 (owner: 10Jforrester) [08:27:59] (03Merged) 10jenkins-bot: jjb: Provide tox-mysqld job to share between repos [integration/config] - 10https://gerrit.wikimedia.org/r/1343103 (owner: 10Jforrester) [08:30:58] (03PS2) 10Jforrester: Zuul: Use tox-mysqld over bespokes [integration/config] - 10https://gerrit.wikimedia.org/r/1343104 [08:32:44] (03PS2) 10Hashar: jjb: Drop equivalents to tox-mysqld, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343105 (owner: 10Jforrester) [08:35:44] (03CR) 10Hashar: [C:03+2] Zuul: Use tox-mysqld over bespokes [integration/config] - 10https://gerrit.wikimedia.org/r/1343104 (owner: 10Jforrester) [08:35:49] (03CR) 10Hashar: [C:03+2] jjb: Drop equivalents to tox-mysqld, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343105 (owner: 10Jforrester) [08:37:17] (03Merged) 10jenkins-bot: Zuul: Use tox-mysqld over bespokes [integration/config] - 10https://gerrit.wikimedia.org/r/1343104 (owner: 10Jforrester) [08:37:23] (03Merged) 10jenkins-bot: jjb: Drop equivalents to tox-mysqld, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1343105 (owner: 10Jforrester) [08:46:06] (03CR) 10Hashar: [C:03+2] "I have deleted the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/1343096 (owner: 10Jforrester) [08:48:28] (03CR) 10Hashar: [C:03+2] "I have deleted the jobs!" [integration/config] - 10https://gerrit.wikimedia.org/r/1343105 (owner: 10Jforrester) [09:00:58] (03CR) 10Dreamy Jazz: [C:03+1] Zuul: [mediawiki/extension/NewUserMessage] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1344105 (https://phabricator.wikimedia.org/T437743) (owner: 10Neriah) [09:04:16] 10Phabricator, 06Release-Engineering-Team: Related changes in GitLab are not always shown - https://phabricator.wikimedia.org/T438826#12353843 (10Aklapper) 05Stalledβ†’03Open [09:11:13] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12353882 (10ABran-WMF) The minor-vs-patch question from the Phase 0 section above is resolved: @jmeybohm confirmed on [[https://gerrit.wikimedia.org/r/c/o... [09:38:54] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 06Test Platform (Arusha 30): Investigate the slowdown in quibble-with-gated-extensions-vendor-mysql-php83 - https://phabricator.wikimedia.org/T438928#12353960 (10hashar) CC @kostajh and @ArthurTaylor who introduced the phpunit results cache tool as... [09:49:55] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 06Test Platform (Arusha 30): Investigate the slowdown in quibble-with-gated-extensions-vendor-mysql-php83 - https://phabricator.wikimedia.org/T438928#12353990 (10ArthurTaylor) @hashar The PHPUnit Results cache should only determine the way tests ar... [10:04:20] (03PS1) 10Hashar: zuul: add reference links for phpunit-results-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1344217 [10:12:22] (03CR) 10Phedenskog: [C:03+2] zuul: add reference links for phpunit-results-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1344217 (owner: 10Hashar) [10:14:04] (03Merged) 10jenkins-bot: zuul: add reference links for phpunit-results-cache [integration/config] - 10https://gerrit.wikimedia.org/r/1344217 (owner: 10Hashar) [10:16:46] 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 06Test Platform (Arusha 30): Investigate the slowdown in quibble-with-gated-extensions-vendor-mysql-php83 - https://phabricator.wikimedia.org/T438928#12354056 (10hashar) > in the presence of a cache result, the buckets are balanced based on the pre... [10:29:27] codders: thank you for the quick answers about the phpunit result cache! :] [10:29:42] and for the review of bumping the timeout + adding connect_timeout :party: [10:29:46] no worries - thanks for the +2! happy to see that it's getting some attention [10:39:34] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikidata sitelinks are wrong - https://phabricator.wikimedia.org/T438841#12354091 (10Ifeatu_Nnaobi_WMDE) [10:39:37] 10Beta-Cluster-Infrastructure, 10Wikidata, 07ContentSecurityPolicy, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735#12354093 (10Ifeatu_Nnaobi_WMDE) [10:39:40] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikidata sitelinks are wrong - https://phabricator.wikimedia.org/T438841#12354096 (10Ifeatu_Nnaobi_WMDE) [10:39:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 07ContentSecurityPolicy, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735#12354098 (10Ifeatu_Nnaobi_WMDE) [10:45:21] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extension/NewUserMessage] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1344105 (https://phabricator.wikimedia.org/T437743) (owner: 10Neriah) [10:47:21] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extension/NewUserMessage] Add CheckUser dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1344105 (https://phabricator.wikimedia.org/T437743) (owner: 10Neriah) [10:57:29] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354212 (10Ifeatu_Nnaobi_WMDE) 05Resolvedβ†’03Open a:05Aklapperβ†’03None [10:57:39] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354218 (10Ifeatu_Nnaobi_WMDE) [10:58:37] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12354220 (10Aklapper) Note to myself: Test https://we.phorge.it/T16806 in our instance after updating as I cannot reproduce upstream or locally, some change in... [10:58:49] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354221 (10Ifeatu_Nnaobi_WMDE) [10:59:03] 10Phabricator: Request for acl*Batch-Editors for seanleong-WMDE - https://phabricator.wikimedia.org/T438948 (10seanleong-WMDE) 03NEW [10:59:59] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354232 (10Ifeatu_Nnaobi_WMDE) @Aklapper πŸ™πŸΌ [11:01:11] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354261 (10Aklapper) @Ifeatu_Nnaobi_WMDE: Please don't vandalize existing valid tasks but create a separate ticket for a separate request. Thanks! [11:02:24] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T412273#12354264 (10Ifeatu_Nnaobi_WMDE) sorry about that. long day. making a new request. please forgive me [11:03:16] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Request for acl*Batch-Editors for KReid_WMF - https://phabricator.wikimedia.org/T412273#12354267 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper [11:04:26] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for acl*Batch-Editors for seanleong-WMDE - https://phabricator.wikimedia.org/T438948#12354275 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper I added you to https://phabricator.wikimedia.org/project/members/13/ [11:16:21] 10Phabricator: Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T438951 (10Ifeatu_Nnaobi_WMDE) 03NEW [11:18:45] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for acl*Batch-Editors for Ifeatu_Nnaobi_WMDE - https://phabricator.wikimedia.org/T438951#12354319 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Hi, I added you to https://phabricator.wikimedia.org/project/members/13 [11:18:51] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request for acl*Batch-Editors for seanleong-WMDE - https://phabricator.wikimedia.org/T438948#12354324 (10seanleong-WMDE) Thanks [11:34:29] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Beta cluster wikidata sitelinks are wrong - https://phabricator.wikimedia.org/T438841#12354412 (10seanleong-WMDE) [11:34:32] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects, 07ContentSecurityPolicy: Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735#12354414 (10seanleong-WMDE) [12:19:34] Project mediawiki-core-doxygen build #23011: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23011/ [12:56:31] !log manually run puppet agent [12:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:58:47] !log add seanleong-wmde to deployment-prep [12:58:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:01:47] (03PS1) 10Marco Fossati: ReaderExperiments: add CirrusSearch Zuul dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) [13:07:34] (03CR) 10Matthias Mullie: [C:03+1] ReaderExperiments: add CirrusSearch Zuul dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [13:24:00] Deploying a beta-only config now. [13:32:23] (03PS6) 10Hashar: utils: script to verify package-lock.json integrity [integration/config] - 10https://gerrit.wikimedia.org/r/1343551 (https://phabricator.wikimedia.org/T437697) [13:33:26] (03CR) 10Hashar: ReaderExperiments: add CirrusSearch Zuul dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [13:34:54] Yippee, build fixed! [13:34:54] Project mediawiki-core-doxygen build #23012: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23012/ [13:35:07] (03PS2) 10Hashar: Zuul: Add CirrusSearch for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [13:35:16] (03CR) 10Hashar: [C:03+2] Zuul: Add CirrusSearch for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [13:36:56] (03Merged) 10jenkins-bot: Zuul: Add CirrusSearch for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [13:41:22] anyone know why https://gitlab.wikimedia.org/repos/sre/klaxon/-/jobs/972326 reports 'stuck' ? main is a protected branch, and, the repo is in trusted-runners list [13:47:00] 10Phabricator (Upstream), 06MediaWiki-Core-Platform-Team, 10Notifications (Echo), 07Browser-Support-Firefox, 07Regression: Echo notifications: CTRL+Click leads to /wiki/Null in Firefox - https://phabricator.wikimedia.org/T427053#12355047 (10Krinkle) [13:50:12] James_F: about codesniffer upgrade for REL1_45 to address the upstream security issue, I somehow missed you already have done upstream version bump in a 48.x branch https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/1321916 [13:50:31] so yeah that ones could use to be released as 48.0.1. Idon't think I have the credentials to do so [13:50:48] or maybe it is all about pushing a tag [13:51:13] and a release note bah [13:52:16] 10Scap, 06SRE Observability: Scap's logstash checker for channels outside "error" and "exception" - https://phabricator.wikimedia.org/T438570#12355090 (10herron) I would say `tags: *throttle_warning*` is a reasonable query for the time being, if logstash is on the table. The alert that fired with {T438387} wa... [13:57:50] working on it [13:59:13] (03open) 10jforrester: releases: Bump REL1_45 to codesniffer 48.0.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/148 (https://phabricator.wikimedia.org/T434187) [13:59:32] (03CR) 10Marco Fossati: Zuul: Add CirrusSearch for ReaderExperiments (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1344264 (https://phabricator.wikimedia.org/T437487) (owner: 10Marco Fossati) [14:00:03] hashar: I’ve done it. [14:00:18] damn [14:00:29] why are patches notifications not sent here! :) [14:00:31] let me +2 :) [14:00:35] Ha. [14:00:45] ah no [14:00:49] the release note entry https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/1344288 [14:01:21] which fails on 8.5 AHAH [14:01:43] 14:00:10 /src/vendor/phan/phan/src/Phan/CodeBase.php:590 [8192] Method SplObjectStorage::attach() is deprecated since 8.5, use method SplObjectStorage::offsetSet() instead [14:01:44] fun [14:01:54] ahh, I see, ahh, if you don't have wildcard tag protection, you're gonna have a bad time [14:02:00] would we run php8.5 codesniffer against REL1_45? [14:02:13] cdanis: yeah sorry I don't know much about gitlab protected branch etc :\ [14:18:03] James_F: bump of phan https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/1344289 [14:18:03] and I have updated the HISTORY to mention both php_codesniffer and phan upgrade https://gerrit.wikimedia.org/r/c/mediawiki/tools/codesniffer/+/1344288/ [14:20:37] I am half considering introducing a `#mediawiki-qa` channel for Phan/codesniffer and the various mass updates accross extensions [14:20:43] but well we have so many channels already [15:02:04] hashar: Why? We don’t run PHP 8.5 for REL1_45 and probably never will. [15:02:40] hashar: Why not call the channel #wikimedia-qte? ;-) [16:33:06] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-ParserMigration] Add ParserMigration job [integration/config] - 10https://gerrit.wikimedia.org/r/1344328 (https://phabricator.wikimedia.org/T433086) [16:34:23] (03CR) 10Vaughn Walters: [C:03+2] "Test suite green at https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-ParserMigration/" [integration/config] - 10https://gerrit.wikimedia.org/r/1344328 (https://phabricator.wikimedia.org/T433086) (owner: 10Vaughn Walters) [16:35:53] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-ParserMigration] Add ParserMigration job [integration/config] - 10https://gerrit.wikimedia.org/r/1344328 (https://phabricator.wikimedia.org/T433086) (owner: 10Vaughn Walters) [17:22:00] PROBLEM - Host doc2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:27:53] 06Release-Engineering-Team, 10Ceph, 06Infrastructure-Foundations, 10SRE-swift-storage: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12356575 (10elukey) I ran the GC in dry-run mode and it completed in ~67 minutes, that is a nicer result than the fir... [17:29:50] (03open) 10dancy: scripts/release.sh: Bump MR polling internal to 30 seconds [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/186 [17:30:02] (03update) 10dancy: scripts/release.sh: Bump MR polling interval to 30 seconds [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/186 [17:30:08] (03update) 10dancy: scripts/release.sh: Bump MR polling interval to 30 seconds [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/186 [17:36:15] (03merge) 10dancy: scripts/release.sh: Bump MR polling interval to 30 seconds [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/186 [17:36:52] Project mediawiki-core-doxygen build #23016: 04FAILURE in 18 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23016/ [17:47:51] o/ Search would like to run a series of short-lived deployments to observe the effects of distributing search queries across DCs. Each experiment would only take about 10–20 minutes, so using the regular backport windows would stretch the series over several days. [17:47:51] Would it be possible to reserve a dedicated deployment slot in early October so we can run the series back-to-back? Or is there another rapid-deployment approach you’d recommend for this kind of experiment? [17:49:10] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926#12356625 (10hashar) [18:18:31] Project mediawiki-core-doxygen build #23017: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23017/ [18:29:04] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [18:29:10] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T439024 (10wmcs-alerts) 03NEW [18:31:01] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T439026 (10phaultfinder) 03NEW [18:33:29] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T439026#12356837 (10phaultfinder) [18:33:37] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-ReportIncident] Add ReportIncident job [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) [18:37:56] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T439026#12356845 (10phaultfinder) [18:39:04] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [19:10:46] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:32:23] Yippee, build fixed! [19:32:23] Project mediawiki-core-doxygen build #23018: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/23018/ [19:36:02] !log gerrit: force online reindexing of the change index over ssh hitting gerrit1003 (switch over). `gerrit index start changes --force` [19:36:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:38:04] !log gerrit: started reindexing for accounts, group and projects indices [19:38:05] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:38:10] thcipriani: sobanski: ^^^ [19:38:37] the Lucene indices on gerrit1003 were stall because Gerrit running as replica does not maintain them (a replica has no use for those) [19:39:02] so once switched ot it, it served stall data from the last time we switched, sometime in february maybe [19:39:52] Correct [19:41:41] hashar: before I tried that, I decided to try a small project with `gerrit index changes-in-project 'operations/debs/wmf-laptop'`, but that didn't seem to update the search results for that repo, but your reindex seems to be working. ... Not sure I understand why. [20:01:04] FIRING: PuppetAgentDisabled: Puppet agent disabled on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled [20:03:11] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Upgrade gitlab-cloud-runners Kubernetes to 1.36.3-do.5 - https://phabricator.wikimedia.org/T439035 (10dancy) 03NEW [20:12:07] thcipriani: that was a good idea, no idea why it did not work. I independently did the same for integration/config and it did not work either [20:13:30] !log gerrit: reindexing has been completed [20:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:16:33] (03CR) 10Vaughn Walters: [C:03+2] "tests green : https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-ReportIncident/" [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) (owner: 10Vaughn Walters) [20:24:08] (03CR) 10Vaughn Walters: jjb: [catalyst-daily-ReportIncident] Add ReportIncident job [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) (owner: 10Vaughn Walters) [20:26:52] !log Upgrading istio to 1.30.5 on gitlab-cloud-runners (staging) (T439035) [20:26:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:26:55] T439035: Upgrade gitlab-cloud-runners Kubernetes to 1.36.3-do.5 - https://phabricator.wikimedia.org/T439035 [20:27:26] (03CR) 10Vaughn Walters: [C:03+2] jjb: [catalyst-daily-ReportIncident] Add ReportIncident job [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) (owner: 10Vaughn Walters) [20:27:42] (03CR) 10Vaughn Walters: [V:03+2 C:03+2] jjb: [catalyst-daily-ReportIncident] Add ReportIncident job [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) (owner: 10Vaughn Walters) [20:29:18] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-ReportIncident] Add ReportIncident job [integration/config] - 10https://gerrit.wikimedia.org/r/1344344 (https://phabricator.wikimedia.org/T433088) (owner: 10Vaughn Walters) [20:32:31] (03CR) 10Clare Ming: "I tested this locally with the TK extension patch (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1344116) running in" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) (owner: 10Clare Ming) [20:37:16] !log gerrit: deleted https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1344362 duplicate Change-Id I67dd7b3c879049a1c98586c365c297eac150b963 of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1329697 [20:37:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:37:42] RoanKattouw: ^^ ^ :) [20:37:57] Thank you! [20:38:05] I need to forge the committer haha [20:38:36] RoanKattouw: so your rogue change , I have deleted it [20:39:07] When will it be safe to try to reupload my patchset? [20:39:10] and since I had it on my local machine, I send it back [20:39:21] so https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1329697 has the patchset from the change I deleted [20:39:23] I don't want to do that if it's just going to create another rogue commit that you need to clean up [20:39:38] Oh wow it does, thank you! [20:48:33] (03open) 10dancy: .pipeline/blubber.yaml: Bump to golang1.22:1.22-20260920 [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/44 [20:49:09] RoanKattouw: I wrote some explanation at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1321624/comments/e71ba1ed_b543db0f [20:49:21] and I think sobanski wrote one as well [20:49:29] thcipriani: ^^^ the doc for dupe change [20:50:34] (03merge) 10dancy: .pipeline/blubber.yaml: Bump to golang1.22:1.22-20260920 [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/44 [20:55:49] 10WikimediaDebug, 07Essential-Work, 13Patch-For-Review, 10Test Kitchen (Experiment Platform Sprint 30): Update WikimediaDebug to fetch active experiment config and allow selection of experiments and groups - https://phabricator.wikimedia.org/T438283#12357205 (10cjming) === Testing Instructions === To test... [20:56:21] 10WikimediaDebug, 07Essential-Work, 13Patch-For-Review, 10Test Kitchen (Experiment Platform Sprint 30): Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240#12357207 (10cjming) See testing instructions at https://phabricator.wikimedia.org/T438283#12357205 [21:01:50] (03CR) 10Clare Ming: "more granular testing instructions at https://phabricator.wikimedia.org/T438283#12357205" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1344123 (https://phabricator.wikimedia.org/T438283) (owner: 10Clare Ming) [21:40:17] dancy: re. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344336 - that directory was created by me yesterday (https://sal.toolforge.org/log/xL5Gy6ABDAZZyZXn-wPc), any clue why Scap failed nevertheless? [21:41:50] Scap failed because that puppet change hasn't been merged yet. I created it when I saw that you needed to work around a bug. [21:43:02] Right. Want me to cherry-pick it to the Puppetserver? [21:43:30] Yeah go for it [21:43:40] Alright [21:45:15] By the way, I have been !logging all my action on the Beta Cluster on the deployment-prep SAL. Should I use the releng SAL instead? [21:46:31] https://sal.toolforge.org/releng does show the deployment-prep SAL though :) [21:47:09] Cherry-pick done. Thanks for your help! [21:47:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Collaboration-Services: Zuul Zookeeper does not generate logs - https://phabricator.wikimedia.org/T435503#12357346 (10Dzahn) zookeeper is now logging on cloudcontrol-dev in codfw. [21:48:01] I think this channel is customary for deployment-prep work [21:48:39] Southparkfan: Thank you for your heroic work on beta cluster! [21:48:59] It's my pleasure [21:49:15] Alright, will use this channel from now on [21:50:30] Feel like it's time to start decommissioning the PHP 8.3 hosts.... [21:50:54] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - releases2003, doc1004, people2004 - https://phabricator.wikimedia.org/T439026#12357352 (10Dzahn) [21:51:09] Agreed [21:51:39] Thanks for confirming [22:06:52] !log decommission Beta Cluster PHP 8.3 hosts deployment-mediawiki13, deployment-mediawiki14, deployment-jobrunner05, deployment-mwmaint03 - T435393 [22:06:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:06:55] T435393: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393 [22:11:04] RESOLVED: PuppetAgentDisabled: Puppet agent disabled on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled [22:13:05] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:16:03] 10GitLab (Infrastructure): Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046 (10thcipriani) 03NEW [22:16:36] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12357435 (10thcipriani) [22:18:54] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - releases2003, doc1004, people2004 - https://phabricator.wikimedia.org/T439026#12357440 (10Dzahn) [22:19:11] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12357442 (10Southparkfan) The PHP 8.3 hosts hav... [22:20:25] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12357455 (10dancy) p:05Triageβ†’03High [22:20:36] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - releases2003, doc1004, people2004 - https://phabricator.wikimedia.org/T439026#12357458 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn checked all 3 hosts and the services (doc, releases, people) and they are up (now). linked to codfw power pare... [22:24:32] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12357493 (10dancy) Accessing the same URL from the host (outside of buildkitd container) works: ` dancy@gitlab-runner2002... [22:47:52] 06Release-Engineering-Team (Doing 😎), 10QS-Test-Automation, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Investigate retry mechanism for catalyst provisioning error - https://phabricator.wikimedia.org/T437955#12357571 (10jeena) Ohh, I didn't see the increased timeout, sorry!!