[02:10:43] 10Continuous-Integration-Infrastructure, 10Tool-coverage, 07Test-Coverage: Migrate https://coverage.toolforge.org/charts/ to CI infrastructure - https://phabricator.wikimedia.org/T182751#12342954 (10Legoktm) [05:22:41] 10Continuous-Integration-Infrastructure, 10Tool-coverage, 07Test-Coverage: Migrate https://coverage.toolforge.org/charts/ to CI infrastructure - https://phabricator.wikimedia.org/T182751#12343046 (10Legoktm) 05Openβ†’03Declined I don't think there was ever a good plan or interest in doing this; in {T43... [07:29:38] 10Beta-Cluster-Infrastructure: syntaxhighlighter throwing errors on beta cluster - https://phabricator.wikimedia.org/T438653 (10awight) 03NEW [07:35:57] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12343246 (10MoritzMuehlenhoff) 05Openβ†’03Resolved a:03MoritzMuehlenhoff The combination of https://gerrit.wikimedia.o... [07:37:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing jobs - https://phabricator.wikimedia.org/T438654 (10Marostegui) 03NEW [07:37:17] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343261 (10Marostegui) [07:39:46] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster seems to not propagate wikidata changes to subscribing clients - https://phabricator.wikimedia.org/T438656 (10awight) 03NEW [07:52:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343309 (10LSobanski) Looking at the [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318976?checksRunsSelected=test-prio&tab... [07:58:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343328 (10LSobanski) @hashar could you take a look at whether the CI image exists? [08:01:54] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: handle KEDA autoscaler state transitions correctly - https://phabricator.wikimedia.org/T438659 (10jnuche) 03NEW [08:02:35] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: handle KEDA autoscaler state transitions correctly - https://phabricator.wikimedia.org/T438659#12343373 (10jnuche) [08:09:26] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343390 (10Marostegui) p:05Triageβ†’03Unbreak! I am going to set this to UBN as no puppet jobs are being processed at all blocking... [08:20:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07Zuul: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343434 (10LSobanski) Buliding Bullseye images was disabled on Sep 8 and re-enabled on Sep 18 (https://gerrit.wikimedia.org/r/c/opera... [08:56:01] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343854 (10hashar) The [[ https://integration.wikimedia.org/ci/job/operations-puppet-tests-bullseye/ | operations-puppet-tests-bul... [08:59:28] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343920 (10Marostegui) 05Openβ†’03Resolved a:03hashar My change got verified by CI, so this is working. Thanks for fixing it! [09:01:00] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12343924 (10Marostegui) [09:03:40] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: Create a second Jenkins agent covering operations/puppet test job - https://phabricator.wikimedia.org/T438690 (10hashar) 03NEW [09:35:34] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: Create a second Jenkins agent covering operations/puppet test job - https://phabricator.wikimedia.org/T438690#12344105 (10hashar) 05Openβ†’03Resolved a:03hashar I have created `integration-agent-puppet-docker-1005` and ad... [09:36:28] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: zuul seems to be stuck and not processing puppet jobs - https://phabricator.wikimedia.org/T438654#12344120 (10hashar) > add a new agent in Jenkins so we would have two able to run that specific job We have added a new agent... [10:03:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12344311 (10fgiunchedi) Today I re-run the provision cookbook for cloudvirt1077 to bring it back to the st... [10:30:35] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Add WEBP to files.viewable-mime-types - https://phabricator.wikimedia.org/T438712 (10Aklapper) 03NEW p:05Triageβ†’03Low [10:31:37] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Add WEBP to files.viewable-mime-types - https://phabricator.wikimedia.org/T438712#12344427 (10Aklapper) Done in https://gitlab.wikimedia.org/repos/phabricator/deployment/-/commit/2ffc747e0fce914baa3d0ee9b1eea0d516f771d9 [10:36:39] 10Diffusion, 10Phabricator, 07Security: Diffusion 'view raw file' policy and behavior is unclear and inconsistent (PHP files displayed in browser; JS files offer downloading file) - https://phabricator.wikimedia.org/T149152#12344438 (10Aklapper) Steps to reproduce, for the lazy like me: * PHP file, downstrea... [10:37:10] 10Diffusion, 10Phabricator, 07Security: Diffusion 'view raw file' policy and behavior is unclear and inconsistent (PHP files displayed in browser; JS files offer downloading file) - https://phabricator.wikimedia.org/T149152#12344439 (10Aklapper) 05Openβ†’03Declined As written above by Mukunda, > it sho... [10:45:43] 10Phabricator (Upstream), 07Upstream: Phabricator activity feed shows old versions of comments that are edited - https://phabricator.wikimedia.org/T381631#12344479 (10Aklapper) p:05Triageβ†’03Low [10:45:50] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator doesn't fetch/display a linked comment when too much activity has occurred in the task since it was made - https://phabricator.wikimedia.org/T409907#12344480 (10Aklapper) p:05Triageβ†’03Medium [10:46:28] 10Phabricator (Search), 06Release-Engineering-Team (Priority Backlog πŸ“₯): Revisit Elasticsearch in Phorge - https://phabricator.wikimedia.org/T373130#12344482 (10Aklapper) p:05Triageβ†’03Low [11:06:54] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Aruotes - https://phabricator.wikimedia.org/T437405#12344561 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper @Aruotes Hi, I approved your GitLab account. (PS: [I might not be a sir.](https://www.me... [11:07:39] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for [atomic] - https://phabricator.wikimedia.org/T437786#12344583 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper @Atomic Hi, I've approved your GitLab account. Happy hacking! [11:09:09] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for [Rohitpal08] - https://phabricator.wikimedia.org/T438155#12344596 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Hi, I approved your GitLab account. Happy hacking! [11:09:58] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: handle KEDA autoscaler state transitions correctly - https://phabricator.wikimedia.org/T438659#12344602 (10jnuche) 05Openβ†’03Resolved [11:10:01] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for amintorbat - https://phabricator.wikimedia.org/T438603#12344603 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Hi, I've approved your GitLab account. Happy hacking! [11:18:31] (03CR) 10Hashar: [C:03+2] zuul: add Ow0cast to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1342827 (owner: 10Apehitkey) [11:19:41] (03PS2) 10Hashar: tox: update flake8 4.0.1..7.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/1343137 [11:20:09] (03Merged) 10jenkins-bot: zuul: add Ow0cast to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1342827 (owner: 10Apehitkey) [11:32:09] (03PS1) 10Hashar: Docker: [ajv] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343501 (https://phabricator.wikimedia.org/T438085) [11:32:37] (03PS2) 10Hashar: Docker: [sonar-scanner] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343075 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:33:12] (03CR) 10Hashar: "I have movedΒ `ajv`Β to its own change I5ebfbfd954f6bcaeb7d7b848d4dd68054f5d0c39. The image is only used by PipelineLib. I'll deploy it righ" [integration/config] - 10https://gerrit.wikimedia.org/r/1343075 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:33:54] (03CR) 10Hashar: [C:03+2] Docker: [ajv] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343501 (https://phabricator.wikimedia.org/T438085) (owner: 10Hashar) [11:34:08] (03CR) 10CI reject: [V:04-1] Docker: [sonar-scanner] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343075 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:35:27] (03Merged) 10jenkins-bot: Docker: [ajv] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343501 (https://phabricator.wikimedia.org/T438085) (owner: 10Hashar) [11:37:19] !log Build docker-registry.wikimedia.org/releng/ajv:0.4.1-s5 (upgrade Node from 24.18.0 to 26.8.2) Used by PipelineLib # T438085 [11:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:37:22] T438085: Upgrade all CI jobs for WMF-deployed projects from Node 24 to Node 26 - https://phabricator.wikimedia.org/T438085 [11:39:36] (03PS3) 10Hashar: Docker: [sonar-scanner] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343075 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:39:50] (03PS2) 10Jforrester: jjb: Migrate sonar-scanner to images with Node 26 [integration/config] - 10https://gerrit.wikimedia.org/r/1343076 (https://phabricator.wikimedia.org/T438085) [11:39:57] (03PS2) 10Jforrester: Docker: Drop Node 24 images, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/1343077 (https://phabricator.wikimedia.org/T438085) [11:41:27] (03CR) 10CI reject: [V:04-1] jjb: Migrate sonar-scanner to images with Node 26 [integration/config] - 10https://gerrit.wikimedia.org/r/1343076 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:41:33] (03CR) 10CI reject: [V:04-1] Docker: Drop Node 24 images, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/1343077 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:41:38] (03CR) 10CI reject: [V:04-1] Docker: [sonar-scanner] Update Node from 24.18.0 to 26.8.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1343075 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:50:11] bah I broke the chain :b [11:52:04] (03CR) 10Hashar: "I don't remember offhand why the Sonar Scanner has NodeJS, probably to run ESLint or similar. The job is shared across multiple MediaWiki" [integration/config] - 10https://gerrit.wikimedia.org/r/1343076 (https://phabricator.wikimedia.org/T438085) (owner: 10Jforrester) [11:56:24] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Patchdemo: Do not delete wiki automatically on error - https://phabricator.wikimedia.org/T438722 (10jnuche) 03NEW [11:57:03] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Patchdemo: Do not delete wiki automatically on error - https://phabricator.wikimedia.org/T438722#12344859 (10jnuche) [12:11:59] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster seems to not propagate wikidata changes to subscribing clients - https://phabricator.wikimedia.org/T438656#12344877 (10awight) a:03awight [12:12:23] 10GitLab, 06Release-Engineering-Team: kokkuri:setup-variables failing for repos/releng/release!268 (due to being a fork?) - https://phabricator.wikimedia.org/T436899#12344881 (10Tgr) Same happening at Patchdemo (https://gitlab.wikimedia.org/tgr/patchdemo/-/pipelines/229200/failures) BTW kokkuri isn't indexed... [12:13:11] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Patchdemo: Do not delete wiki automatically on error - https://phabricator.wikimedia.org/T438722#12344884 (10jnuche) [12:19:02] 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Patchdemo: Do not delete wiki automatically on error - https://phabricator.wikimedia.org/T438722#12344900 (10jnuche) 05Openβ†’03Resolved [12:49:35] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster seems to not propagate wikidata changes to subscribing clients - https://phabricator.wikimedia.org/T438656#12345025 (10awight) [13:01:21] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster seems to not propagate wikidata changes to subscribing clients - https://phabricator.wikimedia.org/T438656#12345100 (10awight) [13:01:43] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster seems to not propagate wikidata changes to subscribing clients - https://phabricator.wikimedia.org/T438656#12345102 (10awight) [13:06:19] channel topic needs an update to fix the link to public logs [13:07:48] But I'm here to ask, is there a known reason that I would see beta cluster job-runner errors involving an old version of extension code? The filesystem shows php-master is up-to-date. [13:08:12] My specific error, if anyone is interested: /srv/mw-log/exception.log:2026-09-21 06:40:39.230302 [004ed7c5-8b07-445f-bf16-5de215b3a7ed] [13:15:58] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735 (10awight) 03NEW [13:16:02] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster wikibase repo cannot add sitelinks - https://phabricator.wikimedia.org/T438735#12345186 (10awight) [13:19:20] Unrelated: late-night attempts to run old PHP can be seen in php8.3-fpm.log . Maybe a cronjob? [13:34:22] (03PS1) 10Hashar: utils: script to verify package-lock.json integrity [integration/config] - 10https://gerrit.wikimedia.org/r/1343551 (https://phabricator.wikimedia.org/T437697) [13:37:09] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster jobrunner may be running stale code - https://phabricator.wikimedia.org/T438656#12345324 (10awight) [13:37:37] ^ I'm getting more convinced this is an issue with the beta cluster infrastructure. Maybe I have to manually refresh something for the jobrunner? [13:53:52] Anyone feel like helping me purge the opcache? [13:54:25] I'm imagining I just run "php7adm /opcache-free" [13:55:36] let's do this. [13:56:15] !log Purging opcache on deployment-jobrunner06 [13:56:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:58:28] That fixes my problem. [13:59:06] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster jobrunner may be running stale code - https://phabricator.wikimedia.org/T438656#12345398 (10awight) Solved by clearing the opcache: https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL#... [13:59:22] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Integration in Wikimedia projects (Kanban Board): Beta cluster jobrunner may be running stale code - https://phabricator.wikimedia.org/T438656#12345399 (10awight) a:05awightβ†’03None [14:04:04] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#12345439 (10Vgutierrez) `purge_old_cert` was a guard introduced to test... [14:11:36] (03PS2) 10Hashar: utils: script to verify package-lock.json integrity [integration/config] - 10https://gerrit.wikimedia.org/r/1343551 (https://phabricator.wikimedia.org/T437697) [14:21:29] (03open) 10jelto: digitaocean: use new vpc, egress gateway and isolated_workers in staging [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/633 (https://phabricator.wikimedia.org/T425441) [14:49:52] 10Continuous-Integration-Infrastructure, 10Castor, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12345729 (10dcaro) Hi, is this already sorted? Or there's anything on t... [14:58:33] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Quota-requests): Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345770 (10Andrew) Seems fine to me, but please check back and revert if the cache turns out to not help! +1 [15:06:26] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Quota-requests): Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345818 (10fnegri) +1 [15:07:02] 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Quota-requests): Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345821 (10dcaro) 05Openβ†’03In progress a:03dcaro [15:07:14] 10Continuous-Integration-Infrastructure, 06tools-platform-team, 10Cloud-VPS (Quota-requests): Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345824 (10dcaro) [15:13:33] 10Continuous-Integration-Infrastructure, 06tools-platform-team, 10Cloud-VPS (Quota-requests): Disk quota increase for the integration project - https://phabricator.wikimedia.org/T438516#12345867 (10dcaro) 05In progressβ†’03Resolved This should be up and running :) {F104136872} [15:13:45] 10WikimediaDebug, 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 30): Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240#12345876 (10KReid-WMF) [15:50:44] 10Continuous-Integration-Infrastructure, 06Collaboration-Services: Investigate using php-fpm for CI role - https://phabricator.wikimedia.org/T337299#12346158 (10LSobanski) 05Stalledβ†’03Declined We won't be updating the CI role as it's still slated for deprecation. [15:54:15] !log gerrit: on mediawiki/extensions/QuickSurveys deleted branch `master-backup` which was pointing at 1ad54717c059bcd49093d902eab2c098b4efe91a (which is in `master`) [15:54:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:59:42] 10Continuous-Integration-Infrastructure, 10Castor, 07Spike, 06Test Platform (Arusha 30): Test out the https://github.com/git-pkgs/proxy/ for CI - https://phabricator.wikimedia.org/T437005#12346198 (10AMarkossyan-WMF) 05Openβ†’03Resolved [15:59:48] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Arusha 30): Run npm cache verify once a day per cache instead of on every build - https://phabricator.wikimedia.org/T437377#12346200 (10AMarkossyan-WMF) 05Openβ†’03Resolved [16:04:51] (03open) 10dancy: build: Error when `run-variant` finds no entrypoint [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/178 [16:16:38] (03update) 10aklapper: Avoid incorrect mw.org user page URIs on Phab profile [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/28 (https://phabricator.wikimedia.org/T294719) [16:24:23] (03merge) 10dancy: Add developer-activity-pipelines [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/189 (https://phabricator.wikimedia.org/T433696 https://phabricator.wikimedia.org/T438042) (owner: 10aqu) [16:43:39] (03merge) 10dancy: perform-release: add user agent to requests going out to GitLab [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1314 (https://phabricator.wikimedia.org/T425441) (owner: 10jnuche) [16:46:07] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-extensions-DonationInterface, 06Moderator-Tools-Team, 10PersonalDashboard, and 8 others: Noisy log: normalized_message: ResourceLoader duplicate registration warning. Another module has already b... - https://phabricator.wikimedia.org/T438387#12346441 [16:47:19] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-extensions-DonationInterface, 06Moderator-Tools-Team, 10PersonalDashboard, and 5 others: Noisy log: normalized_message: ResourceLoader duplicate registration warning. Another module has already b... - https://phabricator.wikimedia.org/T438387#12346446 [16:48:59] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [16:51:25] (03update) 10dancy: Add linked artifact lambdas repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/191 (owner: 10akhatun) [16:52:22] (03merge) 10dancy: Add linked artifact lambdas repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/191 (owner: 10akhatun) [16:56:31] (03update) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [16:56:41] (03merge) 10dancy: deploy-service: Name the group of the plan from the plan [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1301 (https://phabricator.wikimedia.org/T412941) [16:57:48] (03update) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) [17:00:04] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12346546 (10dancy) Thanks @MoritzMuehlenhoff ! [17:00:25] (03merge) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941) [17:05:58] 06Release-Engineering-Team (Doing 😎), 10QS-Test-Automation, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Investigate retry mechanism for catalyst provisioning error - https://phabricator.wikimedia.org/T437955#12346577 (10jnuche) 05Openβ†’03Resolved Retry mechanism merged.... [17:06:41] (03update) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [17:08:59] 06Release-Engineering-Team (Doing 😎), 10QS-Test-Automation, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Investigate retry mechanism for catalyst provisioning error - https://phabricator.wikimedia.org/T437955#12346602 (10jnuche) [17:09:25] (03merge) 10dancy: install-world: Add an --inventory mode [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1315 [17:25:32] (03open) 10dancy: tests: Check that the apiserver serves every Vue route [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1318 [17:28:22] (03merge) 10dancy: tests: Check that the apiserver serves every Vue route [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1318 [17:58:32] #6 ERROR: docker-registry.wikimedia.org/golang1.15:1.15-1-20260621: not found [17:58:40] https://integration.wikimedia.org/ci/job/codesearch-pipeline-test/648/console for https://gerrit.wikimedia.org/r/c/labs/codesearch/+/1336608 [17:58:50] Not sure what changed but I'm guessing this isnt due to my patch [18:01:48] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for amintorbat - https://phabricator.wikimedia.org/T438603#12346933 (10Amintorbat) Thanks for the approval! Much appreciated 🌹 >>! In T438603#12344603, @Aklapper wrote: > Hi, I've approved your GitLab... [18:03:38] Krinkle: Is that a very old branch or stale repo? When the docker-registry was re-provisioned last month (earlier this month?) the very old images that hadn’t had any pulls for a while didn’t get copied across. https://docker-registry.wikimedia.org/ doesn’t know about golang 1.15, only 1.19/1.21/1.22/1.24/1.25 [18:05:56] it's whatever hound depends on in its blubber file [18:06:12] https://codesearch.wmcloud.org/search/?q=golang1.15&files=&excludeFiles=&repos= [18:07:26] I dont' know where those images are useful. Afaik codesearch service doesn't actually pull this image [18:07:32] so it's presumably dev-only [18:07:54] the frontend has its own Dockerfile, and the hound service ... [18:08:00] ah yes, that indeed pulls it from docker-registry [18:08:05] $ sudo docker pull docker-registry.wikimedia.org/wikimedia/labs-codesearch:hound [18:08:05] $ sudo systemctl restart hound_proxy # Trigger a rolling restart of all instances [18:08:14] Huh, yeah, very stale then? [18:08:16] so it's running on wmcs as we speak with that parent [18:08:39] Hmm, no, https://gerrit.wikimedia.org/r/plugins/gitiles/labs/codesearch/+/c4365e013c36f24dba16d49b50b61cfa603e1ea0%5E%21/.pipeline/blubber.yaml is from mutante from June. [18:08:44] (03open) 10dancy: jobrunner: Run jobs in separate queues concurrently [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1319 (https://phabricator.wikimedia.org/T412941) [18:09:10] it makes sense it wouldn't have pulls because the running server has it scached [18:09:16] and that layer doesnt' change [18:09:26] Right, but unfortunate. [18:10:09] hound hasn't had a release for a while, and I don't know much about golang to know if it has breaking changes [18:10:17] v0.7.1 supports golang 1.4 or later [18:10:28] (03update) 10dancy: jobrunner: Run jobs in separate queues concurrently [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1319 (https://phabricator.wikimedia.org/T412941) [18:10:30] (03update) 10dancy: jobrunner: Run jobs in separate queues concurrently [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1319 (https://phabricator.wikimedia.org/T412941) [18:10:31] which upstream tested at the time with: ["1.18", "1.17", "1.14"] [18:10:45] so.. in theory this might work [18:11:05] I guess this is as good a time as any to try to upgrade and see what happens [18:11:11] I dont' want to lose the existing reference though [18:12:42] did it actually get removed from git? [18:12:43] https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1337931 [18:12:45] yes it did [18:12:53] September 8 [18:13:02] * Krinkle files task [18:13:37] If for no other reason than to make sure next time codesearch is checked before removal [18:13:59] 10Deployments, 06Release-Engineering-Team (Radar), 06ServiceOps: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232#12347005 (10MLechvien-WMF) @dancy Does this need Serviceops support, or do you have the means to control this? [18:16:53] 10Deployments, 06Release-Engineering-Team (Radar), 06ServiceOps: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232#12347029 (10dancy) >>! In T438232#12347005, @MLechvien-WMF wrote: > @dancy Does this need Serviceops support, or do you have the... [18:17:14] (03open) 10thcipriani: Train dashboard: add two-week-avg-lead-time [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/73 [18:18:01] (03merge) 10thcipriani: Train dashboard: add two-week-avg-lead-time [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/73 [18:21:34] 10Deployments, 06Release-Engineering-Team (Radar), 06ServiceOps (Next quarter), 07Serviceops-easywins: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232#12347064 (10MLechvien-WMF) p:05Triageβ†’03Medium Ack. Setting to Medium as I understand... [18:24:17] James_F: thanks for connecting the dots [18:24:47] ref T438777) [18:24:48] T438777: Codesearch can't build: ERROR: golang1.15 not found - https://phabricator.wikimedia.org/T438777 [18:27:13] (03update) 10dancy: jobrunner: Run jobs in separate queues concurrently [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1319 (https://phabricator.wikimedia.org/T412941) [18:29:49] (03merge) 10dancy: jobrunner: Run jobs in separate queues concurrently [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1319 (https://phabricator.wikimedia.org/T412941) [18:32:29] (03open) 10dancy: spiderpig: Add support for retrying a service deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1320 (https://phabricator.wikimedia.org/T412941) [18:35:20] (03merge) 10dancy: spiderpig: Add support for retrying a service deployment [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1320 (https://phabricator.wikimedia.org/T412941) [18:38:08] (03open) 10dancy: spiderpig: Update notifications to handle multiple jobs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1321 (https://phabricator.wikimedia.org/T412941) [18:40:27] (03merge) 10dancy: spiderpig: Update notifications to handle multiple jobs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1321 (https://phabricator.wikimedia.org/T412941) [18:41:49] (03open) 10dancy: Update image references [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/179 [18:45:18] (03update) 10dancy: web/eslint.config.js: Lint TypeScript files too [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1322 [18:45:19] (03open) 10dancy: web/eslint.config.js: Lint TypeScript files too [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1322 [18:48:14] (03merge) 10dancy: web/eslint.config.js: Lint TypeScript files too [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1322 [18:48:53] (03merge) 10dancy: Update image references [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/179 [18:49:35] (03update) 10dancy: build: Error when `run-variant` finds no entrypoint [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/178 [18:51:42] (03open) 10dancy: spiderpig: Share the formatter of a linkified message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1323 [18:54:56] (03merge) 10dancy: spiderpig: Share the formatter of a linkified message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1323 [18:55:38] (03update) 10dancy: build: Error when `run-variant` finds no entrypoint [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/178 [18:57:07] (03update) 10dancy: spiderpig: Add a note to a MediaWiki error message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1324 [18:57:10] (03open) 10dancy: spiderpig: Add a note to a MediaWiki error message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1324 [18:58:54] (03approved) 10dduvall: build: Error when `run-variant` finds no entrypoint [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/178 (owner: 10dancy) [19:08:08] 10Phabricator (2026-09-15): Assign user - https://phabricator.wikimedia.org/T438783 (10---2044-50775-60) 03NEW [19:38:05] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:01:04] FIRING: PuppetAgentDisabled: Puppet agent disabled on instance deployment-mwmaint03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentDisabled [20:01:10] 10Beta-Cluster-Infrastructure: Puppet agent disabled on instance deployment-mwmaint03 in project deployment-prep - https://phabricator.wikimedia.org/T438787 (10wmcs-alerts) 03NEW [20:57:39] (03merge) 10dancy: build: Error when `run-variant` finds no entrypoint [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/178 [21:00:11] (03open) 10dancy: examples: Check the error before the deferred close [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/180 [21:04:09] (03update) 10dancy: spiderpig: Add a note to a MediaWiki error message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1324 [21:04:12] (03update) 10dancy: spiderpig: Add a note to a MediaWiki error message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1324 [21:13:08] (03merge) 10dancy: spiderpig: Add a note to a MediaWiki error message [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1324 [21:14:56] (03update) 10dancy: spiderpig: Keep the navigation bar in step with the page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1316 (https://phabricator.wikimedia.org/T433287) [21:16:35] (03merge) 10dancy: examples: Check the error before the deferred close [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/180 [21:17:43] (03merge) 10dancy: spiderpig: Keep the navigation bar in step with the page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1316 (https://phabricator.wikimedia.org/T433287) [21:19:13] (03open) 10dancy: spiderpig: JobHistory: Only show jobs of a particular type [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1325 [21:20:48] (03merge) 10dancy: spiderpig: JobHistory: Only show jobs of a particular type [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1325 [21:22:26] (03open) 10dancy: logstash: Strip the exception prefix in one place [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1326 [21:24:56] (03merge) 10dancy: logstash: Strip the exception prefix in one place [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1326 [21:48:19] (03open) 10dancy: spiderpig: Add Phatality-style Phabricator error message search [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1327 (https://phabricator.wikimedia.org/T397094) [21:50:47] (03merge) 10dancy: spiderpig: Add Phatality-style Phabricator error message search [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1327 (https://phabricator.wikimedia.org/T397094) [21:53:13] (03open) 10dancy: integration/test_backport.py: Retrieve the commit-msg hook over HTTP [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1328 [21:55:59] (03merge) 10dancy: integration/test_backport.py: Retrieve the commit-msg hook over HTTP [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1328 [22:02:24] (03update) 10dancy: Release 4.290.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1329 [22:02:27] (03open) 10dancy: Release 4.290.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1329 [22:05:28] (03merge) 10dancy: Release 4.290.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1329 [22:23:01] (03update) 10dancy: Makefile: Add targets for the local development environment [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/181 [22:23:02] (03update) 10dancy: CONTRIBUTING.md: Update instructions [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/182 [22:23:16] (03open) 10dancy: Makefile: Add targets for the local development environment [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/181 [22:23:16] (03open) 10dancy: CONTRIBUTING.md: Update instructions [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/182