[01:53:13] 06Project-Admins: Request to create project: Wikidata Reference Validator - https://phabricator.wikimedia.org/T403556#11142149 (10Bugreporter) [01:56:13] 06Project-Admins: Request to create project: Wikidata Reference Validator - https://phabricator.wikimedia.org/T403556#11142155 (10Bugreporter) Where does the tool host? If it is hosted in Toolforge, you don't need to file a task here. Please instead visit https://toolsadmin.wikimedia.org/tools/id/. [06:32:45] 06Project-Admins: Request to create project: Wikidata Reference Validator - https://phabricator.wikimedia.org/T403556#11142299 (10Aklapper) 05Openβ†’03Stalled > Suggested tags: I'm not sure what that means / where it comes from. See https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects for... [08:43:27] 10Phabricator: Fix commit identity of Valerio Bozz. - https://phabricator.wikimedia.org/T381461#11142653 (10Aklapper) [09:59:41] 10Phabricator, 10Wikibugs: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11143080 (10Aklapper) p:05Triageβ†’03Low [10:24:32] 10Phabricator (Upstream), 07CSS, 07Upstream: "Change Story Points (Estimate)" overflow - https://phabricator.wikimedia.org/T264253#11143211 (10Aklapper) 05Openβ†’03Resolved This seems to work as expected nowadays: The overflown text is shown ellipsized in this very case. {F65952232} [10:49:28] 10Phabricator, 10Phabricator (Upstream), 07Upstream: Phabricator needs to handle bounces/errors from non-existent email addresses - https://phabricator.wikimedia.org/T100400#11143318 (10Aklapper) I have not seen any SMTP errors in `/var/log/phd/daemons.log` on `phab1004` for ages (it's nearly only repository... [11:00:07] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06DBA: Drop unexpected/unneeded database tables in Phabricator - https://phabricator.wikimedia.org/T403542#11143347 (10Ladsgroup) Once you have been 100% we can drop it. Ping me and I do the needful. [12:38:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:38:33] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616 (10wmcs-alerts) 03NEW [12:39:35] (03CR) 10Jforrester: "Adding an extension to the gate is an extremely expensive step in terms of engineering time, and potentially CI time." [integration/config] - 10https://gerrit.wikimedia.org/r/1184176 (https://phabricator.wikimedia.org/T403560) (owner: 10Aude) [12:58:28] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [13:13:08] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11143877 (10cscott) We put frequently-modified extensions into gate, but... [13:28:20] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11143924 (10Reedy) It’s also the number of tests, their speed, and in man... [13:31:52] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11143945 (10cscott) Yeah, I strongly suspect if we tried to run all 100+... [14:06:14] (03CR) 10Krinkle: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [14:08:26] (03CR) 10CI reject: [V:04-1] Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [14:21:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Security: Jenkins plugins security advisory 2025-09-03 - https://phabricator.wikimedia.org/T403623 (10jnuche) 03NEW [14:37:05] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11144164 (10Krinkle) Looks like it ain't me this time. `counterexample krinkle@deployment-cache-text08:~$ sudo run-puppet-agent Info: Using envi... [14:51:12] (03CR) 10Hashar: "That is never ending, I wonder why it was not an issue previously so I guess I am going to debug it!" [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [14:58:21] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.24.0 released - https://phabricator.wikimedia.org/T403625 (10dancy) 03NEW [15:11:36] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06DBA: Drop unexpected/unneeded database tables in Phabricator - https://phabricator.wikimedia.org/T403542#11144271 (10Aklapper) Tyler had the idea that this could have been related to our ElasticSearch backend experiments, however grep'ing our... [15:12:28] (03CR) 10Hashar: "I found the issue and that is due to tox v4." [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:12:34] (03PS6) 10Hashar: Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 [15:16:47] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11144290 (10bd808) This is a new dataset added to prod: * https://gerrit.wikimedia.org/r/c/operations/puppet/+/1181090 * https://gerrit.wikimedia... [15:19:35] (03CR) 10Hashar: [C:03+1] "This time I have done a rebuild of the failing job against patchset 6 and it failed eventually:" [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:19:43] (03CR) 10Hashar: [C:04-1] Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:23:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:27:02] (03PS7) 10Hashar: Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 [15:30:55] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06DBA: Drop unexpected/unneeded database tables in Phabricator - https://phabricator.wikimedia.org/T403542#11144363 (10Aklapper) [15:38:52] (03PS8) 10Hashar: Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 [15:44:02] (03CR) 10Hashar: [C:03+1] "I have added `query_plugins_info = False`Β to the JJB config, else it queries info about each of the plugins which is kind of slow." [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [15:55:03] maintenance-disconnect-full-disks build 733849 integration-agent-docker-1053 (/: 31%, /srv: 100%, /var/lib/docker: 32%): OFFLINE due to disk space [16:00:03] maintenance-disconnect-full-disks build 733850 integration-agent-docker-1053 (/: 31%, /srv: 12%, /var/lib/docker: 30%): RECOVERY disk space OK [16:47:06] (03PS9) 10Jforrester: Replace utils/jenkins-jobs-list.py by ./jjb-list [integration/config] - 10https://gerrit.wikimedia.org/r/1183076 (owner: 10Hashar) [17:11:35] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11144883 (10Jdlrobson-WMF) p:05Triageβ†’03Medium [17:22:21] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11144936 (10bd808) `lang=shell-session,counterexample root@deployment-puppetserver-1:/srv/puppet_fileserver/volatile# systemctl status dump_datac... [17:25:03] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11144941 (10bd808) [17:53:46] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11145083 (10ssingh) @SLyngshede-WMF, @Vgutierrez: For https://gerrit.wikimedia.org/r/c/operations/puppet/+/1181090 and https://gerrit.wikimedia.o... [17:54:18] 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T403616#11145087 (10ssingh) [18:28:21] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11145246 (10Jdlrobson-WMF) Given ReadingLists is a feature used by all of... [18:35:55] 10Release-Engineering-Team (Doing 😎), 07Epic, 07OKR-Work: [FY24-25 WE6.2.1] Publish pre-train single version containers - https://phabricator.wikimedia.org/T369115#11145294 (10CCiufo-WMF) [18:36:07] 10Release-Engineering-Team (Doing 😎), 07OKR-Work, 10Test-Platform (Radar): [FY24-25 WE6.2.6] Create design document for Pretrain (nΓ©e Group -1) deployment - https://phabricator.wikimedia.org/T379683#11145295 (10CCiufo-WMF) [18:36:24] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Epic, 07OKR-Work: [FY25-26 WE6.1.1] Move image build to deployment server and update for backports - https://phabricator.wikimedia.org/T398868#11145296 (10CCiufo-WMF) [20:28:40] (03PS2) 10Jforrester: jjb: Send Jenkins alerts to Slack for 3 Readers group repos' daily Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/1184164 (owner: 10Jdlrobson) [20:29:34] (03PS3) 10Jforrester: jjb: Send Jenkins alerts to Slack for 3 Readers group repos' daily Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/1184164 (owner: 10Jdlrobson) [20:29:54] (03CR) 10Jforrester: [C:03+2] "Deployed. If there are any issues, please shout!" [integration/config] - 10https://gerrit.wikimedia.org/r/1184164 (owner: 10Jdlrobson) [20:31:16] (03Merged) 10jenkins-bot: jjb: Send Jenkins alerts to Slack for 3 Readers group repos' daily Selenium [integration/config] - 10https://gerrit.wikimedia.org/r/1184164 (owner: 10Jdlrobson) [20:50:44] (03CR) 10Jforrester: [C:04-1] zuul: Add ReadingLists to extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/1184176 (https://phabricator.wikimedia.org/T403560) (owner: 10Aude) [20:53:15] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11145777 (10Jdforrester-WMF) [20:53:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10MW-on-K8s, 07Epic, 13Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674#11145778 (10Jdforrester-WMF) [22:28:41] (03open) 10bd808: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52 [22:39:14] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11146064 (10bd808) In another version of this discussion @dancy wondered if we could just use the registry.cloud.releng.team Reggie instance. I am tr... [23:02:03] (03update) 10bd808: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52 [23:06:22] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11146116 (10dancy) Relevant parts of the log prior to the final buildctl failure: ` #18 [downloads] πŸ–₯️ @65533 $ [script@66acb954] 0 0 0... [23:09:14] 10Continuous-Integration-Config, 06Reader Growth Team, 06SRE Observability, 10Reader Experience Team (REx Sprint 4 [Q1 Aug 26-Sept 8 '25]), and 2 others: Setup web team performance alerts in Slack - https://phabricator.wikimedia.org/T392298#11146121 (10Jdlrobson-WMF) [23:09:48] 10Continuous-Integration-Config, 06Reader Growth Team, 06SRE Observability, 10Reader Experience Team (REx Sprint 4 [Q1 Aug 26-Sept 8 '25]), and 2 others: Setup web team performance alerts in Slack - https://phabricator.wikimedia.org/T392298#11146124 (10Jdlrobson-WMF) a:05Jdlrobson-WMFβ†’03SToyofuku-WMF O... [23:10:03] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11146126 (10bd808) My build in T396924#11146064 was failing because of github download rate limits I think. https://gitlab.wikimedia.org/repos/releng... [23:23:05] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11146141 (10dancy) It looks like we do have things configured to deny non-GET/HEAD requests to Reggie that do not originate from the gitlab-cloud-run... [23:24:57] 10GitLab (CI & Job Runners): kokkuri cannot publish "public" images from WMCS runners due to a lack of a local registry - https://phabricator.wikimedia.org/T396924#11146145 (10bd808) @thcipriani found the same bits for me. So I guess we could poke a hole for 185.15.56.1 (nat.cloudgw.eqiad1.wikimediacloud.org) or...