[00:06:08] 10Beta-Cluster-Infrastructure, 07Wikimedia-production-error: DBQueryError "The MariaDB server is running with the --read-only option" fails MainStash in Beta Cluster - https://phabricator.wikimedia.org/T401227#11165786 (10bd808) 05Open→03Resolved a:03bd808 Last error seen at 2025-09-09T21:36:08.642Z... [00:47:11] 06Release-Engineering-Team, 10Catalyst (PatchDemo): Enable creation of a patchdemo with all repos rolled back to a specific custom moment in time - https://phabricator.wikimedia.org/T403854#11165856 (10Tacsipacsi) Would that also mean that stack trace links #wikimedia-production-error tasks that haven’t been a... [06:07:37] Hello, if any folks with phab-account-disabling permissions are online: https://phabricator.wikimedia.org/p/Alejano564/ [07:53:21] ^ that got dealt with [08:00:03] maintenance-disconnect-full-disks build 735770 integration-agent-docker-1063 (/: 24%, /srv: 97%, /var/lib/docker: 46%): OFFLINE due to disk space [08:05:03] maintenance-disconnect-full-disks build 735771 integration-agent-docker-1063 (/: 24%, /srv: 42%, /var/lib/docker: 45%): RECOVERY disk space OK [08:06:24] 10Phabricator, 06Project-Admins: Create project tag for ESEAP Hub - https://phabricator.wikimedia.org/T403908#11166212 (10Aklapper) Requested public project #eseap-hub has been created: https://phabricator.wikimedia.org/project/view/8165/ (In case you need to edit the project or project workboard itself at so... [08:06:31] 10Phabricator, 06Project-Admins: Create project tag and private Space for ESEAP Hub - https://phabricator.wikimedia.org/T403908#11166213 (10Aklapper) [08:10:10] hashar: looks like codesearch is down... who do I ping about that? [08:10:38] I won't make it to our chat today btw, I have to get stuff done before the envoy training in the afternoon [08:34:56] duesen: sorry I am late [08:40:35] I have filed codesearch being unresponsive as https://phabricator.wikimedia.org/T404163 [08:42:32] 10Phabricator, 06Project-Admins: Create project tag and private Space for ESEAP Hub - https://phabricator.wikimedia.org/T403908#11166277 (10Biyanto-ESEAP) Hi @Aklapper can you add my username to the #Trusted-Contributors? I want to add a subproject and edit the details, but I am unable to do so, thank you! [08:48:42] 10Phabricator, 06Project-Admins: Create project tag and private Space for ESEAP Hub - https://phabricator.wikimedia.org/T403908#11166310 (10Robertsky) >>! In T403908#11166277, @Biyanto-ESEAP wrote: > Hi @Aklapper can you add my username to the #Trusted-Contributors? I want to add a subproject and edit the deta... [08:53:57] 10Phabricator, 06Project-Admins: Create project tag and private Space for ESEAP Hub - https://phabricator.wikimedia.org/T403908#11166320 (10Robertsky) @Biyanto-ESEAP what is need is membership to [[ https://phabricator.wikimedia.org/project/profile/1776/| acl*Project-Admins ]]. I am requesting it there for us... [08:56:04] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11166325 (10Robertsky) Hi there, requesting for myself and @Biyanto-ESEAP to ACL Project Admin to manage project/sub-projects at https://phabricator.wikimedia.... [09:00:30] (03CR) 10Hashar: "That is `zuul_deps` being `None` which comes from `yaml.safe_load()` reading nothing. So I guess `zuul/layout.yaml` is empty/invalid?" [integration/config] - 10https://gerrit.wikimedia.org/r/1182115 (https://phabricator.wikimedia.org/T389998) (owner: 10Hashar) [09:02:53] duesen: code search is back up and still warming up :] [09:31:55] Project mediawiki-core-doxygen build #13454: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13454/ [10:18:03] hashar: thank you :) [10:31:12] Yippee, build fixed! [10:31:12] Project mediawiki-core-doxygen build #13455: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13455/ [10:46:12] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11166737 (10Jgiannelos) After applying this ^ patch locally and after allowi... [11:18:30] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11166849 (10Jgiannelos) i just verified it, PCS on deployment-prep should be... [11:33:39] duesen: the codesearch instance required a full reboot [11:55:38] 10Continuous-Integration-Config, 10Testing Support: Jenkins selenium job should fail when all tests are skipped - https://phabricator.wikimedia.org/T324480#11166966 (10zeljkofilipin) [11:58:26] 10Beta-Cluster-Infrastructure, 10Testing Support, 07OKR-Work: Make selenium users use botflags at beta-cluster by setting the bot flag (passing "bot=1" parameter) when saving edits - https://phabricator.wikimedia.org/T116027#11167016 (10zeljkofilipin) [14:04:40] 10Continuous-Integration-Config, 06Reader Growth Team, 06SRE Observability, 10Reader Experience Team (REx Sprint 4 [Q1 Aug 26-Sept 8 '25]), and 2 others: Setup web team performance alerts in Slack - https://phabricator.wikimedia.org/T392298#11167634 (10SToyofuku-WMF) 05Open→03Resolved We are indeed... [15:16:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 10function-evaluator, and 4 others: Wikifunctions function orchestrator and evaluator test suites failing on GitLab CI with OOM errors - https://phabricator.wikimedia.org/T399348#11168011 (10bd808) @ecarg & @cmassaro: @dancy... [15:26:58] bd808: what mutante said :) i hacked on argument support a while back and it turned out to be more complicated than i originally thought due to the question of scope: where and when should argument expressions be evaluated? [15:27:49] if it's only in command/script values, that's quite easy (we add an env variable into the llb.state chain just like other build instructions). if it should apply to other configuration fields, that's a bit harder (we'd probably need some kind of pre-validation substitution) [15:44:56] (03update) 10brennen: Merge Phorge upstream master as of 2025-09-08 into wmf/stable [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/99 (https://phabricator.wikimedia.org/T404134) [15:45:08] dduvall: yeah, I remember a bit of discussion with you about some of this when I was adding the RUNS_AS bits for my local dev stuff. The thing I'm thinking about right now could be scoped to allowing an ARG to set an ENVVAR. I'm not sure how broadly this would be useful. [15:48:16] i could see it being useful even with the reduced scope. blubber's own `blubber.yaml` would definitely benefit https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/c6ddfb1851a3e9eafee82ae789e71fc0344ef76b/.pipeline/blubber.yaml#L64 [15:48:39] (the `make` targets in those variants) [16:14:59] 10Phabricator: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214 (10Damilare) 03NEW [16:22:07] 10Phabricator, 06Release-Engineering-Team: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11168391 (10brennen) Will need some investigation. May be Phorge upstream (#phabricator-upstream). [16:22:33] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11168406 (10hashar) >>! In T401157#11156425, @Aklapper wrote: > @hashar: Uh, thanks for digging into that.... [16:28:35] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11168467 (10Aklapper) Display glitch upstreamed as https://we.phorge.it/T16261 [16:41:12] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11168585 (10hashar) Thank you @Aklapper! [17:12:58] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: buildkit v0.24.0 released - https://phabricator.wikimedia.org/T403625#11168824 (10dancy) CPU usage of one of the droplets that was running buildkitd. {F66003418 size=full} [17:16:30] 10Beta-Cluster-Infrastructure, 10RESTBase, 07Beta-Cluster-reproducible, 13Patch-For-Review: HyperSwitch/errors/not found (404) on beta cluster: There was an issue displaying this preview - https://phabricator.wikimedia.org/T402206#11168835 (10Jdlrobson-WMF) Thanks for looking at this! I'm still not seeing... [17:32:15] 10Phabricator, 06Release-Engineering-Team: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11168913 (10Aklapper) 05Open→03Stalled @Damilare: Hi, what is an exact call with parameters which triggers this / what are good-enough steps to reproduce? Which o... [17:34:26] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11168923 (10Aklapper) ...and I guess someone overwrote the visualization again. Is there any... [17:47:57] (03update) 10bd808: Draft: hiera: Wrap YAML file imports to avoid dirty diffs [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/51 (https://phabricator.wikimedia.org/T398643) [17:49:02] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11168976 (10aude) ReadingLists tests broke again due to a change in core... [17:50:17] (03update) 10bd808: Upgrade cloudvps provider and wrap YAML imports to avoid dirty diffs [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/51 (https://phabricator.wikimedia.org/T398643) [17:53:00] 10Phabricator, 06Release-Engineering-Team: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11169019 (10Damilare) Thanks @Aklapper, I just made another attempt at September 10, 5:47 PM UTC. Here's the [[ https://gitlab.wikimedia.org/damilare/phab_tools/-/blo... [17:54:45] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11169029 (10Jdlrobson-WMF) [18:09:18] (03open) 10brennen: Merge upstream Phorge master as of 2025-09-08 into wmf/stable [repos/phabricator/arcanist] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/arcanist/-/merge_requests/5 (https://phabricator.wikimedia.org/T404134) [18:10:14] 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Merge Phorge's upstream master into our wmf/stable - https://phabricator.wikimedia.org/T404134#11169110 (10brennen) [18:16:41] (03open) 10brennen: update submodules to recent phorge upstream master (2025-09-08) [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/80 (https://phabricator.wikimedia.org/T404134) [18:28:26] 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: Merge Phorge's upstream master into our wmf/stable - https://phabricator.wikimedia.org/T404134#11169251 (10brennen) [18:30:01] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11169259 (10Aklapper) 05Stalled→03Open p:05Triage→03Medium a:03Aklapper [18:32:32] (03PS1) 10SomeRandomDeveloper: Zuul: Add Peter Li to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1187061 [18:37:28] (03open) 10dancy: train-dev: Fix typo [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/159 [18:37:31] (03update) 10dancy: train-dev: Fix typo [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/159 [18:37:51] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11169337 (10thcipriani) >>! In T401614#11121806,... [18:38:34] (03merge) 10dancy: train-dev: Fix typo [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/159 [18:40:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11169344 (10Dzahn) ` root@zuul1001:/home/dzahn# s... [18:41:20] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11169350 (10Aklapper) Thanks! I can reproduce locally now. Fix is a simple one-liner that we may want to backport: https://we.phorge.it/D263... [18:44:04] 10Continuous-Integration-Config, 06Growth-Team, 10Metrics Platform, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11169377 (10Jdforrester-WMF) The decision is not mine to make, but Release Engineering. See T403560#11150311 for the same set of conce... [18:48:09] (03CR) 10Jforrester: [C:03+2] Zuul: Add Peter Li to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1187061 (owner: 10SomeRandomDeveloper) [18:49:42] (03Merged) 10jenkins-bot: Zuul: Add Peter Li to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1187061 (owner: 10SomeRandomDeveloper) [18:50:34] !log Zuul: Add Peter Li to CI allowlist [18:50:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:51:04] (03PS2) 10Jforrester: Revert^2 "Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1185128 [18:51:07] (03CR) 10Jforrester: [C:03+2] Revert^2 "Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1185128 (owner: 10Jforrester) [18:52:38] (03Merged) 10jenkins-bot: Revert^2 "Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency" [integration/config] - 10https://gerrit.wikimedia.org/r/1185128 (owner: 10Jforrester) [18:53:07] !log Zuul: [mediawiki/extensions/PageViewInfo] Drop Graph dependency (again) [18:53:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:41] (03open) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:12:44] (03update) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:13:09] bd808/mutante: successful nerd sniping [19:13:13] ^ [19:13:54] (03update) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:14:12] dduvall: hah!:) nice [19:17:36] (03update) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:17:36] (03update) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:17:39] (03update) 10dduvall: arguments: Support build arguments [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/150 [19:28:21] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 06Reader Experience Team, 13Patch-For-Review: ReadingLists tests not run in jenkins wmf-quibble-core-vendor-mysql-php81 - https://phabricator.wikimedia.org/T403560#11169615 (10thcipriani) >>! In T403560#11160551, @Jdlrobson-WMF wrote: >... [20:12:38] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream: Phabricator manifest edit API failing with a 200 response - https://phabricator.wikimedia.org/T404214#11169773 (10Damilare) Awesome, thanks! Yes the last time I successfully ran the script was in late June. The 200 response made it a bit hard... [20:16:41] (03merge) 10bd808: Upgrade cloudvps provider and wrap YAML imports to avoid dirty diffs [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/51 (https://phabricator.wikimedia.org/T398643) [20:18:45] (03update) 10bd808: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52 [20:29:17] James_F: I am going to bed, but I replied to your question about ./utils/mw-requires.py ( https://gerrit.wikimedia.org/r/c/integration/config/+/1182115 ). I think the issue is an empty dependencies.yaml or a wrong file somehow :/ [20:29:21] 10Scap: scap complaining about dependency which is already merged - https://phabricator.wikimedia.org/T388025#11169823 (10dancy) @Reedy reported another instance of this problem today in IRC ` reedy@deploy1003:/srv/mediawiki-staging$ scap backport 1187062 1187063 1187066 1187065 20:23:37 Checking whether reques... [20:30:26] 10Scap: scap complaining about dependency which is already merged - https://phabricator.wikimedia.org/T388025#11169828 (10Reedy) `lang=bash reedy@deploy1003:/srv/mediawiki-staging$ scap backport 1187062 1187063 1187066 1187065 20:26:26 Checking whether requested changes are in a branch deployed to production and... [20:31:50] Project mediawiki-core-doxygen build #13467: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13467/ [20:33:42] (03CR) 10Hashar: "Based on the commit message, I wrote a script to generate the list of dependencies ( https://gerrit.wikimedia.org/r/c/integration/config/+" [integration/config] - 10https://gerrit.wikimedia.org/r/1132543 (https://phabricator.wikimedia.org/T389998) (owner: 10Krinkle) [20:59:17] 10Scap: scap complaining about dependency which is already merged - https://phabricator.wikimedia.org/T388025#11169945 (10hashar) > 1124783 is an 1.44.0-wmf.18 backport. 1121797 was merged to master sufficiently long ago that it's now included in 1.44.0-wmf.18. Not a big deal but it would be nice if scap could f... [21:32:51] Yippee, build fixed! [21:32:51] Project mediawiki-core-doxygen build #13468: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/13468/ [22:01:21] (03PS1) 10Jforrester: jjb: Rename the wmf-quibble… jobs to quibble-with-gated-extensions… [integration/config] - 10https://gerrit.wikimedia.org/r/1187112 [22:01:21] (03PS1) 10Jforrester: Zuul: Use quibble-with-gated-extensions… jobs over wmf-quibble… [integration/config] - 10https://gerrit.wikimedia.org/r/1187113 [22:01:21] (03PS1) 10Jforrester: jjb: Drop old wmf-quibble- jobs, renamed [integration/config] - 10https://gerrit.wikimedia.org/r/1187114 [22:04:18] (03CR) 10Jforrester: "Antoine, what do you think of this rename? There's a lot of confusion about what gated extensions currently do, and I think this would hel" [integration/config] - 10https://gerrit.wikimedia.org/r/1187112 (owner: 10Jforrester) [22:23:10] Project beta-update-databases-eqiad build #87575: 04FAILURE in 3 min 9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/87575/ [22:32:35] 10Continuous-Integration-Config, 07Developer Productivity, 10MobileFrontend (JavaScript core library): Export Popups and MobileFrontend dist files as CI artifacts - https://phabricator.wikimedia.org/T352240#11170244 (10Jdlrobson-WMF) [22:38:57] 10Continuous-Integration-Config, 06Growth-Team, 10Metrics Platform, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11170288 (10thcipriani) > From chat: @thcipriani can you give additional clarity or guidance on next steps here from a releng POV? From... [23:24:01] Yippee, build fixed! [23:24:01] Project beta-update-databases-eqiad build #87576: 09FIXED in 4 min 0 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/87576/ [23:47:55] (03update) 10bd808: ci: Use wmcs runners and registry.cloud.releng.team [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/52