[03:20:04] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11076479 (10Eileenmcnaughton) Yes - this looks good - the test issue does not seem to relate now [06:34:05] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build - missing patch & mysqli? - https://phabricator.wikimedia.org/T401451#11076724 (10hashar) 05In progressβ†’03Resolved Great, thank you for the confirmation :) [06:38:00] 10GitLab (Infrastructure), 06collaboration-services: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11076727 (10Jelto) >>! In T400971#11074365, @Milimetric wrote: > I know of a few examples of gitlab packaging jobs failing but here's one that I tried to run... [07:16:06] (03CR) 10Hashar: [C:03+2] build: Updating mediawiki/mediawiki-phan-config to 0.17.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1177870 (owner: 10Libraryupgrader) [07:32:37] 10Gerrit: Gerrit rate limit exceptions for Wikimania 2025 - https://phabricator.wikimedia.org/T401205#11076859 (10Chlod) 05Openβ†’03Invalid [07:51:24] 06Release-Engineering-Team, 06Fundraising-Backlog, 07Essential-Work: Update nose to pytest on fundraising-tools - https://phabricator.wikimedia.org/T399304#11076942 (10hashar) [09:40:02] 10GitLab (Infrastructure), 06collaboration-services: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11077302 (10Milimetric) @Jelto : sorry, should've mentioned, yes we retried and it worked with the throttling disabled. [10:38:35] (03merge) 10hashar: Drop all buster-based images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/83 (https://phabricator.wikimedia.org/T382379) (owner: 10dancy) [11:02:17] Hi folks. We've been seeing consistent failures in the daily selenium jobs for a few weeks now, across multiple components, see T372068#11075998. The issue is that API requests to beta produce an invalid JSON response, which could possibly mean an HTML response. In T372068#11076592, Peter mentioned the possibility of our test runner being blocked. Would it be possible for someone to verify if this is the case? [11:02:18] T372068: Selenium test failure when deprecation messages happen in API calls: "invalidjson: No valid JSON response" - https://phabricator.wikimedia.org/T372068 [11:02:43] A recent run is https://integration.wikimedia.org/ci/job/selenium-daily-beta-CampaignEvents/499/console, which made an API request today between 2:18:37 and 2:18:42 UTC [11:03:23] can you somehow run the tests in a way that it prints out the http response body? [11:04:52] I think that's what folks in QTE wanted to try doing next. mwbot doesn't do that so that'd be a local hack for the time being [11:05:52] And in general, we definitely should have something like that for proper debugging. But for the immediate issue, I think it'd be worth ruling out IP blocks given the situation with beta... [11:16:04] was there a reason to abort my job? [11:19:02] (a bunch of jobs here https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1176475 show "Aborted by Daimona Eaytoy", but I wanted to check in before I resubmit them) [11:19:28] I aborted them because one of the jobs for that build had already failed [11:19:45] And I was trying to reduce CI load to avoid further flakiness [11:20:04] okay. There are some ongoing phan issues in Wikibase because of the fallout from the Status change [11:20:10] but it's still useful to me to get Selenium results [11:21:22] Is CI struggling? Or can I put the jobs back in? [11:24:31] It's not struggling too badly, so I think you can put it back in. It's just that Wikibase builds run a lot of tests (understandably), so I'd rather avoid unnecessary runs if at all possible [11:24:31] (also the patch that would fix phan for Wikibase seems to have been aborted during gate-and-submit... https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1177971) [11:25:14] Yeah, I aborted that too because it already had a Random Selenium Failure (TM) in another job [11:25:51] fair. then I'll wait for that to get merged before I resubmit my other tests [11:29:58] Yeah that would work. Hopefully stuff passes this time. [11:34:11] Daimona: mwbot request exception has the response attached when it rejects the promise [11:35:05] For some reason the resubmit is just waiting for jobs https://integration.wikimedia.org/zuul/#q=1177971 . Doesn't even look like there's a queue [11:35:05] https://github.com/gesinn-it-pub/mwbot/blob/master/src/index.js#L254-L260 [11:35:09] Yeah, I merged a patch that prints the full exception object. It's almost done, assuming no selenium surprises [11:35:21] :) [11:35:37] and I had a quick look at https://logstash-beta.wmcloud.org/ [11:35:49] but it does not seem to have MediaWiki logs anymore [11:36:04] Alright patch just got merged. Lemme see if I can manually trigger the daily run [11:36:41] I looked in beta logstash last time, but it wasn't useful. I didn't find anything for the failing requests, so maybe it is indeed the case that they aren't reaching MW (or beta logstash is broken) [11:37:01] I assume the later (logging stack is broken) [11:37:15] (seems happier now) [11:38:41] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.45.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T396375#11077600 (10hashar) I have fixed up the tasks that failed to run during the night and the trains has started its journey! ` Checking version on https:... [11:41:49] I triggered a rebuild, let's see: https://integration.wikimedia.org/ci/job/selenium-daily-beta-CampaignEvents/500/console [11:42:56] Aaaaaaaaaaaaaaaaaaaaaaah "308 Permanent Redirect", of course! Beta was moved to wmcloud but the build parameters still have wmflabs [11:43:28] jjb patch coming [11:54:45] (03PS1) 10Daimona Eaytoy: jjb: Update beta domain for selenium-daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1177975 (https://phabricator.wikimedia.org/T393428) [11:56:12] (03PS2) 10Daimona Eaytoy: jjb: Update beta domain for selenium-daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1177975 (https://phabricator.wikimedia.org/T393428) [11:56:47] Daimona: mouahah I did see that in logstash! [11:57:04] I was like "what is a 308 already"? And then decided to not pursue yet another trail [11:57:34] Bah I did not expect that [11:57:39] maybe mwbot request should be made to follow redirects [11:57:51] anyway if it now displays the response, I guess it is good [11:58:51] Yeah, I think I proposed that somewhere, but with the conversation happening in 3 tasks at once I already forgot where. [11:59:08] At any rate, once the jjb patch is merged we should be good, and ready to find the next failure mode [12:01:40] had we had a 301, I guess I would understood [12:01:46] 308 looks roughly similar [12:01:47] anywya [12:03:27] (03CR) 10Hashar: [C:03+2] jjb: Update beta domain for selenium-daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1177975 (https://phabricator.wikimedia.org/T393428) (owner: 10Daimona Eaytoy) [12:03:45] Daimona: I have updated the selenium-daily-beta-* jobs [12:04:53] (03Merged) 10jenkins-bot: jjb: Update beta domain for selenium-daily tests [integration/config] - 10https://gerrit.wikimedia.org/r/1177975 (https://phabricator.wikimedia.org/T393428) (owner: 10Daimona Eaytoy) [12:05:34] (03PS1) 10Cyndywikime: Add MetricsPlatform as a CI and Phan dependency for GE [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 [12:06:13] Nice, thank you! And I made T401673 about following redirects by default (didn't make an upstream issue because I don't fully appreciate the implications) [12:06:13] T401673: Consider following redirects in mwbot - https://phabricator.wikimedia.org/T401673 [12:07:24] (03PS1) 10Hashar: jjb: update beta domain for MediaWiki Selenium daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177979 (https://phabricator.wikimedia.org/T398310) [12:07:51] Daimona: you have missed the job that runs for mediawiki/core https://gerrit.wikimedia.org/r/1177979 [12:08:06] Oh yeah, whoops. [12:08:23] if you can +1 one that one, I ill merge it [12:08:27] I have updated the job [12:08:34] (03CR) 10Daimona Eaytoy: [C:03+1] jjb: update beta domain for MediaWiki Selenium daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177979 (https://phabricator.wikimedia.org/T398310) (owner: 10Hashar) [12:09:06] I have triggered a rebuild, now the API requests succeed and we're back to another random selenium failure (element not found) [12:09:08] Progress! [12:10:37] (03CR) 10Hashar: [C:03+2] jjb: update beta domain for MediaWiki Selenium daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177979 (https://phabricator.wikimedia.org/T398310) (owner: 10Hashar) [12:11:19] there is that story about computer science having two hard things: [12:11:19] * cache invalidation [12:11:19] * naming things [12:11:19] * off-by one errors [12:11:40] I'll add renaming things, though it is a variant of cache invalidation [12:12:04] ("**two** hard things" is the joke) [12:12:08] (03Merged) 10jenkins-bot: jjb: update beta domain for MediaWiki Selenium daily [integration/config] - 10https://gerrit.wikimedia.org/r/1177979 (https://phabricator.wikimedia.org/T398310) (owner: 10Hashar) [12:12:27] I am gonna grab a coffee [12:17:13] (03CR) 10Cyndywikime: "This patch is ready for review" [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (owner: 10Cyndywikime) [12:18:56] hashar: Re. ^^^, be careful β€” MP/EventLogging dependencies in ultra-tested gate code like this has blown up CI multiple times in the past few months due to recursion of dependencies. :-( Maybe don't land until the non-recursive mode is landed? [12:21:53] (03CR) 10Urbanecm: [C:03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (owner: 10Cyndywikime) [13:09:01] (03PS2) 10Hashar: Zuul: add MetricsPlatform as dependency of GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [13:09:15] (03CR) 10Hashar: [C:03+2] Zuul: add MetricsPlatform as dependency of GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [13:10:37] (03Merged) 10jenkins-bot: Zuul: add MetricsPlatform as dependency of GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [13:13:48] hashar: Ahem. :-P [13:16:13] (03CR) 10Hashar: [C:03+2] "Deployed and I have send a `recheck` on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1176500" [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [14:26:13] hm.. it seems I'm unable to use docker-pkg to update changelogs. I thought this command worked offline, but either way, I'm online and it's connection to something (unspecified) is refused. [14:26:23] https://phabricator.wikimedia.org/P81133 [14:28:13] wow… that’s even more than less than unhelpful [14:28:23] guess: could it be trying (and failing) to connect to your local docker daemon? [14:28:41] why yes, Indeed I turned that off yesterday. Great. [14:28:44] let me try again [14:28:54] ha!. Thanks :D [14:28:58] \o/ :D [14:29:07] works offline, for some definitions of offline [14:29:07] [...]/docker/transport/unixconn.py being in the stack trace supports that theory [14:29:34] (03PS4) 10Krinkle: mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) [14:29:34] (03PS4) 10Krinkle: quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 [14:31:05] (03CR) 10CI reject: [V:04-1] quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 (owner: 10Krinkle) [14:31:11] (03CR) 10CI reject: [V:04-1] mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [14:46:05] (03merge) 10dancy: build_image_incr.py: Add label referencing parent image [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/202 [14:52:26] (03update) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [14:52:31] (03update) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [14:59:47] 06Release-Engineering-Team, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 05MW-1.45-release: Remove SwaggerUI from MW-1.45 release - https://phabricator.wikimedia.org/T397473#11078433 (10Atieno) a:03Atieno [15:21:10] (03CR) 10Urbanecm: [C:03+1] "Thank you! I rebased it on top of I4cd576c6f4757312f76724305140f12194d8dc3a, and phan now does not complain." [integration/config] - 10https://gerrit.wikimedia.org/r/1177978 (https://phabricator.wikimedia.org/T401308) (owner: 10Cyndywikime) [15:23:39] (03update) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [15:26:17] (03update) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [15:38:51] (03open) 10dancy: patches.py: Set umask for all commands [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/968 (https://phabricator.wikimedia.org/T401672) [15:38:55] (03update) 10dancy: patches.py: Set umask for all commands [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/968 (https://phabricator.wikimedia.org/T401672) [15:40:09] !log Unblock 89.12.0.0/14 (T401670) [15:40:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:00] (03merge) 10dancy: patches.py: Set umask for all commands [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/968 (https://phabricator.wikimedia.org/T401672) [15:42:59] (03open) 10dancy: Release 4.199.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/969 [15:45:15] (03merge) 10dancy: Release 4.199.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/969 [15:48:01] 10WikimediaDebug, 07Security: Seeking Advice and Assistance: Upgrading Debian Buster XHGui Instances - https://phabricator.wikimedia.org/T340713#11078807 (10Aklapper) [15:53:04] 10Beta-Cluster-Infrastructure, 07Wikimedia-production-error: DBQueryError "The MariaDB server is running with the --read-only option" fails MainStash in Beta Cluster - https://phabricator.wikimedia.org/T401227#11078832 (10bd808) `lang=shell-session $ ssh deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cl... [15:56:17] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#11078869 (10Aklapper) Side note: I looked into "risk.summary" and "risk.impacted" out of curiosity (plus was trying t... [15:57:06] !log Upgrading scap to 4.199.0 in beta cluster [15:57:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:07:38] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11078959 (10Aklapper) Thanks for that quick reply! It displays "No results found" for me (wh... [16:09:11] (03open) 10dancy: main.py: Don't report Restarting php-fpm if no hosts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/970 [16:09:14] (03update) 10dancy: main.py: Don't report Restarting php-fpm if no hosts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/970 [16:09:47] (03update) 10dancy: main.py: Don't report Restarting php-fpm if no hosts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/970 [16:12:02] (03merge) 10dancy: main.py: Don't report Restarting php-fpm if no hosts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/970 [16:13:00] (03approved) 10dduvall: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 (owner: 10dancy) [16:14:53] (03update) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [16:16:12] (03merge) 10dancy: kubernetes.py: prune_local_images: don't remove parent images [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/967 [16:18:52] (03open) 10dancy: main.py: Don't mention unused sync--k8s stages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/971 [16:18:56] (03update) 10dancy: main.py: Don't mention unused sync--k8s stages [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/971 [16:24:05] (03open) 10dancy: Release 4.200.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/972 [16:24:54] (03merge) 10dancy: Release 4.200.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/972 [16:25:02] (03PS5) 10Krinkle: mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) [16:25:06] (03PS5) 10Krinkle: quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 [16:25:42] (03CR) 10CI reject: [V:04-1] mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [16:25:46] (03PS1) 10Krinkle: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 [16:26:14] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11079080 (10colewhite) 05Resolvedβ†’03Open Looks like it got changed back. [16:26:25] (03PS6) 10Krinkle: mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) [16:26:33] (03PS6) 10Krinkle: quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 [16:26:33] (03PS2) 10Krinkle: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 [16:27:07] !log Upgrading scap to 4.200.0 in beta cluster [16:27:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:29:39] 10Phabricator, 06Language and Product Localization, 10Observability-Logging: Generic Logstash Visualization "Events over time" got overwritten with "ContentTranslation" - https://phabricator.wikimedia.org/T401579#11079101 (10colewhite) 05Openβ†’03Resolved Restored original visualization from backup. [16:32:58] (03CR) 10Krinkle: [C:03+2] mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [16:33:09] (03CR) 10Krinkle: [C:03+2] quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 (owner: 10Krinkle) [16:34:41] (03Merged) 10jenkins-bot: mwext-phpunit-coverage: Switch from legacy phpunit.php to vendor/bin/phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/1176721 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [16:34:53] (03Merged) 10jenkins-bot: quibble-bullseye-php81-coverage: Remove unused mwservice-phpunit-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/1176722 (owner: 10Krinkle) [16:37:24] 10Phabricator (Upstream), 07Upstream: Herald rules based on age - https://phabricator.wikimedia.org/T365366#11079182 (10Aklapper) [16:59:28] 10Scap, 06serviceops: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721 (10Scott_French) 03NEW [16:59:34] (03update) 10dduvall: docs: Refactor README to focus on common usage patterns [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/140 [16:59:37] (03update) 10dduvall: docs: Refactor README to focus on common usage patterns [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/140 [17:00:18] (03update) 10dduvall: docs: Refactor README to focus on common usage patterns [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/140 [17:00:32] (03update) 10dduvall: docs: Refactor README to focus on common usage patterns [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/140 [17:04:53] (03merge) 10dduvall: docs: Refactor README to focus on common usage patterns [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/140 [17:20:06] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 13Patch-For-Review, 07Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11079378 (10Daimona) I made patches for all extensions, as well as development tools (plus a couple cleanup patches for things... [17:24:21] (03update) 10addshore: Draft: Rewrite (maybe version 1)? [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/628 (https://phabricator.wikimedia.org/T292906) [17:33:51] 10Continuous-Integration-Infrastructure: Add Python 3.13 to Wikimedia CI - https://phabricator.wikimedia.org/T396273#11079478 (10Xqt) @Hashar: is this ready to use with Pywikibot tests for CI? [17:34:37] (03update) 10addshore: Draft: Reorganize code [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/628 (https://phabricator.wikimedia.org/T292906) [17:49:41] (03approved) 10addshore: Draft: Reorganize code [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/628 (https://phabricator.wikimedia.org/T292906) [17:49:58] (03update) 10addshore: Reorganize code [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/628 (https://phabricator.wikimedia.org/T292906) [17:50:08] (03merge) 10addshore: Reorganize code [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/628 (https://phabricator.wikimedia.org/T292906) [17:50:35] (03update) 10addshore: Update default ELASTICSEARCH_IMAGE [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/631 (owner: 10sihe) [17:58:02] (03update) 10dancy: patches.py: Update PatchBot-related code [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/919 (https://phabricator.wikimedia.org/T295925) (owner: 10jnuche) [18:11:37] 10Continuous-Integration-Infrastructure: Add Python 3.13 to Wikimedia CI - https://phabricator.wikimedia.org/T396273#11079595 (10hashar) @Xqt https://gerrit.wikimedia.org/r/c/integration/config/+/1166761 added python 3.13 but I did not rebuild the child image `releng/tox-v3-pywikibot` :/ [18:19:56] (03PS1) 10Hashar: dockerfiles: add Python 3.13 to tox-v3-pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/1178056 (https://phabricator.wikimedia.org/T396273) [18:19:58] (03PS1) 10Hashar: jjb: add Python 3.13 to pywikibot/core [integration/config] - 10https://gerrit.wikimedia.org/r/1178057 (https://phabricator.wikimedia.org/T396273) [18:20:00] (03PS1) 10Hashar: Zuul: [pywikibot/core] Add Python 3.13 fasttest [integration/config] - 10https://gerrit.wikimedia.org/r/1178058 (https://phabricator.wikimedia.org/T396273) [18:21:33] (03update) 10brennen: spiderpig log view: migrate to sortable/filterable vuetify data table component [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/882 (https://phabricator.wikimedia.org/T397091) [18:22:45] (03CR) 10Hashar: [C:03+2] dockerfiles: add Python 3.13 to tox-v3-pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/1178056 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:23:45] (03CR) 10Hashar: "This would switch the jobs to the image built from I10c87e19463300dc3ccbd6ad5320cc79bff98329 and create `pywikibot-core-tox-fasttest-py313" [integration/config] - 10https://gerrit.wikimedia.org/r/1178057 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:24:20] (03Merged) 10jenkins-bot: dockerfiles: add Python 3.13 to tox-v3-pywikibot [integration/config] - 10https://gerrit.wikimedia.org/r/1178056 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:25:02] oh joy stuff not built/deployed [18:29:29] (03CR) 10Hashar: [C:03+1] "I have built the image:" [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (owner: 10Krinkle) [18:29:55] hashar: sorry, got distracted. I have the tab open on `fab deploy_docker` but didn't press "y" [18:30:00] (03CR) 10Hashar: [C:03+2] "I have built the image:" [integration/config] - 10https://gerrit.wikimedia.org/r/1178056 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:33:50] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Add Python 3.13 to Wikimedia CI - https://phabricator.wikimedia.org/T396273#11079695 (10hashar) I have built the image: ` Successfully published image docker-registry.wikimedia.org/releng/tox-v3-pywikibot:1.1-s2 ` All the pythons installed by pyvenv:... [18:36:32] 10Continuous-Integration-Config, 10docker-pkg: Hostnames displayed by docker-pkg are not the canonical ones - https://phabricator.wikimedia.org/T401733 (10hashar) 03NEW [18:39:26] 10Continuous-Integration-Config, 10docker-pkg: Hostnames displayed by docker-pkg are not the canonical ones - https://phabricator.wikimedia.org/T401733#11079728 (10hashar) [18:43:37] (03CR) 10Xqt: [C:03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/1178057 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:44:39] (03CR) 10Xqt: [C:03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/1178058 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:45:16] (03merge) 10addshore: Update default ELASTICSEARCH_IMAGE [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/631 (owner: 10sihe) [18:46:06] (03CR) 10Hashar: [C:03+2] "Thank you for the +1. I went to update all jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/1178057 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:46:13] (03CR) 10Hashar: [C:03+2] Zuul: [pywikibot/core] Add Python 3.13 fasttest [integration/config] - 10https://gerrit.wikimedia.org/r/1178058 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:47:32] (03Merged) 10jenkins-bot: jjb: add Python 3.13 to pywikibot/core [integration/config] - 10https://gerrit.wikimedia.org/r/1178057 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:47:42] (03Merged) 10jenkins-bot: Zuul: [pywikibot/core] Add Python 3.13 fasttest [integration/config] - 10https://gerrit.wikimedia.org/r/1178058 (https://phabricator.wikimedia.org/T396273) (owner: 10Hashar) [18:48:42] 10Scap, 06serviceops: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11079780 (10Scott_French) 05Openβ†’03In progress p:05Triageβ†’03Medium [18:49:02] (03open) 10swfrench: Draft: make-container-image: php version metadata label [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/203 (https://phabricator.wikimedia.org/T401721) [18:49:16] !log Reloaded Zuul for [pywikibot/core] Add Python 3.13 fasttest [integration/config] - https://gerrit.wikimedia.org/r/1178058 - T396273 [18:49:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:19] T396273: Add Python 3.13 to Wikimedia CI - https://phabricator.wikimedia.org/T396273 [18:49:33] (03open) 10swfrench: Draft: kubernetes: set php.version based on image labels [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/973 (https://phabricator.wikimedia.org/T401721) [19:22:08] 10Scap, 06serviceops, 13Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11079929 (10Scott_French) I should mention here, there's one notable downside to using a metadata label like this: If for some reason the selected image no longer exist... [19:36:35] 10Continuous-Integration-Config, 10docker-pkg, 13Patch-For-Review: Hostnames displayed by docker-pkg are not the canonical ones - https://phabricator.wikimedia.org/T401733#11079969 (10hashar) a:03hashar [19:46:25] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11080011 (10Dzahn) Getting back to check the list of bind mound points from Corvus' recommendation. Just looking at "do the mount points exist" from pup... [19:51:40] 10Continuous-Integration-Infrastructure: Add Python 3.13 to Wikimedia CI - https://phabricator.wikimedia.org/T396273#11080060 (10Xqt) Thank you @Hashar. Most test works with Python 3.13 except of plain {name}-tox job. I get a missing interpreter error when pre-commit is running with lint-py313. I guess the dock... [20:04:56] (03CR) 10Krinkle: "Even though this updates mwcore-/mwskin-/mwext- jobs, the actual change only affects mwext jobs, and specifically only the postmerge mwext" [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (owner: 10Krinkle) [20:05:52] (03CR) 10Krinkle: [C:04-1] "I've reverted the update." [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (owner: 10Krinkle) [20:09:23] 10Scap, 06serviceops, 13Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11080142 (10dancy) >>! In T401721#11079929, @Scott_French wrote: > I should mention here, there's one notable downside to using a metadata label like this: If for some... [20:16:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11080184 (10bd808) >>! In T395938#11080011, @Dzahn wrote: >> Web: >> * Listens on port 9000 by default > > Where exactly should we... [20:17:18] (03CR) 10Krinkle: [C:04-1] "Before:" [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (owner: 10Krinkle) [20:18:40] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11080194 (10Dzahn) Thank you for this comment, bd808. This is similar to what I was thinking about. Great to get confirmation for i... [20:18:50] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11080196 (10bd808) >>! In T395938#11080011, @Dzahn wrote: >> Nodepool Launcher: >> * Bind mount a directory with nodepool.yaml into... [20:20:51] (03PS1) 10Krinkle: quibble-bullseye-php81-coverage: Fix bad `--path-to-mw` arg [integration/config] - 10https://gerrit.wikimedia.org/r/1178071 (https://phabricator.wikimedia.org/T395470) [20:21:18] (03PS3) 10Krinkle: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) [20:21:25] (03PS4) 10Krinkle: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) [20:23:46] (03update) 10brennen: spiderpig log view: migrate to sortable/filterable vuetify data table component [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/882 (https://phabricator.wikimedia.org/T397091) [20:27:05] (03update) 10brennen: spiderpig log view: migrate to sortable/filterable vuetify data table component [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/882 (https://phabricator.wikimedia.org/T397091) [20:33:21] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07OKR-Work: Modify wmf/next branching process to avoid leaving an incomplete branch on failure - https://phabricator.wikimedia.org/T401749 (10bd808) 03NEW [20:49:28] (03PS5) 10Krinkle: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) [20:49:36] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11080325 (10thcipriani) >>! In T395938#11080183, @bd808 wrote: >>>! In T395938#11080011, @Dzahn wrote: >>> Web: >>> * Listens on po... [20:49:40] (03CR) 10Krinkle: "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [20:50:19] (03CR) 10Krinkle: [C:03+2] quibble-bullseye-php81-coverage: Fix bad `--path-to-mw` arg [integration/config] - 10https://gerrit.wikimedia.org/r/1178071 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [20:51:47] (03Merged) 10jenkins-bot: quibble-bullseye-php81-coverage: Fix bad `--path-to-mw` arg [integration/config] - 10https://gerrit.wikimedia.org/r/1178071 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [20:52:05] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1178071 [20:52:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:59:04] 10GitLab (CI & Job Runners): Uploading artifacts does not work anymore - https://phabricator.wikimedia.org/T401535#11080331 (10bd808) [21:00:29] (03CR) 10Krinkle: [C:03+2] "Yay." [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [21:01:34] (03CR) 10Krinkle: [C:03+2] "Updated `mwcore*coverage*`, `mwskin*coverage*`, `mwext*coverage*` and `*codehealth*` (only mwext are actually affected, rest no-ops)" [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [21:02:05] (03Merged) 10jenkins-bot: jjb: Update mwext-*-coverage* jobs to quibble-bullseye-php81-coverage:1.14.2-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/1178030 (https://phabricator.wikimedia.org/T395470) (owner: 10Krinkle) [21:14:32] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10MW-1.45-notes (1.45.0-wmf.15; 2025-08-19), 13Patch-For-Review, 07Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11080439 (10Krinkle) [21:54:27] (03update) 10dancy: spiderpig log view: migrate to sortable/filterable vuetify data table component [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/882 (https://phabricator.wikimedia.org/T397091) (owner: 10brennen) [21:54:37] (03approved) 10dancy: spiderpig log view: migrate to sortable/filterable vuetify data table component [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/882 (https://phabricator.wikimedia.org/T397091) (owner: 10brennen) [22:06:37] 10GitLab (CI & Job Runners): Uploading artifacts does not work anymore - https://phabricator.wikimedia.org/T401535#11080572 (10dancy) @Jelto Bringing this to your attention since it may have involved throttling of traffic from Digital Ocean. [22:16:18] (03PS1) 10Catrope: jjb: Set CODEX_MATOMO_URL for Codex doc builds [integration/config] - 10https://gerrit.wikimedia.org/r/1178089 (https://phabricator.wikimedia.org/T368970) [22:16:47] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 06cloud-services-team, 10Striker: Automatically approve GitLab accounts created by Striker integration - https://phabricator.wikimedia.org/T344667#11080590 (10bd808) 05Openβ†’03Resolved a:03bd808 >>! In T344667#9431444, @bd808 wrote: > T... [22:27:37] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#11080625 (10Aklapper) p:05Triageβ†’03Low a:03Aklapper [22:30:14] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T392464#11080631 (10Dzahn) [22:32:12] 10Gerrit (Gerrit 3.12): Upgrade Gerrit hosts from Bullseye to Bookworm - https://phabricator.wikimedia.org/T392464#11080639 (10Dzahn) This is mostly a duplicate of T372804 where we already did the things needed to have a gerrit on bookworm. The difference being just that we started gerrit2003 only and this shou... [23:02:48] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07OKR-Work: Why does build-and-push-container-images take 11 minutes? - https://phabricator.wikimedia.org/T401533#11080751 (10bd808) @dancy, @dduvall, and I spent a bit of time talking about this in a Pretrain team meeting today. Some ideas: * Check on the prog... [23:18:00] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07OKR-Work: Why does build-and-push-container-images take 11 minutes? - https://phabricator.wikimedia.org/T401533#11080772 (10Dzahn) >>! In T401533#11080749, @bd808 wrote: > * Bring buildkitd to deployment server so we can use bake which may have better parallel... [23:21:02] (03CR) 10Krinkle: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1178089 (https://phabricator.wikimedia.org/T368970) (owner: 10Catrope) [23:22:34] (03Merged) 10jenkins-bot: jjb: Set CODEX_MATOMO_URL for Codex doc builds [integration/config] - 10https://gerrit.wikimedia.org/r/1178089 (https://phabricator.wikimedia.org/T368970) (owner: 10Catrope) [23:22:40] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Documentation, 07OKR-Work: Work on narrative and FAQ about how things will be different when Pretrain rolls out - https://phabricator.wikimedia.org/T401763 (10bd808) 03NEW