[01:03:57] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T386216#10660131 (10thcipriani) 05Openβ†’03Resolved [05:40:02] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491#10660414 (10A_smart_kitten) Adding #wikimedia-production-error, as per that tag’s des... [05:55:12] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 07Wikimedia-production-error: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585 (10A_smart_kitten) 03NEW [07:23:40] 10Beta-Cluster-Infrastructure, 10gadget-Cat-a-lot: Temporary interface admin right to beta commons for updating the Cat-a-lot gadget - https://phabricator.wikimedia.org/T387080#10660598 (10A_smart_kitten) 05Openβ†’03Resolved a:03TheresNoTime Closing as resolved, as admin/intadmin rights on beta-commons... [07:31:15] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10660607 (10daniel) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1099189 * **Summary**: ** Stores EditResu... [07:32:26] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10660622 (10daniel) Pinging @Ladsgroup for awareness --^ [08:17:40] Platonides andre: around to banhammer a spammer on wikitech? [08:17:49] eh? [08:18:12] well, no new messages for an hour [08:18:28] but let me do that [08:18:58] oh yes, I forget we have both wikitech, and wikitech-l [08:19:16] done [08:19:28] if you mean the mailing list :P [08:19:44] yes i did [08:19:52] thanks [08:21:57] I dud just check wikitech, the wiki, just encase as well :) [08:51:20] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10660803 (10Ladsgroup) Main stash is `x2` but by the time the train will reach production, we will have actually switched to to ms1. ms2. ms3 clusters... [08:51:30] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10660806 (10Ladsgroup) cc @Marostegui [09:02:31] 10WikimediaDebug, 06SRE, 06Traffic, 07Developer Productivity, 13Patch-For-Review: Let X-Analytics response header pass through with WikimediaDebug - https://phabricator.wikimedia.org/T305794#10660809 (10Krinkle) [09:39:29] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 05Account-Vanishing, 07Wikimedia-production-error: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10660911 (10A_smart_kitten) [09:59:37] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10661063 (10Gehel) [10:13:20] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10661185 (10Krinkle) >>! In T386217#10660606, @daniel wrote: > * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1099189 > * **Summary*... [10:24:29] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10661226 (10Gehel) [10:34:16] 06Project-Admins: Create project tag for MediaWiki-extensions-BoilerPlate - https://phabricator.wikimedia.org/T389568#10661271 (10Peachey88) 05Openβ†’03Resolved a:03Peachey88 #MediaWiki-extensions-BoilerPlate has been created :). [10:43:42] 10Continuous-Integration-Config, 10MediaWiki-extensions-BoilerPlate, 10MediaWiki-extensions-Examples: Separate BoilerPlate extension from extension/examples - https://phabricator.wikimedia.org/T94279#10661298 (10Aklapper) [10:51:16] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10661317 (10daniel) @Krinkle thanks for the analysis... I honestly don't know what the percentages are. My assumption was that it would maybe double t... [11:38:48] 06Release-Engineering-Team, 10MW-on-K8s, 06serviceops: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10661423 (10Clement_Goubert) I think the second option is sound and would greatly simplify adding new images. If we go with thi... [12:37:26] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for cwylo - https://phabricator.wikimedia.org/T389519#10661653 (10Aklapper) 05Openβ†’03Declined [12:53:51] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Consider disabling personal access token forced expiration - https://phabricator.wikimedia.org/T385930#10661714 (10LSobanski) Same thing for gitlab-exporter, we got a 60 day expiry not... [12:54:00] 10Gerrit: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10661717 (10hashar) Using the [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1062376/1/includes/specialpage/LoginSignupSpecialPage.php#79... [13:05:27] 10Gerrit: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10661759 (10Paladox) I filed https://github.com/highlightjs/highlight.js/issues/3995 but was closed without a fix. [13:12:44] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 05Account-Vanishing, 07Wikimedia-production-error: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10661811 (10Dreamy_Jazz) [13:18:51] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 05Account-Vanishing, 07Wikimedia-production-error: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10661830 (10Dreamy_Jazz) Seems to be an issue with queuing jobs specif... [13:38:28] 10Gerrit: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10661884 (10hashar) a:03hashar Using https://github.com/highlightjs/highlight.js, the build is `npm run build`. The tests for PHP are under `... [13:42:29] 10Beta-Cluster-Infrastructure, 10WMF-JobQueue, 05Account-Vanishing, 10Trust and Safety Product Sprint (Sprint Lemon Meringue (March 3 - 21)), 07Wikimedia-production-error: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on ... - https://phabricator.wikimedia.org/T389585#10661894 [14:01:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 10WMF-JobQueue, and 4 others: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10661994 (10A_smart_kitten) (given that this... [14:04:44] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10662015 (10hashar) The issue is the function can contains block comments (`/* */`) but can not contain line comments (`// `) whi... [14:13:50] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10662046 (10hashar) 05Openβ†’03In progress Fix proposed at https://github.com/highlightjs/highlight.js/pull/4230 [14:26:34] (03PS1) 10STran: Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1130132 (https://phabricator.wikimedia.org/T386435) [14:32:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure): Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10662131 (10Daimona) I think we got it! https://integration.wikimedia.org/ci... [14:35:11] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10662147 (10hashar) I have amended to also support comments starting with `#` which would fix upstream https://github.com/highlig... [14:54:40] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 10WMF-JobQueue, and 4 others: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10662252 (10Dreamy_Jazz) [14:56:06] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 10WMF-JobQueue, and 4 others: "JobQueueError: Could not enqueue jobs" when trying to request a global vanish on Beta Cluster - https://phabricator.wikimedia.org/T389585#10662256 (10Dreamy_Jazz) Suggested QA steps... [14:57:58] (03CR) 10Dreamy Jazz: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1130132 (https://phabricator.wikimedia.org/T386435) (owner: 10STran) [15:01:16] 06Project-Admins: Create project tag for MediaWiki-extensions-BoilerPlate - https://phabricator.wikimedia.org/T389568#10662308 (10apaskulin) Thank you! [15:07:38] (03CR) 10Kosta Harlan: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1130132 (https://phabricator.wikimedia.org/T386435) (owner: 10STran) [16:09:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416#10662713 (10Daimona) Seen a new one today:... [16:17:12] 10Phabricator (2025-03-18), 10Release-Engineering-Team (Doing 😎): 'Unhandled Exception ("Error")' displayed when attempting to filter the Phabricator activity feed by IDs/PHIDs - https://phabricator.wikimedia.org/T387479#10662762 (10Aklapper) Hmm, after removing we now get errors in our error log when some... [16:20:54] (03open) 10dancy: spiderpig: whoami route debugging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/696 [16:20:57] (03update) 10dancy: spiderpig: whoami route debugging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/696 [16:23:20] (03merge) 10dancy: spiderpig: whoami route debugging [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/696 [16:35:31] Sigh. Still PHP 7.4 code in prod, so can't really drop the CI hammer yet. :-( [16:35:44] I was so very much hoping to do that today. [16:37:54] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#10662881 (10sbassett) So I made a little `post-commit` hook based on... [16:42:09] James_F: Where are we up to in practice? [16:42:27] Reedy: Everything except mw-script, I believe. [16:42:50] I'm sure that's easy... [16:42:50] And one of the dumps machines, as it's actively running the Wikidata dump and we didn't want to disrupt long-running processes. [16:42:54] * Reedy coughs [16:43:16] Oh, yeah, no blame at all on the excellent s.wfrench-wmf and colleagues. [16:46:17] move fast and break things [16:47:19] I mean, the likelihood that someone will write a PHP 7.4-incompatible change to a maintenance script, and land it, and try to use it in production before we finish the job (next week?) is slight, but we have CI exactly to stop this kind of foul-up. [17:05:28] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#10663041 (10Tgr) Maybe safer to go in the other direction and add a... [17:06:45] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#10663050 (10Tgr) (git-review also has a [[https://docs.opendev.org/o... [17:29:32] 06Release-Engineering-Team, 10MW-on-K8s, 06serviceops: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10663237 (10Scott_French) Thanks, @Clement_Goubert. So, I realized in retrospect that adding a new image kind of course //alread... [17:42:50] (03open) 10dancy: Release 4.143.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/697 [17:45:01] (03merge) 10dancy: Release 4.143.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/697 [18:13:32] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#10663532 (10sbassett) >>! In T388247#10663041, @Tgr wrote: > Maybe s... [18:45:03] maintenance-disconnect-full-disks build 686084 integration-agent-docker-1042 (/: 26%, /srv: 96%, /var/lib/docker: 34%): OFFLINE due to disk space [18:50:03] maintenance-disconnect-full-disks build 686085 integration-agent-docker-1042 (/: 26%, /srv: 46%, /var/lib/docker: 34%): RECOVERY disk space OK [18:55:13] (03open) 10dancy: Revert "spiderpig: whoami route debugging" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/698 [18:55:15] (03update) 10dancy: Revert "spiderpig: whoami route debugging" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/698 [18:56:01] (03open) 10dancy: spiderpig: Add Cache-Control: no-cache to api responses [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/699 [18:56:05] (03update) 10dancy: spiderpig: Add Cache-Control: no-cache to api responses [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/699 [18:56:55] (03update) 10dancy: spiderpig: Add Cache-Control: no-cache to api responses [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/699 [18:57:33] (03merge) 10dancy: Revert "spiderpig: whoami route debugging" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/698 [18:58:19] (03update) 10dancy: spiderpig: Add Cache-Control: no-cache to api responses [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/699 [19:00:29] (03merge) 10dancy: spiderpig: Add Cache-Control: no-cache to api responses [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/699 [19:02:21] (03open) 10dancy: Release 4.143.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/700 [19:04:16] (03merge) 10dancy: Release 4.143.2 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/700 [19:31:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10663818 (10mszabo) I'm not sure if this is a direct c... [19:55:02] 10GitLab-Application-Security-Pipeline, 06Security-Team: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468#10663853 (10sbassett) [20:01:08] 10GitLab-Application-Security-Pipeline, 06Security-Team: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468#10663874 (10sbassett) [20:06:08] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T386217#10663900 (10aaron) > Also per T259103, although it seems like a misunderstanding in the product design, since 2020 this logic re... [20:59:54] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664125 (10hashar) >>! In T389536#10662131, @Daimona... [21:03:00] (03CR) 10Hashar: Fix logging exception when using --resolve-requires (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1129251 (owner: 10Hashar) [21:04:12] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Upstream src/herald/SetSubtypeHeraldAction.php - https://phabricator.wikimedia.org/T389686 (10Aklapper) 03NEW [21:04:34] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Upstream src/herald/SetSubtypeHeraldAction.php - https://phabricator.wikimedia.org/T389686#10664137 (10Aklapper) Should rename "Set type" to "Set subtype" though because confusion is confusion [21:17:15] (03CR) 10Hashar: [C:03+2] Avoid success cache key data collisions using null separator [integration/quibble] - 10https://gerrit.wikimedia.org/r/1129916 (owner: 10Dduvall) [21:18:37] (03CR) 10Hashar: "Note to myself: once released and deployed, that invalidates the old successes :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1129916 (owner: 10Dduvall) [21:30:41] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664206 (10Daimona) [22:08:55] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664285 (10Daimona) Filed https://github.com/wikimedi... [22:14:15] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664301 (10Novem_Linguae) @zeljkofilipin has been ver... [22:22:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664344 (10mszabo) Having rebased https://gerrit.wiki... [22:34:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10664398 (10Daimona) >>! In T389536#10664344, @mszabo... [22:38:06] 10Phabricator (2025-03-18), 10Release-Engineering-Team (Doing 😎): Unable to preview MP4 video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222#10664404 (10Krinkle) Does viewing MP4 files work now? Testing... {F58892403} https://phabricator.wikimedia.org/F58892403... [22:38:15] 10Phabricator (2025-03-18), 10Release-Engineering-Team (Doing 😎): Unable to preview MP4 video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222#10664405 (10Krinkle) 05Resolvedβ†’03Open [22:44:22] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10664412 (10dancy) 05In progressβ†’03Resolved a:03dancy [23:40:25] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaViewer, 06MediaWiki-Platform-Team, 10MW-1.44-notes (1.44.0-wmf.20; 2025-03-11), and 2 others: MediaViewer broken in wmf.20 due to LESS change: Less_Exception_Compiler: error evaluating function `floo... - https://phabricator.wikimedia.org/T388526#10664497