[07:15:00] 10Phabricator (Upstream), 10Upstream: Phabricator search dropdown results should open in new tab on middle-click - https://phabricator.wikimedia.org/T76812 (10Tgr) [08:32:48] 10Continuous-Integration-Infrastructure, 10Jenkins: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10hashar) [08:32:50] 10Continuous-Integration-Infrastructure, 10Jenkins: HTTP ERROR 403 No valid crumb was included in the request - https://phabricator.wikimedia.org/T327988 (10hashar) [08:34:34] 10Continuous-Integration-Infrastructure, 10Jenkins: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10hashar) [08:36:13] 10Continuous-Integration-Infrastructure, 10Jenkins: HTTP ERROR 403 No valid crumb was included in the request - https://phabricator.wikimedia.org/T327988 (10hashar) I have marked this as a duplicate of T326970 which was filed a few days ago. The crumb is tied to the client IP / `X-Forwarded-For` header which... [09:35:44] thcipriani: I'll sneak in now and do a small MediaWiki config deployment, unless anyone here has an objection. We should have used the EU morning window, this is something my team wants to monitor during our work day. [10:28:21] (03PS1) 10Hashar: dockerfiles: add services to Quibble coverage [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) [10:29:26] (03CR) 10CI reject: [V: 04-1] dockerfiles: add services to Quibble coverage [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [10:32:01] (03PS2) 10Hashar: dockerfiles: add services to Quibble coverage [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) [10:43:35] (03PS4) 10Hashar: Zuul: [mediawiki/services/parsoid] Enable code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/882585 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [10:43:37] (03PS1) 10Hashar: jjb: add coverage jobs for MediaWiki services [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) [10:45:41] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Lucas_Werkmeister_WMDE) Grunt 1.6.0 also nominally requires Node 16 now (though CI on [this Gerrit change](https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/88... [10:56:25] (03CR) 10Hashar: [C: 03+2] "This adds support for mediawiki/services/* to the coverage image docker-registry.wikimedia.org/releng/quibble-buster-php74" [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [10:57:48] (03Merged) 10jenkins-bot: dockerfiles: add services to Quibble coverage [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [10:59:22] !log Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/884835/ [10:59:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:03:08] (03CR) 10Hashar: "I have created the two jenkins jobs for services:" [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [11:41:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Cleanup and attach volumes for gitlab-runners WMCS project - https://phabricator.wikimedia.org/T328283 (10Jelto) [12:53:08] Going in to revert the Kartographer configuration above. [12:55:09] Project beta-scap-sync-world build #88409: 04FAILURE in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/88409/ [12:55:36] Alarming scap backport failure: > Fatal error: Cannot declare class Wikimedia\MWConfig\ServiceConfig, because the name is already in use in /srv/mediawiki-staging/src/ServiceConfig.php on line 9 [12:56:09] in beta or production? [12:58:14] In production [12:58:35] taavi: Even worse, it doesn't seem to be limited to scap backport but "scap sync-file" is also failing with the same error. [13:05:14] Project beta-scap-sync-world build #88410: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/88410/ [13:06:39] Okay the beta-scap-sync-world error is the same thing I'm seeing in production. [13:08:57] 10Phabricator: Maximum execution time of 30 seconds exceeded /srv/deployment/phabricator/deployment-cache/revs/f68dc24350bc897955d57eeac681db872b0b9e61/phabricator/src/applications/calendar/parser/data/PhutilCalendarRecurrenceRule.php:1251 - https://phabricator.wikimedia.org/T328118 (10Aklapper) p:05Triage→0... [13:15:14] Project beta-scap-sync-world build #88411: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/88411/ [13:15:56] _joe_: ^ maybe beta deployment needs the same fix? [13:16:25] <_joe_> awight: for sure, it will be fixed in the next 30 minutes [13:16:36] all good, thanks [13:18:16] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team (Design-Systems-Sprint): docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10bmartinezcalvo) a:05bmartinezcalvo→03None The design was done and this task is Ready f... [13:20:01] Project beta-update-databases-eqiad build #64806: 04FAILURE in 0.52 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64806/ [13:21:53] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10Security-Team, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) @bmueller mentioned that this might also be in the realm of #Security-Team. Comments welcome. I... [13:25:57] Yippee, build fixed! [13:25:57] Project beta-scap-sync-world build #88412: 09FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/88412/ [13:36:29] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10Jelto) I just found grafana-cloud.wikimedia.org. So we can also try to add dashboards for Shared Runners. There seems to be some metrics available like:... [14:05:50] (03CR) 10Hashar: "I gave it a try on https://integration.wikimedia.org/ci/job/mwservice-phpunit-coverage-patch-docker/1/console against mediawiki/services/" [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:16:27] (03CR) 10Hashar: jjb: add coverage jobs for MediaWiki services (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:26:57] (03CR) 10Hashar: "I found one issue which is that the phpunit-suite-edit injects the wrong coverage path in MediaWiki tests/phpunit/suite.xml" [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:30:07] Yippee, build fixed! [14:30:08] Project beta-update-databases-eqiad build #64807: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64807/ [14:33:31] (03CR) 10Hashar: jjb: add coverage jobs for MediaWiki services (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:33:40] (03PS2) 10Hashar: jjb: add coverage jobs for MediaWiki services [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) [14:33:42] (03PS5) 10Hashar: Zuul: [mediawiki/services/parsoid] Enable code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/882585 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [14:33:45] (03PS1) 10Hashar: dockerfiles: fix coverage path for MediaWiki services [integration/config] - 10https://gerrit.wikimedia.org/r/884904 (https://phabricator.wikimedia.org/T321837) [14:34:54] (03CR) 10Hashar: [C: 03+2] dockerfiles: add services to Quibble coverage (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/884835 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:36:14] (03CR) 10Hashar: [C: 03+2] "This would create a fixed up `docker-registry.wikimedia.org/releng/quibble-buster-php74-coverage:1.4.7-s7` image." [integration/config] - 10https://gerrit.wikimedia.org/r/884904 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:37:20] (03Merged) 10jenkins-bot: dockerfiles: fix coverage path for MediaWiki services [integration/config] - 10https://gerrit.wikimedia.org/r/884904 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [14:39:13] !log Building Docker image for https://gerrit.wikimedia.org/r/884904 [14:39:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:46:21] (03PS2) 10David Caro: tox-buster: add bats to the list of packages to install [integration/config] - 10https://gerrit.wikimedia.org/r/871178 [14:46:34] (03CR) 10David Caro: tox-buster: add bats to the list of packages to install (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [14:49:06] (03CR) 10David Caro: tox: add tox-bullseye and tox-poetry-bullseye images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/868624 (owner: 10David Caro) [14:59:14] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php74-coverage:1.4.7-s7" [integration/config] - 10https://gerrit.wikimedia.org/r/884904 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [15:02:38] (03CR) 10David Caro: tox-buster: add bats to the list of packages to install (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [15:10:32] (03CR) 10Hashar: "I have updated the two mwservice job." [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [15:54:30] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): GitLab comments should come from a GitLabBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10dancy) [16:35:35] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Cleanup and attach volumes for gitlab-runners WMCS project - https://phabricator.wikimedia.org/T328283 (10LSobanski) p:05Triage→03Low [16:41:51] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10LSobanski) p:05Triage→03Medium [17:35:20] I just noticed I still automatically move my mouse to the left after clicking a Gerrit link in Zuul lol [17:35:36] but I’m so happy that Zuul now links to the console output directly, so I don’t actually have to click the link myself :D [18:32:22] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/PushAll] Add basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/884370 (owner: 10Umherirrender) [18:32:25] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/PushAll] Add basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/884370 (owner: 10Umherirrender) [18:33:30] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PushAll] Add basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/884370 (owner: 10Umherirrender) [18:35:22] !log Zuul: [mediawiki/extensions/PushAll] Add basic tests [18:35:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:19:26] (03PS3) 10Hashar: jjb: add coverage jobs for MediaWiki services [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) [21:19:28] (03PS6) 10Hashar: Zuul: [mediawiki/services/parsoid] Enable code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/882585 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [21:19:30] (03PS1) 10Hashar: dockerfiles: use service phpunit rather than phpunit.php [integration/config] - 10https://gerrit.wikimedia.org/r/885037 (https://phabricator.wikimedia.org/T321837) [21:21:13] (03CR) 10Hashar: [C: 03+2] "A bit more mess to fix up following this morning session. MediaWiki core tests/phpunit.php has nothing to find the tests under services/p" [integration/config] - 10https://gerrit.wikimedia.org/r/885037 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [21:22:20] (03Merged) 10jenkins-bot: dockerfiles: use service phpunit rather than phpunit.php [integration/config] - 10https://gerrit.wikimedia.org/r/885037 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [21:27:00] (03CR) 10Jforrester: [C: 03+2] "This was deployed earlier, whoops." [integration/config] - 10https://gerrit.wikimedia.org/r/882675 (https://phabricator.wikimedia.org/T292818) (owner: 10Jforrester) [21:28:42] (03Merged) 10jenkins-bot: jjb: [helm-lint] Switch to image with sextant installed [integration/config] - 10https://gerrit.wikimedia.org/r/882675 (https://phabricator.wikimedia.org/T292818) (owner: 10Jforrester) [22:06:12] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10Patch-For-Review: Rewrite kokkuri in a high level language - https://phabricator.wikimedia.org/T327331 (10dancy) [23:06:42] (03PS1) 10Ahmon Dancy: kartotherian: rename kartotherian pipeline to publish [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) [23:07:46] (03CR) 10CI reject: [V: 04-1] kartotherian: rename kartotherian pipeline to publish [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:10:16] (03CR) 10Ahmon Dancy: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:16:28] (03CR) 10Jforrester: "You need to change the references to trigger-kartotherian-pipeline-kartotherian in zuul/layout.yaml for CI to pass." [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:16:30] (03CR) 10Jforrester: "You need to change the references to trigger-kartotherian-pipeline-kartotherian in zuul/layout.yaml for CI to pass." [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:16:39] (03PS2) 10Ahmon Dancy: kartotherian: rename kartotherian pipeline to publish [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) [23:17:42] (03CR) 10Ahmon Dancy: kartotherian: rename kartotherian pipeline to publish (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:18:05] (03CR) 10Jforrester: kartotherian: rename kartotherian pipeline to publish (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:18:32] (03CR) 10Jforrester: "Want me to deploy?" [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:18:51] (03CR) 10Ahmon Dancy: kartotherian: rename kartotherian pipeline to publish (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:21:52] (03PS3) 10Jforrester: jjb, Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:22:08] (03CR) 10Jforrester: [C: 03+2] "New jjb jobs deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:23:24] (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:23:47] !log Zuul: [mediawiki/services/kartotherian] Rename jobs to 'publish' for T327332 [23:23:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:23:50] T327332: Make a tool to convert .pipeline/config.yaml to .gitlab-ci.yaml - https://phabricator.wikimedia.org/T327332 [23:24:36] (03CR) 10Jforrester: [C: 03+2] "Deployed and old jobs removed." [integration/config] - 10https://gerrit.wikimedia.org/r/885047 (https://phabricator.wikimedia.org/T327332) (owner: 10Ahmon Dancy) [23:43:48] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10eoghan) a:03eoghan