[02:33:20] 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10Tgr) >>! In T328667#8677811, @dancy wrote: > However, `scap sync-file` is being executed with stdout/stderr redirected to a pipe, so... [04:18:52] 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10dancy) >>! In T328667#8678423, @Tgr wrote: >>>! In T328667#8677811, @dancy wrote: >> However, `scap sync-file` is being executed with... [06:09:55] 10Project-Admins, 10Data-Persistence (work done), 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-issue - https://phabricator.wikimedia.org/T305639 (10Marostegui) I still stand on what I mentioned at T305639#7839050. Don't really mind what is decided... [08:22:38] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10zeljkofilipin) [08:26:38] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Novem_Linguae) Thanks for the feedback. I created {T331598} and {T331595}. Feel free to create... [08:47:53] (03PS1) 10Zfilipin: WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) [08:48:03] (03CR) 10CI reject: [V: 04-1] WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [08:53:42] (03PS2) 10Zfilipin: WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) [08:54:52] (03CR) 10CI reject: [V: 04-1] WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [08:56:42] (03PS3) 10Zfilipin: WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) [09:04:01] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10daniel) [09:05:28] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10daniel) [09:36:55] 10Scap, 10Keyholder, 10serviceops, 10serviceops-collab, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10Clement_Goubert) [[ https://wikitech.wikimedia.org/wiki/Switch_Datacenter/DeploymentServer | Documentation ]] updated. [09:43:56] 10Scap, 10Keyholder, 10serviceops, 10serviceops-collab, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10hashar) 05Open→03Resolved Awesome thank you! [10:01:27] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10MediaWiki-extensions-Gadgets, 10Security-Team, 10Security: Allow Javascript files from Wikimedia GitLab to be loaded as scripts in Wikimedia wikis - https://phabricator.wikimedia.org/T321458 (10Bugreporter) @thcipriani What about hosting the resour... [10:17:42] (03CR) 10Hashar: "We need to inject `Echo` in the Phan dependencies which is a different dictionary." [integration/config] - 10https://gerrit.wikimedia.org/r/895839 (owner: 10Sbisson) [10:18:10] (03PS2) 10Hashar: Add Echo as a dependency to Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/895839 (owner: 10Sbisson) [10:19:52] (03CR) 10Hashar: [C: 03+2] Add Echo as a dependency to Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/895839 (owner: 10Sbisson) [10:21:01] (03Merged) 10jenkins-bot: Add Echo as a dependency to Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/895839 (owner: 10Sbisson) [10:23:44] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895839 [10:23:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:28:57] (03CR) 10Hashar: [C: 03+2] doc: Add 'Test coverage' link for MW core and a few others (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/895857 (owner: 10Krinkle) [10:29:35] (03Merged) 10jenkins-bot: doc: Add 'Test coverage' link for MW core and a few others [integration/docroot] - 10https://gerrit.wikimedia.org/r/895857 (owner: 10Krinkle) [10:32:57] (03CR) 10Hashar: [C: 03+2] "Deployed (the page is in cache for up to an hour though)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/895857 (owner: 10Krinkle) [10:45:22] (03PS4) 10Zfilipin: WIP Revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) [10:51:36] (03PS1) 10Lucas Werkmeister (WMDE): Allow more BuildKit frontend image names (v2) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/896056 (https://phabricator.wikimedia.org/T329553) [10:52:46] (03CR) 10Lucas Werkmeister (WMDE): Allow more BuildKit frontend image names (v2) (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/896056 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [11:11:23] (03PS5) 10Zfilipin: Partial revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) [11:16:12] James_F, hashar: could you please take a look at T330426 and https://gerrit.wikimedia.org/r/c/integration/config/+/896026 ? [11:16:12] T330426: selenium-daily Jenkins jobs failing with `node: ../src/coroutine.cc:134: void* find_thread_id_key(void*): Assertion `thread_id_key != 0x7777' failed` - https://phabricator.wikimedia.org/T330426 [11:34:35] (03PS6) 10Hashar: Partial revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [11:36:15] (03PS7) 10Hashar: Partial revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [11:36:59] !log Rolling back selenium-daily Jenkins jobs back to NodeJS 14 | https://gerrit.wikimedia.org/r/c/integration/config/+/896026/ | T330426 [11:37:00] zeljkof: ^ :) [11:37:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:37:02] T330426: selenium-daily Jenkins jobs failing with `node: ../src/coroutine.cc:134: void* find_thread_id_key(void*): Assertion `thread_id_key != 0x7777' failed` - https://phabricator.wikimedia.org/T330426 [11:37:24] (03CR) 10Hashar: [C: 03+2] "I have updated the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [11:38:33] (03Merged) 10jenkins-bot: Partial revert "jjb: Switch incidental uses of nodejs images from 14 to 16" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [11:47:15] thanks hashar ! [11:53:48] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10kostajh) It seems to not be working again. Should we re-open this task, or make a new one? [12:28:19] 10Scap: helm env vars not set when running as mwpresync user - https://phabricator.wikimedia.org/T331479 (10Aklapper) [13:07:01] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10hashar) After four years, it is better to file a new task since the root cause is most probably different :) [13:29:18] 10Gerrit: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10kostajh) [13:29:38] 10Gerrit: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983 (10kostajh) >>! In T222983#8679607, @hashar wrote: > After four years, it is better to file a new task since the root cause is most probably different :) Filed as T331626. [14:04:52] (03CR) 10Jforrester: "C-1" [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [14:17:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10LSobanski) @hashar A follow up question, is there a plan to configure rel... [14:25:02] 10Gerrit: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10valhallasw) Not entirely sure what's going on. The bot is running but there do not seem to be any emails coming in: https://gerrit-reviewer-bot.toolforge.org/ --> is the mediawiki-commits mailing list broken or did Gerrit stop... [15:00:10] 10Gerrit: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10kostajh) @hashar maybe you know about Gerrit + mediawiki-commits mailing list? [15:23:03] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10hashar) I know nothing about the `mediawiki-commits`, in mailman3 it doesn't show up at all and the archive show 0 discussions at https://lists.wikimedia.org... [15:54:02] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10hashar) I am guessing it is an issue with Mailman. https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3 shows a large queue **since March 7th 14:12**:... [16:00:38] (03CR) 10Sbisson: "Merci!" [integration/config] - 10https://gerrit.wikimedia.org/r/895839 (owner: 10Sbisson) [16:04:40] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10hashar) Icinga says OK: mailman3 queues are below the limits, but there is an alert about the runners: PROCS CRITICAL: 13 processes with UID = 38 (... [16:15:28] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10Marostegui) It looks like the restart I made fixed it or at least it is slowly going down: https://grafana.wikimedia.org/d/GvuAmuuGk/mailman3?orgId=... [16:16:11] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10hashar) 05Open→03Resolved a:03hashar Mail should be emitted again, it will take a bit of time to clear the queue though. That can be monitored... [16:16:26] (03Abandoned) 10Hashar: Revert "Revert "Allow more BuildKit frontend image names"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895775 (owner: 10Dduvall) [16:42:53] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10hashar) >>! In T331626#8680354, @hashar wrote: > PROCS CRITICAL: 13 processes with UID = 38 (list), regex args '/usr/lib/mailman3/bin/runner' > Last... [17:15:50] (03CR) 10Zfilipin: Partial revert "jjb: Switch incidental uses of nodejs images from 14 to 16" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/896026 (https://phabricator.wikimedia.org/T330426) (owner: 10Zfilipin) [17:25:19] (03PS2) 10Dduvall: Allow more BuildKit frontend image names (v2) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/896056 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:26:28] (03CR) 10Dduvall: [C: 03+2] Allow more BuildKit frontend image names (v2) (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/896056 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:27:35] (03Merged) 10jenkins-bot: Allow more BuildKit frontend image names (v2) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/896056 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:36:51] 10Gerrit, 10WikiLambda: [wm-checks-api] support kindrobot - https://phabricator.wikimedia.org/T331651 (10hashar) [18:23:11] 10Gerrit, 10Infrastructure-Foundations, 10Mail, 10SRE, 10Wikimedia-Mailing-lists: reviewer-bot is not working - https://phabricator.wikimedia.org/T331626 (10Legoktm) [19:47:28] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Accessibility: Evaluate automated accessibility integration test solutions and implement them - https://phabricator.wikimedia.org/T170096 (10bwang) [19:47:33] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector (legacy skin), 10Accessibility, and 4 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10bwang) [19:48:22] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector (legacy skin), 10Accessibility, and 4 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Vector - https://phabricator.wikimedia.org/T301184 (10bwang) [19:48:33] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Accessibility: Evaluate automated accessibility integration test solutions and implement them - https://phabricator.wikimedia.org/T170096 (10bwang) [19:55:59] 10Gerrit, 10Abstract Wikipedia team, 10WikiLambda: [wm-checks-api] support kindrobot - https://phabricator.wikimedia.org/T331651 (10SDunlap) I appreciate the support! Perhaps if you're going to integrate this into the Gerrit Checks API, we should have a quick think about what this will look like if (hopefull... [20:53:11] !log scheduled tentative phabricator update window for tuesday 2023-03-14 @ 21:00 UTC [20:53:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:01:52] 10Gerrit, 10Abstract Wikipedia team, 10WikiLambda: [wm-checks-api] support kindrobot - https://phabricator.wikimedia.org/T331651 (10hashar) I'd rather share the credentials and use a single account for that robot. That sounds easier to have only one account reporting to Gerrit which I think will make it easi... [21:02:37] 10Gerrit, 10Abstract Wikipedia team, 10WikiLambda: [wm-checks-api] support kindrobot - https://phabricator.wikimedia.org/T331651 (10hashar) (you could name it {nav DUCT}) that is an excellent robot name, then `aw-gitlab-bot` is fine as well, it is merely an implementation detail. [21:02:59] brennen: <3 [21:29:41] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Remove unused(?) fixed_settings.yaml from Phabricator puppet config - https://phabricator.wikimedia.org/T331673 (10brennen) [22:43:38] 10Phabricator: Restrict maximum file size for Phabricator file uploads - https://phabricator.wikimedia.org/T151544 (10Aklapper) [22:43:45] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Remove unused(?) fixed_settings.yaml from Phabricator puppet config - https://phabricator.wikimedia.org/T331673 (10Aklapper) [22:43:54] 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10Aklapper) [22:44:01] 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10Aklapper) 05Open→03Stalled Proceeding with the patch is bloc...