[00:00:48] It is. And good to know you won't be bothered by this, should there be security alerts :) [00:18:31] musikanimal: Lots of us will get the e-mails. It's on the maintainers to react to them. [00:22:02] (03PS1) 10Krinkle: docs: Add example of selective docker-pgk rebuild to README [integration/config] - 10https://gerrit.wikimedia.org/r/730948 [00:23:20] (03PS1) 10Krinkle: dockerfiles: Update quibble-fresnel to use Fresnel 1.1.0 [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) [00:23:51] (03CR) 10Jforrester: docs: Add example of selective docker-pgk rebuild to README (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/730948 (owner: 10Krinkle) [00:25:30] (03PS1) 10Krinkle: jjb: Update fresnel job to quibble-fresnel:1.1.1-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/730951 (https://phabricator.wikimedia.org/T293150) [00:26:44] (03CR) 10Jforrester: "Also tag against T292684?" [integration/config] - 10https://gerrit.wikimedia.org/r/730951 (https://phabricator.wikimedia.org/T293150) (owner: 10Krinkle) [00:27:17] (03PS2) 10Krinkle: jjb: Update fresnel job to quibble-fresnel:1.1.1-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/730951 (https://phabricator.wikimedia.org/T293150) [00:28:27] (03CR) 10Krinkle: [C: 04-1] docs: Add example of selective docker-pgk rebuild to README (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/730948 (owner: 10Krinkle) [00:29:46] (03CR) 10Krinkle: [C: 04-1] "The example-run script fails:" [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) (owner: 10Krinkle) [00:40:49] (03CR) 10Krinkle: [C: 04-1] dockerfiles: Update quibble-fresnel to use Fresnel 1.1.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) (owner: 10Krinkle) [01:20:31] (03CR) 10Krinkle: [C: 04-1] dockerfiles: Update quibble-fresnel to use Fresnel 1.1.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) (owner: 10Krinkle) [01:39:17] James_F: figured it out... [01:40:52] Apparently we're the only people on earth with a package published on npm that 1) has a dependency on something like puppeteer where a 'postinstall' scirpt is used, 2) uses a lock/shrinkwrap file, 3) has not needed to update their published npm package since 2020. [01:42:31] because 1) npm 7.x has been bugfixed to execute 'postinstall' scripts again (whch broke in 7.0) but only for top-level package, not dependencies. 2) if you don't have a lock file scripts always run, 3) if you're re-published the package since 2020 with npm 7.x or later and thus lockVersion 2, postinstall scripts also run. [01:43:03] npm 7.x is ignoring postinstall completely for packages that when published by their author (not consumer) with lockVersion 1 [01:43:46] I wish Microsoft would just rip off the bandaid and make npm-cli closed source, then the community can move on instead of tryijng to keep making it work with all the security nightmight and endless breaking changes. [01:43:51] nightmare* [01:44:05] note that npm 7.x is already EOL [01:44:11] only bug reports for npm 8.x are accepted now [01:58:29] so yeah, the immediate cause is that quibble didn't pin its npm version,and we now have a second CI image with an odd/acccidental npm change (Quibble, on node 10 with unusual npm 7; and node12 with unusual npm 7) [03:01:38] Project mwcore-phpunit-coverage-master build #1704: 04STILL FAILING in 1 min 32 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1704/ [08:51:55] I'm going to disable puppet on production GitLab (gitlab1001) soon for around two hours to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/728380 on the GitLab replica. If everything looks fine I'm going to deploy on production GitLab too. I expect something around 5-10 minutes of downtime for GitLab [08:52:35] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I'm going to disable puppet on production GitLab (gitlab1001) soon for around two hours to t... [08:58:32] (03CR) 10Hashar: [C: 03+2] Remove 'parsoid' from 'Disambiguator' dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/730898 (owner: 10C. Scott Ananian) [09:00:29] (03Merged) 10jenkins-bot: Remove 'parsoid' from 'Disambiguator' dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/730898 (owner: 10C. Scott Ananian) [09:16:15] 10Continuous-Integration-Config, 10Patch-For-Review: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10Osnard) Thanks @cscott ! I will check if we can use the Parsoid-MediaWikiService in BSVEC. [09:30:27] 10Beta-Cluster-Infrastructure: Giving Adamw and Johanna_Strodt_(WMDE) sysop on test.beta.wikipedia.wmflabs.org - https://phabricator.wikimedia.org/T293453 (10awight) [09:37:44] 10Beta-Cluster-Infrastructure: Giving Adamw and Johanna_Strodt_(WMDE) sysop on test.beta.wikipedia.wmflabs.org - https://phabricator.wikimedia.org/T293453 (10awight) 05Open→03Resolved a:03awight Done! [09:51:20] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) GitLab on the replica looks fine and change of the uid/gid was successful. I used the follow... [10:07:36] (03CR) 10Hashar: "I have rebased Disambiguator parser test that should be failing with parsoid/html and unfortunately it passes: https://gerrit.wikimedia.or" [integration/config] - 10https://gerrit.wikimedia.org/r/730898 (owner: 10C. Scott Ananian) [10:10:42] 10Beta-Cluster-Infrastructure: Giving Adamw and Johanna_Strodt_(WMDE) "staffsupport" group on vote.beta.wikipedia.wmflabs.org - https://phabricator.wikimedia.org/T293453 (10awight) [10:11:00] 10Beta-Cluster-Infrastructure: Giving Adamw and Johanna_Strodt_(WMDE) "staffsupport" group on vote.beta.wikipedia.wmflabs.org - https://phabricator.wikimedia.org/T293453 (10awight) TODO: remove us from the sysop group on beta testwiki [10:16:22] 10Continuous-Integration-Config, 10Patch-For-Review: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10hashar) Thanks @cscott for the explanation and the series of patch. I have deployed the patch to remove `parsoid` from `Disambiguator` dependencies https://gerri... [12:14:09] I'm going to re-enable puppet on production gitlab again on 12:30 utc, I expect a short (~5 minute) downtime [12:30:54] jelto: good luck! :) [12:31:12] as for gitlab csp, there is barely any report so I guess I will get them enabled at some point [12:41:38] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I re-enabled puppet on `gitlab1001` and uid/gid change and git user configuration was succes... [12:45:30] (03CR) 10Hashar: docs: Add example of selective docker-pgk rebuild to README (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/730948 (owner: 10Krinkle) [13:18:28] (03CR) 10Hashar: [C: 03+2] Add A2093064 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/730517 (owner: 10Zabe) [13:20:12] (03Merged) 10jenkins-bot: Add A2093064 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/730517 (owner: 10Zabe) [13:21:12] (03CR) 10Hashar: "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/730517 (owner: 10Zabe) [13:31:43] (03CR) 10Hashar: dockerfiles: Update quibble-fresnel to use Fresnel 1.1.0 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) (owner: 10Krinkle) [14:46:49] 10Release-Engineering-Team (Radar), 10Research (FY2021-22-Research-July-Sept), 10User-brennen: Define Metrics for Change Failure Percentage - https://phabricator.wikimedia.org/T289567 (10Miriam) Presented the results yesterday at Release Engineering's lunch and learn: * Slides: https://docs.google.com/presen... [14:47:30] 10Release-Engineering-Team (Radar), 10Research (FY2021-22-Research-Oct-Dec), 10User-brennen: Define Metrics for Change Failure Percentage - https://phabricator.wikimedia.org/T289567 (10Miriam) [14:50:36] 10Continuous-Integration-Config, 10Patch-For-Review: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10hashar) Isabelle pointed me to https://www.mediawiki.org/wiki/Parsoid#Installation . One has to explicitly: ` wfLoadExtension( 'Parsoid', 'vendor/wikimedia/parsoid... [15:00:15] 10Release-Engineering-Team (Done by Wed 06 Oct), 10GitLab, 10User-brennen: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231 (10sbassett) [15:00:19] 10Release-Engineering-Team (Done by Wed 06 Oct), 10GitLab (Administration, Settings & Policy), 10User-brennen: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231 (10sbassett) [15:01:31] Project mwcore-phpunit-coverage-master build #1705: 04STILL FAILING in 1 min 30 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1705/ [15:32:25] (03PS1) 10Zabe: Zuul: [mediawiki/extensions/MobileFrontendContentProvider] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/731122 [15:52:51] (03PS1) 10C. Scott Ananian: Revert "Remove 'parsoid' from 'Disambiguator' dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/731126 [15:54:44] (03CR) 10jerkins-bot: [V: 04-1] Revert "Remove 'parsoid' from 'Disambiguator' dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/731126 (owner: 10C. Scott Ananian) [16:08:19] (03CR) 10Jdlrobson: [C: 03+1] Zuul: [mediawiki/extensions/MobileFrontendContentProvider] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/731122 (owner: 10Zabe) [16:14:46] !log gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for T292372 [16:14:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:50] T292372: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 [16:20:43] (03PS2) 10Hashar: Revert "Remove 'parsoid' from 'Disambiguator' dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/731126 (owner: 10C. Scott Ananian) [16:21:52] (03CR) 10Hashar: [C: 03+2] "C. Scott gave the rational by chat, we actually want to run the tests with the latest parsoid version that comes from the repository." [integration/config] - 10https://gerrit.wikimedia.org/r/731126 (owner: 10C. Scott Ananian) [16:23:48] (03Merged) 10jenkins-bot: Revert "Remove 'parsoid' from 'Disambiguator' dependencies" [integration/config] - 10https://gerrit.wikimedia.org/r/731126 (owner: 10C. Scott Ananian) [16:28:14] (03CR) 10Hashar: "Deployed and I have done a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Disambiguator/+/655101" [integration/config] - 10https://gerrit.wikimedia.org/r/731126 (owner: 10C. Scott Ananian) [16:34:52] 10Release-Engineering-Team (Priority Backlog 🔥), 10mwcli, 10GitLab (CI & Job Runners), 10User-Addshore, 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10Addshore) 05Open→03Resolved a:03Addshore LGTM! [17:15:44] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [17:25:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [17:45:44] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [17:52:07] is that^ new? [17:52:25] the alert, not the semi-frequent backlog ;) [18:00:57] greg-g: likely so -- jinxer is the ircbot for alertmanager. I think a bunch of grafana alerts were recently transitioned to alertmanager...? [18:00:59] ah, https://phabricator.wikimedia.org/T292284 [18:01:03] https://wikitech.wikimedia.org/wiki/Alertmanager [18:05:15] 10Continuous-Integration-Config: Add PHP 8.1 for PHP extensions CI - https://phabricator.wikimedia.org/T293509 (10Legoktm) [18:07:50] cdanis: aha, thanks! [18:07:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10observability: Migrate Zuul alerting to Grafana / AlertManager - https://phabricator.wikimedia.org/T292284 (10CDanis) BTW, I think this fired today in the releng IRC channel? `lang=irc 17:15:44 (Queue (Jenki... [19:37:49] twentyafterfour: can we remove the default UBN status for prod error tasks? It is almost never accurate and is causing frustration/confusion several times a week now. [19:38:13] Krinkle: sure, I'm not attached to it [19:38:17] want me to do it? [19:38:21] sure :) [19:39:38] Krinkle: should it be "needs triage" or should we give it a default high priority? [19:40:21] I guess I could leave it editable instead of defaulted and hidden [19:40:51] I'd say need triage would be least surprising so that people's workflows can pick it up and triage it. "High" is the most common, but I don't know if we should adopt it as that would mean it'll be indistinguishable from an intentional action. [19:41:14] I don't expect people to change the default when filing a task. Especially in the current culture where most tasks are filed not by the owning team that knos the software in question. [19:41:35] ok so leave it defaulted to needs triage and hidden [19:42:10] I suppose it could be visible for when you file a task in your own team so that you can immediately traige it correctly. I think that's how most task forms are set up. But in other cases people can ignore it indeed. [19:42:22] the same way as assignee basically. [19:42:28] it can be handy at times. [19:46:43] done [19:49:02] (03PS2) 10Krinkle: dockerfiles: Update quibble-fresnel to use Fresnel 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/730949 (https://phabricator.wikimedia.org/T293150) [20:02:29] +1 for needs triage, thanks twentyafterfour [20:03:08] i was gonna bring this up at some point this past week but kept forgetting about it in the shuffle of filing bugs [20:20:32] (03Abandoned) 10Catrope: Review access change [design/codex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/730727 (owner: 10Kara Payne) [20:43:15] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [20:46:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [20:49:51] 10Release-Engineering-Team (Radar), 10GitLab (Administration, Settings & Policy), 10Upstream, 10User-brennen: Investigate whether issues, operations, wikis, etc. can be disabled globally on GitLab - https://phabricator.wikimedia.org/T264231 (10brennen) 05Open→03Stalled [20:50:18] 10Release-Engineering-Team (Radar), 10GitLab (Administration, Settings & Policy), 10Upstream, 10User-brennen: Look into whether GitLab time tracking can be disabled - https://phabricator.wikimedia.org/T264230 (10brennen) 05Open→03Stalled [20:51:34] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab (Infrastructure), 10User-brennen: Enable incoming mail handling for GitLab - https://phabricator.wikimedia.org/T284961 (10brennen) p:05Triage→03Low [20:52:16] 10Continuous-Integration-Config, 10Patch-For-Review: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10cscott) @hashar and I had a convo about what adding parsoid as a dependency does; I think he is going to put that on-wiki and then we can reference that here. @Os... [20:59:46] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) Looks good from my end - looks like there's some ongoing work with restore scripts, but fe... [21:01:52] 10Release-Engineering-Team (Doing), 10Security-Team, 10ContentSecurityPolicy, 10GitLab (Administration, Settings & Policy), and 3 others: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10brennen) [21:04:35] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) That's right. The restore script works when started manually but it does not work, and unfor... [21:16:12] 10Release-Engineering-Team (Priority Backlog 🔥), 10GitLab (Auth & Access), 10User-brennen: Reproduce GitLab 2fa failures - https://phabricator.wikimedia.org/T293528 (10brennen) [21:22:26] 10Release-Engineering-Team (Priority Backlog 🔥), 10MediaWiki-Docker, 10dev-images, 10mwcli, 10User-brennen: Create a PHP 8 dev image for use with mediawiki-docker-dev - https://phabricator.wikimedia.org/T283426 (10brennen) [21:26:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [21:41:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [21:56:11] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [22:32:34] (03PS2) 10Jforrester: Start branching WikiLambda for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/713947 (https://phabricator.wikimedia.org/T283031) [22:32:38] (03PS3) 10Jforrester: Start branching WikiLambda for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/713947 (https://phabricator.wikimedia.org/T283031) [22:32:47] (03CR) 10Jforrester: [C: 03+2] Start branching WikiLambda for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/713947 (https://phabricator.wikimedia.org/T283031) (owner: 10Jforrester) [22:34:15] (03Merged) 10jenkins-bot: Start branching WikiLambda for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/713947 (https://phabricator.wikimedia.org/T283031) (owner: 10Jforrester) [22:49:36] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [22:59:25] 10Continuous-Integration-Infrastructure, 10Zuul: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10bd808) Anecdotal data point: at this moment, the postmerge for https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/730941 has been queued for 1 hour and 36 minutes.... [23:00:04] building pipelinelib images on postmerge is grrrr right now ^ [23:00:37] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:06:55] 10Release-Engineering-Team (Priority Backlog 🔥), 10dev-images, 10mwcli: Add php-luasandbox to dev-images used by mwcli - https://phabricator.wikimedia.org/T286678 (10jeena) [23:08:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10jeena) [23:10:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:14:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Release Pipeline: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10bd808) [23:14:36] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [23:34:36] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org