[07:31:56] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Joe) >>! In T266055#7445971, @Krinkle wrote: >>>! In T266055#7275900, @dancy wrote: >> We ran the test today. @jijiki suppl... [08:29:20] 10Continuous-Integration-Infrastructure, 10observability: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10hashar) [08:30:16] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) [08:31:37] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10hashar) >>! In T283582#7606038, @Dzahn wrote: > for the record: I have absolutel... [08:33:09] (03CR) 10Hashar: [C: 03+2] zuul: [openstack/horizon/wmf-proxy-dashboard] Add tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/752168 (owner: 10Majavah) [08:34:59] (03Merged) 10jenkins-bot: zuul: [openstack/horizon/wmf-proxy-dashboard] Add tox-docker [integration/config] - 10https://gerrit.wikimedia.org/r/752168 (owner: 10Majavah) [08:35:58] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/752168 (owner: 10Majavah) [08:40:03] (03CR) 10Hashar: [C: 03+2] "Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/752155 (owner: 10Hashar) [08:41:58] (03Merged) 10jenkins-bot: jjb: ensure docker macro options is always a string [integration/config] - 10https://gerrit.wikimedia.org/r/752155 (owner: 10Hashar) [08:53:28] (03PS2) 10Hashar: jjb: move docker macro entrypoint out of "options" [integration/config] - 10https://gerrit.wikimedia.org/r/752156 [08:53:30] (03PS1) 10Hashar: jjb: move --entrypoint to be first option [integration/config] - 10https://gerrit.wikimedia.org/r/752601 [09:11:18] (03CR) 10Hashar: [C: 03+2] jjb: move --entrypoint to be first option [integration/config] - 10https://gerrit.wikimedia.org/r/752601 (owner: 10Hashar) [09:12:28] (03CR) 10Hashar: [C: 03+2] "It is a noop after https://gerrit.wikimedia.org/r/c/integration/config/+/752601 :]" [integration/config] - 10https://gerrit.wikimedia.org/r/752156 (owner: 10Hashar) [09:13:27] (03Merged) 10jenkins-bot: jjb: move --entrypoint to be first option [integration/config] - 10https://gerrit.wikimedia.org/r/752601 (owner: 10Hashar) [09:14:44] (03Merged) 10jenkins-bot: jjb: move docker macro entrypoint out of "options" [integration/config] - 10https://gerrit.wikimedia.org/r/752156 (owner: 10Hashar) [09:24:29] (03PS1) 10Hashar: jjb [integration/config] - 10https://gerrit.wikimedia.org/r/752607 [09:24:48] (03PS2) 10Hashar: jjb: move more docker macro entrypoint out of "options" [integration/config] - 10https://gerrit.wikimedia.org/r/752607 [09:25:35] (03CR) 10Hashar: [C: 03+2] "Missed from previous change :(" [integration/config] - 10https://gerrit.wikimedia.org/r/752607 (owner: 10Hashar) [09:27:28] (03Merged) 10jenkins-bot: jjb: move more docker macro entrypoint out of "options" [integration/config] - 10https://gerrit.wikimedia.org/r/752607 (owner: 10Hashar) [09:40:55] (03CR) 10Hashar: [C: 03+2] Add 'parsoid' to 'wikihiero' extension dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/752164 (https://phabricator.wikimedia.org/T272931) (owner: 10Isabelle Hurbain-Palatin) [09:42:36] (03CR) 10Hashar: [C: 03+1] "Kosta: lets deploy that when you have time to pair on it and verify it is fine. Just poke me :)" [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [09:42:49] (03Merged) 10jenkins-bot: Add 'parsoid' to 'wikihiero' extension dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/752164 (https://phabricator.wikimedia.org/T272931) (owner: 10Isabelle Hurbain-Palatin) [09:43:02] hi hashar [09:44:07] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) GitLab Runner in `eqiad` and `codfw` export Prometheus metrics now. I created a [GitLab CI overview... [09:46:58] (03CR) 10Kosta Harlan: zuul: Add VisualEditor/parsoid dependencies to GrowthExperiments (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [09:58:23] kostajh: lets do it ;) [09:58:45] hashar: let me just see what's going on with the current build [09:59:20] hashar: ok, it failed, so yes, let's try to deploy the patch [09:59:22] I did the same change with Isabelle for wikihierio [09:59:25] ok [09:59:35] (03CR) 10Hashar: [C: 03+2] "Pairing it with Kosta ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [10:01:30] (03Merged) 10jenkins-bot: zuul: Add VisualEditor/parsoid dependencies to GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [10:01:58] kostajh: deployed! [10:02:10] hashar: danke, let's see what happens [10:02:28] that would also cause parsoid to be added to any extension that also depends on GrowthExperiments [10:02:33] we will see [10:07:16] quick afk [10:07:58] kostajh: the tests fail :-\ [10:08:24] 00:03:36.118 [0-0] Error in "add link.link inspector appears after clicking through task from Special:Homepage" [10:08:25] 00:03:36.118 Error: element (".mw-ge-recommendedLinkToolbarDialog") still not displayed after 20000ms [10:08:30] hashar: that's before the deployment [10:08:34] i'm doing a recheck now [10:08:51] AHH [10:09:04] 😅 [10:09:15] watching https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-selenium-docker/95471/console now [10:13:19] hashar: success :) [10:14:13] but then https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/129651/ fails [10:14:22] cause it does not have mediawiki/services/parsoid [10:18:44] oh [10:20:20] what do you suggest to do? [10:24:33] no idea :( [10:24:52] beside having the test detect whther parsoid is there or not [10:25:29] or we check with parsoid team whether it is fine to have mediawiki/services/parsoid in the wmf-quibble jobs [10:25:39] but those are intended to test what is going to be pushed to prod [10:33:02] phab spam for removal: https://phabricator.wikimedia.org/T298853#7607883 user already disabled [10:34:22] p858snake: I don't have access to my 2FA token right now :/ [11:03:34] (03PS1) 10Hashar: jjb: drop extra space in docker Jinja2 template [integration/config] - 10https://gerrit.wikimedia.org/r/752629 [11:07:47] hashar: yeah, I'm not sure. do you have a preference? [11:08:07] kostajh: it is a bit of a mess to untangle :-\ [11:08:48] hashar: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-selenium-docker/95475/artifact/log/Homepage-Heading-shows-the-logged-in-user's-name-2022-01-10T10-24-24-690Z.mp4 heh [11:11:18] hashar: i've gone with a conditional load in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/752616, so we can keep the jjb/zuul patch merged, and when parsoid needs to be manually loaded, it will be. I guess the difference is that it's loaded from vendor/wikimedia/parsoid instead of services [11:11:37] (03PS1) 10Hashar: jjb: simplify Jinja2 whitespace with volume [integration/config] - 10https://gerrit.wikimedia.org/r/752633 [11:12:26] kostajh: there are some notes from a discussion I had with C. Scott back in october https://www.mediawiki.org/wiki/Parsoid#Continuous_Integration [11:12:34] and the problem is known to the team [11:12:50] they do have projects to add parsoid support to several extension such as translate or wikihiero [11:13:05] and injecting mediawiki/services/parsoid seems to be the easiest solution for now [11:13:34] we would probably need to rationalize all of that eventually. No idea how though [11:13:41] lunch & [11:40:13] hashar: I think https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/752616 should work, if you can +2 when you're back. in the meantime all other builds for GrowthExperiments are broken :\ [11:43:09] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: Upgrade deployment-prep Swift cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298253 (10MatthewVernon) [11:43:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10MatthewVernon) [11:44:12] 10Beta-Cluster-Infrastructure, 10SRE-swift-storage: Upgrade deployment-prep Swift cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298253 (10MatthewVernon) [much of the prod swift infrastructure is still running on Stretch, FWIW; Thanos frontends are now Bullseye] [12:01:09] 10Continuous-Integration-Infrastructure, 10observability: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10jbond) This is happening because ` $facts['ipmi_lan'] ` is not populated and a closer look suggests idrac is unresponsive, both via bmc_config and ssh, i think you w... [13:10:49] (03PS1) 10Sebastian Berlin (WMSE): Review access change [labs/tools/Isa] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/752276 [13:30:07] (03PS1) 10Sebastian Berlin (WMSE): Add gate-and-submit-l10n to Isa [integration/config] - 10https://gerrit.wikimedia.org/r/752658 (https://phabricator.wikimedia.org/T222291) [13:32:09] (03PS2) 10Sebastian Berlin (WMSE): Review access change [labs/tools/Isa] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/752276 (https://phabricator.wikimedia.org/T222291) [13:57:49] (03CR) 10Jforrester: [C: 03+2] Add gate-and-submit-l10n to Isa [integration/config] - 10https://gerrit.wikimedia.org/r/752658 (https://phabricator.wikimedia.org/T222291) (owner: 10Sebastian Berlin (WMSE)) [13:59:26] (03PS7) 10David Caro: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 [14:00:07] hashar: I know it's been a while, but I'd like to retake https://gerrit.wikimedia.org/r/c/integration/config/+/701121, what needs to be done to get it moving? [14:00:40] (03Merged) 10jenkins-bot: Add gate-and-submit-l10n to Isa [integration/config] - 10https://gerrit.wikimedia.org/r/752658 (https://phabricator.wikimedia.org/T222291) (owner: 10Sebastian Berlin (WMSE)) [14:01:15] !log Zuul: Add gate-and-submit-l10n to Isa for T222291 [14:01:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:01:17] T222291: Add support for ISA in translatewiki.net - https://phabricator.wikimedia.org/T222291 [15:16:07] dcaro: I guess I forgot about that Poetry image :D [15:16:22] I will see what I can do this week but it is going to be busy [15:18:38] hashar: no problem, let me know if I can do anything/help anyhow [15:44:57] dcaro: building it [15:45:37] (03PS8) 10Hashar: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [15:45:48] dcaro: I just had to fix the email address in the changelog [15:46:08] (03CR) 10Hashar: [C: 03+2] "In the changelog I have fixed the domain name in your email address ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [15:46:22] hashar: \o/ [15:48:46] (03Merged) 10jenkins-bot: tox-poetry-buster: Added a new image for poetry based builds [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [15:54:08] (03PS1) 10Hashar: Add tox-poetry job for cloud/wmcs-backups [integration/config] - 10https://gerrit.wikimedia.org/r/752686 [15:55:03] (03CR) 10Hashar: "Here is the Jenkins job and the Zuul CI glue to run poetry on cloud/wmcs-backups. The image is still building though." [integration/config] - 10https://gerrit.wikimedia.org/r/752686 (owner: 10Hashar) [15:55:50] dcaro: the image is still building cause it is taking ages to download the parent releng/tox-buster image [15:55:51] oh [15:55:56] done after 5+ minutes bah [15:56:06] Poetry (1.1.12) is installed now. Great! [15:56:41] (03CR) 10jerkins-bot: [V: 04-1] Add tox-poetry job for cloud/wmcs-backups [integration/config] - 10https://gerrit.wikimedia.org/r/752686 (owner: 10Hashar) [15:56:58] (03CR) 10Hashar: "Successfully published image docker-registry.discovery.wmnet/releng/tox-poetry-buster:0.0.1" [integration/config] - 10https://gerrit.wikimedia.org/r/701121 (owner: 10David Caro) [15:58:25] (03PS2) 10Hashar: Add tox-poetry job for cloud/wmcs-backups [integration/config] - 10https://gerrit.wikimedia.org/r/752686 [15:59:47] (03CR) 10David Caro: Add tox-poetry job for cloud/wmcs-backups (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/752686 (owner: 10Hashar) [16:00:49] hashar: 🎉 awesome! thanks! [16:03:43] (03CR) 10Hashar: [C: 03+2] Add tox-poetry job for cloud/wmcs-backups [integration/config] - 10https://gerrit.wikimedia.org/r/752686 (owner: 10Hashar) [16:03:53] dcaro: I have deployed the Jenkins job, will reload Zuul once ^ is merged [16:05:36] (03Merged) 10jenkins-bot: Add tox-poetry job for cloud/wmcs-backups [integration/config] - 10https://gerrit.wikimedia.org/r/752686 (owner: 10Hashar) [16:06:43] rechecking https://gerrit.wikimedia.org/r/c/cloud/wmcs-backups/+/701380 [16:07:19] https://integration.wikimedia.org/ci/job/tox-poetry-docker/1/console [16:09:32] neat [16:09:41] hmm success [16:09:53] the job would keep reinstalling the dependencies until a change is successfully merged [16:10:00] which would then trigger a save of /cache [16:10:16] and from there future build would rsync the cache in the container which will make future builds faster ;) [16:10:21] so hmm congratulations [16:10:45] can probably announce the new feature to wikitech-l for those interested in using Poetry [16:10:46] \o/ [16:11:47] \o/ [16:16:23] is that something you do? Or should I? [16:30:28] (03PS4) 10Ahmon Dancy: Make scap prep re-runnable [tools/scap] - 10https://gerrit.wikimedia.org/r/752189 [16:30:36] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch: Upgrade deployment-prep Elastic cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298252 (10Gehel) [16:33:03] 10Beta-Cluster-Infrastructure, 10Discovery-Search, 10Elasticsearch: Upgrade deployment-prep Elastic cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298252 (10Majavah) [16:37:26] p858snake: eventually the Phabricator user got disabled ;) [16:38:39] (03CR) 10Hashar: [C: 03+2] jjb: drop extra space in docker Jinja2 template [integration/config] - 10https://gerrit.wikimedia.org/r/752629 (owner: 10Hashar) [16:39:15] (03CR) 10Hashar: [C: 03+2] "And it is a noop!" [integration/config] - 10https://gerrit.wikimedia.org/r/752633 (owner: 10Hashar) [16:40:28] (03Merged) 10jenkins-bot: jjb: drop extra space in docker Jinja2 template [integration/config] - 10https://gerrit.wikimedia.org/r/752629 (owner: 10Hashar) [16:40:58] (03Merged) 10jenkins-bot: jjb: simplify Jinja2 whitespace with volume [integration/config] - 10https://gerrit.wikimedia.org/r/752633 (owner: 10Hashar) [17:13:45] !log Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace [17:13:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:50:50] (03PS1) 10Accraze: inference: add publishing pipelines for postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/752714 (https://phabricator.wikimedia.org/T297823) [19:56:58] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) ACK, understood @Hashar, then this goes back to #ops-codfw [19:58:58] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul it seems that "broken DRAC" is actually the reason for b... [19:59:20] 10Continuous-Integration-Infrastructure, 10observability, 10ops-codfw: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10Dzahn) [20:00:10] 10Continuous-Integration-Infrastructure, 10observability, 10ops-codfw: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10Dzahn) [20:00:16] 10Release-Engineering-Team (Seen), 10serviceops: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [20:00:28] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Dzahn) [20:01:24] 10Continuous-Integration-Infrastructure, 10observability, 10ops-codfw: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10Dzahn) added both T283582 and T294276 as parent tasks cc: #releng-radar [20:03:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 10observability, 10ops-codfw, 10serviceops-radar: contint2001.mgmt disappeared from Icinga - https://phabricator.wikimedia.org/T298861 (10Dzahn) [20:04:50] 10Release-Engineering-Team (Seen), 10serviceops: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) This would also fix T283582 and T298861 and putting further effort into that kind of thing [21:54:59] (03PS1) 10Hashar: jjb: drop extra double quotes in docker run -e [integration/config] - 10https://gerrit.wikimedia.org/r/752745 [22:00:06] (03PS1) 10Hashar: jjb: add 'environment' to Jinja2 docker macro [integration/config] - 10https://gerrit.wikimedia.org/r/752746 [23:00:04] 10MediaWiki-Releasing, 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Write and send supplementary release announcement for extensions and skins with security patches (1.35.5/1.36.3/1.37.1) - https://phabricator.wikimedia.org/T292236 (10Mstyles) [23:01:07] 10MediaWiki-Releasing, 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Write and send supplementary release announcement for extensions and skins with security patches (1.35.5/1.36.3/1.37.1) - https://phabricator.wikimedia.org/T292236 (10Mstyles) [23:35:55] (03CR) 1020after4: [C: 03+1] "One general comment, not a blocker. I do think we need to generally reconsider the workflow for patches and especially how it relates to t" [tools/scap] - 10https://gerrit.wikimedia.org/r/752189 (owner: 10Ahmon Dancy)