[02:12:41] 10Gerrit: Uroš LEGIJA Šuša WIKISCAN - https://phabricator.wikimedia.org/T302945 (10SHIBICU-UROSHINO) [02:13:26] 10Gerrit: Uroš LEGIJA Šuša WIKISCAN - https://phabricator.wikimedia.org/T302945 (10SHIBICU-UROSHINO) {F34973218} [07:14:24] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10Legoktm) [07:15:11] 10Project-Admins, 10User-DannyS712: Create User-ItamarWMDE project - https://phabricator.wikimedia.org/T302225 (10DannyS712) a:03DannyS712 [07:17:46] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10DannyS712) [07:17:54] 10Project-Admins, 10User-DannyS712: Create User-ItamarWMDE project - https://phabricator.wikimedia.org/T302225 (10DannyS712) 05Open→03Resolved Requested public project #user-itamarwmde has been created: https://phabricator.wikimedia.org/project/view/5804/ Please encourage interested people to visit the pr... [08:37:46] (03PS2) 10Kosta Harlan: Switch QUnit tests to use the apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [08:46:13] (03PS1) 10Hashar: dockerfiles: update to Quibble 1.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/767708 [09:09:13] (03PS2) 10Kosta Harlan: dockerfiles: update to Quibble 1.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/767708 (owner: 10Hashar) [09:09:45] (03CR) 10Kosta Harlan: [C: 03+1] dockerfiles: update to Quibble 1.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/767708 (owner: 10Hashar) [09:11:44] hmm [09:11:51] it should have failed cause I haven't tagged 1.4.2 yet [09:19:01] i didn't think the CI jobs checked for image existence? [09:20:17] !log Tag Quibble 1.4.2 @ 63d2855a1e # T302226 T302707 [09:20:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:20:20] T302226: Merges in mediawiki/core failing with "KeyError: 'server'" - https://phabricator.wikimedia.org/T302226 [09:20:20] T302707: quibble-fullrun-extensions: Fails on ve.dm.MWGraphModel.static test. - https://phabricator.wikimedia.org/T302707 [09:20:41] kostajh: I think I got confused, I wrote a test recently to ensure the quibble changelog versions are in sync with the pip installed version [09:20:44] but that does not check the tag [09:21:09] I have send the tag, building the images [09:21:36] (03CR) 10Hashar: [C: 03+2] "Quibble 1.4.2 has been tagged" [integration/config] - 10https://gerrit.wikimedia.org/r/767708 (owner: 10Hashar) [09:23:24] (03Merged) 10jenkins-bot: dockerfiles: update to Quibble 1.4.2 [integration/config] - 10https://gerrit.wikimedia.org/r/767708 (owner: 10Hashar) [09:24:51] !log Building Docker images for Quibble 1.4.2 [09:24:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:25:37] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) [09:26:55] 10Gerrit, 10Upstream: Polygerrit should fold not current line comments by default - https://phabricator.wikimedia.org/T186408 (10hashar) 05Open→03Resolved a:03hashar We have since upgraded to Gerrit 3.3, upstream has done a lot of work on the "new ui" and old comments are now automatically hidden. I thin... [09:30:35] 10Gerrit: Make a PolyGerrit plugin to welcome new users - https://phabricator.wikimedia.org/T201246 (10hashar) 05Open→03Declined I am declining this, it is a good idea but we don't have the resources to implement that feature. [09:31:00] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) Hi @brennen! Thank you for your time. Can [[ https://www.wikimedia.it/ | Wikimedia IT ]] help in any way to create a... [09:39:52] (03PS1) 10Hashar: dockerfiles: update to Quibble 1.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/767724 [09:41:35] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) [10:22:02] !log Tagged Quibble 1.4.3 @ cf5cd1a0a07 [10:22:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:22:38] (03CR) 10Hashar: [C: 03+2] "1.4.3 tagged" [integration/config] - 10https://gerrit.wikimedia.org/r/767724 (owner: 10Hashar) [10:26:34] (03Merged) 10jenkins-bot: dockerfiles: update to Quibble 1.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/767724 (owner: 10Hashar) [10:30:18] !log Building Docker images for Quibble 1.4.3 [10:30:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:51:19] (03PS1) 10Hashar: Review access change [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767695 [10:51:50] (03PS2) 10Hashar: Allow manual creation of wmf/ branches [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767695 (https://phabricator.wikimedia.org/T302964) [10:52:16] (03CR) 10Hashar: [V: 03+2 C: 03+2] Allow manual creation of wmf/ branches [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767695 (https://phabricator.wikimedia.org/T302964) (owner: 10Hashar) [11:09:17] (03PS1) 10Hashar: Review access change [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767700 [11:09:38] (03PS2) 10Hashar: Review access change [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767700 (https://phabricator.wikimedia.org/T302964) [11:09:54] (03CR) 10Hashar: [V: 03+2 C: 03+2] Review access change [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767700 (https://phabricator.wikimedia.org/T302964) (owner: 10Hashar) [11:24:48] 10Project-Admins: Archive the extension/skin - https://phabricator.wikimedia.org/T302966 (10K8rill) [11:27:06] 10Project-Admins: Archive the extension/skin - https://phabricator.wikimedia.org/T302966 (10K8rill) [11:53:00] kostajh: the GrowthExperiment change for wmf branch is passing CI now https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/767690 :]] [11:59:41] (03PS1) 10Hashar: jjb: update Quibble jobs from 1.3.0 to 1.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/767749 (https://phabricator.wikimedia.org/T300340) [12:00:20] ^^ will push that after lunch [12:14:13] \o/ [12:31:11] Hello. Quick question. Is there any easy way to install a package from one of our components within blubber? [12:31:11] I'd like to build a container that has `deb http://apt.wikimedia.org/wikimedia buster-wikimedia thirdparty/confluent` so that I can install a kafka client with `apt: { packages: [ confluent-kafka-2.11 ] }` in my blubber.yaml file. [12:31:49] Has this been done before, or am I barking up the wrong tree? Thanks. [12:32:57] btullis: you can do something like https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/Shellbox/+/refs/heads/master/.pipeline/blubber.yaml#105 [12:34:22] taavi: Nice, thanks. That looks perfect. I'll update https://wikitech.wikimedia.org/wiki/Blubber/User_Guide with an example. [12:36:46] Works a treat. 👍 [12:47:38] !log Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/ [12:47:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:57:01] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group: API Platform - https://phabricator.wikimedia.org/T301164 (10WDoranWMF) @brennen Amazing thank you so much [13:01:49] (03CR) 10Hashar: [C: 03+2] "jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/767749 (https://phabricator.wikimedia.org/T300340) (owner: 10Hashar) [13:03:46] (03Merged) 10jenkins-bot: jjb: update Quibble jobs from 1.3.0 to 1.4.3 [integration/config] - 10https://gerrit.wikimedia.org/r/767749 (https://phabricator.wikimedia.org/T300340) (owner: 10Hashar) [13:11:29] 10Release-Engineering-Team (Done by Feb 23 🧟), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10Joe) [13:23:43] 10GitLab (Infrastructure), 10serviceops: Automate setup of GitLab test instance - https://phabricator.wikimedia.org/T302976 (10Jelto) [13:23:47] (03PS1) 10Btullis: Add three more container build pipelines to datahub [integration/config] - 10https://gerrit.wikimedia.org/r/767783 (https://phabricator.wikimedia.org/T301453) [13:23:54] 10GitLab (Infrastructure), 10serviceops: Automate setup of GitLab test instance - https://phabricator.wikimedia.org/T302976 (10Jelto) p:05Triage→03Low [13:24:33] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [13:32:39] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) 05In progress→03Resolved a:03Jelto I created a dedicated task to automate the test instance creation: T302976 I also changed the flavor of `gitlab-p... [13:46:47] (03PS1) 10Hashar: Review access change [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 [13:52:52] (03PS2) 10Hashar: Prevent accidental branch push by team members [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 [14:35:34] btullis: hiii I am deploying your change at https://gerrit.wikimedia.org/r/c/integration/config/+/767783 :) [14:35:54] i am not sure why you end up having to create images for mysql, elasticsearch, kafka though [14:36:10] maybe we can come up with reference images that could be shared among different projects [14:36:19] but it is a different can of worms :] [14:37:44] the mediawiki/libs/metrics-platform repos has a few rules in zuul/layout.yaml which prevent all jobs from running [14:38:24] so that eg the job that build their java sub project only triggers when a change touches files under ./java (or .pipeline/* ) etc [14:39:48] (03CR) 10Hashar: [C: 03+2] "Jobs created :)" [integration/config] - 10https://gerrit.wikimedia.org/r/767783 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [14:41:54] (03Merged) 10jenkins-bot: Add three more container build pipelines to datahub [integration/config] - 10https://gerrit.wikimedia.org/r/767783 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [14:43:17] hashar: Ah, thanks. That's a good idea about using triggering rules. [14:43:49] btullis: it depends on how your source repo is layed out though :] [14:43:59] !log Reloading Zuul for Iae45cae8ec209a3e795fe4fd7dd92290565277db [14:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:03] btullis: you can send a dummy change to analytics/datahub to verify the jobs are all working fine :] [14:45:27] (03CR) 10Subramanya Sastry: "Will this prevent git push origin $TAG too? We use that regularly (every week) as part of our weekly deploy routine." [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [14:45:55] btullis: and an example of filtering is https://gerrit.wikimedia.org/r/c/integration/config/+/752714/3/zuul/layout.yaml [14:46:18] which should probably be documented somewhere on the wiki. I am guessing that is a useful trick [14:46:49] I'm not 100% keen on the multiple images either, but the reason I did it this way is that I'm using a third-party set of helm charts for inspiration. They use this approach to run setup jobs for the backends. I thought about using one image for it, but in the interests of saving time I just copied their approach. https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/templates [14:47:50] Nice. Thanks. I will add those rules when I come back to finesse it a bit. :-) [14:48:02] (03PS3) 10Hashar: Prevent accidental branch push by team members [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 [14:49:21] (03CR) 10Hashar: "Oops well caught Subbu. I have amended with:" [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [14:49:48] btullis: yeah it is fine, no reasons to be blocked now :] [14:51:39] (03CR) 10Subramanya Sastry: [C: 03+1] Prevent accidental branch push by team members [services/parsoid] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767703 (owner: 10Hashar) [14:51:41] I was wondering, could I temporarily get force-push access to the `wmf` branch of `analytics/datahub` in gerrit? I'd like to squash my history, so that we only have one commit in that branch. Is this permissible? Should I put a ticket request in? [14:54:22] we typically don't force push nor do we rewrite history [14:54:47] specially if commits went through Gerrit as changes, they have reviews and the history of approval [14:56:36] what I am wondering is what it will end up breaking :] [14:56:52] Yes, but I just meant temporarily, for a one-off squash. The build and release process of this particular component is being bootstrapped and so far I'm the only committer to this branch. I was just thinking of keeping it tidy. If it's likely to break something, then let's not worry about it. [14:57:48] looks like the repo also has some super large object in it :) [14:57:57] anyway [14:58:31] the way to do it is to change the permissions of the repository which can be done at https://gerrit.wikimedia.org/r/admin/repos/analytics/datahub,access [14:58:38] the repository is owned by the Analytics group [14:59:03] so you should have sufficient permissions [14:59:11] at the bottom there is an [EDIT] link [14:59:31] you would then have to change the Push permission to allow force pushing. Currently it shows: [14:59:39] Ah, great. Many thanks. I didn't realize I had the power. [14:59:44] ALLOW Administrators [Allow pushing (but not force pushing) [15:00:10] if you force push a rewrite of the history eventually the changes will disappear from gerrit ( https://gerrit.wikimedia.org/r/q/project:analytics/datahub ) [15:00:29] the various url pointing to those changes will be broken which can be annoyed when they got refered to in phabricator tasks [15:00:37] but maybe that is not a big issue :] [15:00:55] Thanks. I'll just try it for this one-off cleanup and then set it back to what it was before. [15:01:41] there is also some very large object stored in git which might make the rpeo bigger than it should [15:02:20] 245cf25d1d31 19MiB docs/imgs/feature-add-owners.gif [15:02:20] 57b8ad852dd5 20MiB docs/imgs/feature-create-new-tag.gif [15:02:20] 48ad79567002 21MiB docs/imgs/feature-rich-documentation.gif [15:02:20] 0cbfcdbd5b4e 28MiB docs/imgs/feature-create-policy.gif [15:02:20] 64ffe8a5d0d5 35MiB docs/imgs/feature-table-usage-and-stats.gif [15:03:00] oh video assets :D [15:04:02] we have git LFS support if one wanna offload those big videos files out of the main repo :] [15:05:19] Ah, sorry about that. It's just a fork of the upstream repo. https://github.com/linkedin/datahub [15:05:26] oh [15:06:00] no worries :] [15:06:21] so the upstream branch is in our Gerrit master branch [15:06:31] and the Gerrit wmf branch holds our customization isn't it? [15:07:02] in the project branch configuration you can make the repo to default to the `wmf` branch instead of the copy of the upstream change [15:07:07] can be done at https://gerrit.wikimedia.org/r/admin/repos/analytics/datahub,branches [15:07:17] by changing HEAD to point to `wmf` instead of `master` [15:07:39] this way git clone would checkout the wmf branch which is probably what most would want [15:07:52] That's right. We'd like to sync master to upstream and periodically refresh it, then rebase our wmf branch (and it's squashed commit for now) against a certain tag. [15:08:06] Great. I'll update that default branch. 👍 [15:08:17] we have multiple repo using the same pattern [15:08:33] what I do for gerrit is that I manually pull the upstream branches and tags [15:08:37] push them back to our gerrit [15:08:53] I then have my customizations in a wmf/ [15:09:10] and whenever I want to incorporate upstream changes I push the upstream code and tag [15:09:18] then do a git merge of their branch into my wmf/ [15:09:28] so something like [15:09:38] git remote update linkedin-github [15:09:59] git push gerrit linkedin-github/master:master [15:10:03] git push gerrit --tags [15:10:18] then I checkout my customization branch and merge the new tag [15:10:19] git checkout wmf [15:10:30] git merge 1.2.3 [15:10:46] and send the resulting merge commit for review :) [15:11:14] Excellent. Many thanks. OK, I'll try this out . I thought that a rebase without a merge commit would bea cleaner, but your way sounds fine to me and it other repos are using it, that's all good. [15:11:38] the problem with the rebase is it rewrites the history [15:11:53] and the new git commits are not known to gerrit [15:12:11] then you could push the rebased change directly to gerrit without creating any new change [15:12:29] I find it easier to keep the history of customizations as is and merge the upstream changes in :] [15:12:53] Oh yeah. Cool. Well I've got one to do probably next week, so I'll see how it goes. Thanks again. [15:16:22] btullis: the idea is https://phabricator.wikimedia.org/P21797 :] [15:16:35] anyway feel free to poke me about it anytime, I am more happy to assist [15:26:04] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service,rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:34:46] (03CR) 10Jforrester: [C: 03+2] "OK, after 2 hours locally this finally fully built and I verified everything seems to still work. Let's go." [integration/config] - 10https://gerrit.wikimedia.org/r/742796 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [15:36:22] (03CR) 10Reedy: "party time!" [integration/config] - 10https://gerrit.wikimedia.org/r/742796 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [15:36:35] Reedy: What could possibly go wrong?! [15:36:35] (03Merged) 10jenkins-bot: dockerfiles: [sury-php] Migrate from stretch to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/742796 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [15:37:03] !log Docker: Publishing sury-php images based on bullseye not stretch and cascade for T278203 [15:37:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:37:05] T278203: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 [16:01:19] ohh [16:01:28] ? [16:01:42] I haven't realized we still have stretch based images [16:01:54] Yeah, after this we still have 11 stretch images in CI. :-( [16:02:49] Well, ci-stretch plus gradle, helm-linter, java8, lintr, operations-puppet, perl, scap-deps-stretch, tox, typos, zuul-cloner. [16:02:51] and only one WMCS instances that is stretch based: integration-castorXX :] [16:03:07] Yeah, the integration migration has been done by awesome people. [16:03:14] :D [16:05:23] zuul-cloner should be fine, but breaking it will take down all of CI and would make people sad… [16:05:27] continious improvement [16:05:32] Quite. [16:07:36] I think zuul-cloner is only used for a few jobs that requires cloning multiple repos [16:07:55] for mediawiki repos I have copy pasted the code directly inside Quibble :] [16:08:19] maybe we can fill subtasks for each of them [16:08:27] helm-linter / operations-puppet would need sync with SRE for sure [16:08:38] gradle it might be straightforward, it probably only relies on java [16:08:53] for java8 we have an apt component to install it on buster (or bullseye i cant remember) [16:09:00] typos is probably just a grep command [16:09:14] lintr I have no idea, it might not even be used nowadays [16:09:34] anyway off to meeting :] [16:25:23] Ack. [16:32:06] RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:49:17] 10Release-Engineering-Team (Next), 10Scap: Integrate deploy-promote and stage-train scripts into Scap - https://phabricator.wikimedia.org/T302488 (10jnuche) [17:11:22] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10Jdlrobson) [17:12:42] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10Jdlrobson) The above is a train blocker as there may be some parser cache pollution occurring. Will aim to fix this today... [17:28:22] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10brennen) Hi @valerio.bozzolan - apologies this has been delayed. I'll get you set up momentarily. [17:32:44] 10Project-Admins, 10User-TheresNoTime: Create project tag for ipfs - https://phabricator.wikimedia.org/T302890 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #ipfs has been created: https://phabricator.wikimedia.org/project/view/5805/ (In case you need to edit the project itself at... [17:44:45] Project beta-code-update-eqiad build #382212: 04FAILURE in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/382212/ [17:46:42] Project beta-mediawiki-config-update-eqiad build #221: 04FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/221/ [17:54:51] Yippee, build fixed! [17:54:52] Project beta-code-update-eqiad build #382213: 09FIXED in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/382213/ [18:10:13] 10Release-Engineering-Team, 10Scap: Scap counter lists hosts twice as both "in-flight" and "left" - https://phabricator.wikimedia.org/T302990 (10Majavah) [18:10:35] 10Release-Engineering-Team, 10Scap: Scap counter lists hosts twice as both "in-flight" and "left" - https://phabricator.wikimedia.org/T302990 (10Majavah) [18:18:17] (03PS1) 10Jforrester: jjb: Update jobs using images deribed from sury-php now it uses bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/767858 (https://phabricator.wikimedia.org/T278203) [18:21:42] James_F: deribed? :D [18:30:04] Reedy: Yeah yeah. [18:30:36] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10Jdlrobson) [18:40:26] (03PS2) 10Jforrester: jjb: Update jobs using images derived from sury-php now it uses bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/767858 (https://phabricator.wikimedia.org/T278203) [18:40:28] (03PS1) 10Jforrester: [WIP] dockerfiles: [quibble-buster-php74] Migrate from sury-php to Wikimedia PHP [integration/config] - 10https://gerrit.wikimedia.org/r/767861 (https://phabricator.wikimedia.org/T293851) [18:42:43] i could use a selenium expert to have a look at https://phabricator.wikimedia.org/T302993 [18:42:54] (also, were there any recent CI setup changes that could cause this issue?) [18:51:11] MatmaRex: https://sal.toolforge.org/releng – possibly the new quibble version might have broken things if it started in the last few hours? [18:51:52] is this the same quibble version that broke things for me the last time? D: [18:52:02] MatmaRex: Alternatively, if it started yesterday, https://gerrit.wikimedia.org/r/c/integration/config/+/767493 (adding MF for Echo). [18:52:38] No, it's a newly-new version that h.ashar rolled out today. The broken one was 1.4.0 (and then 1.4.1 and 1.4.2 apparently.) [18:52:58] hmm, that could explain why it started failing [18:53:06] because we depend on Echo [18:53:11] Yeah. [18:53:15] not sure if the dependencies are transitive here? [18:53:22] They are. [18:53:35] (For 'real' dependencies, but not for phan dependencies. Don't ask.) [18:53:44] "Let's see if stuff breaks with this applied. Keeping revert hammer handy..." doesn't exactly fill one with confidence. ;-) [18:53:46] i shan't [18:54:10] 10Release-Engineering-Team, 10Scap: Scap counter lists hosts twice as both "in-flight" and "left" - https://phabricator.wikimedia.org/T302990 (10dancy) I was wondering if that would cause confusion. This answers that question. I'll subtract 'in-flight' from 'left'. [18:54:20] Reverting the CI config change will break Echo instead, which will still break you. [18:54:24] * James_F sighs. [18:54:35] i must say tht i am particularly surprised to see it fail, but see the screenshots from test artifacts clearly show the correct, working behavior [18:55:10] James_F: i proposed a patch to skip those tests in minerva [18:55:23] (now i'm waiting to see that it actually fixes the issue) [18:56:41] Yeah, that might be the right fix for now. :-( [18:57:14] (brb) [18:58:20] Quibble issue? [18:58:26] hashar: Maybe not. [18:59:42] if it is a specific test which is flappy or newly broke the easiest would be to mark it skipped and fill a task for investigation [18:59:53] Yeah, that's what Bartosz was doing. [18:59:57] Kosta and I can investigate (and others of courze) [19:01:48] I might try reproducing it w8the the quibble container and compare quibble 1.3.0 vs 1.4.3 to at least rule that out [19:02:03] I can investigate it tomorrow but it doesn’t seem quibble related. [19:02:05] Or there is a bad interactions between extensions [19:07:13] (03PS1) 10Ahmon Dancy: Don't trigger beta-mediawiki-config-update-eqiad [integration/config] - 10https://gerrit.wikimedia.org/r/767864 (https://phabricator.wikimedia.org/T302905) [19:10:14] (03CR) 10Ahmon Dancy: [C: 03+2] Don't trigger beta-mediawiki-config-update-eqiad [integration/config] - 10https://gerrit.wikimedia.org/r/767864 (https://phabricator.wikimedia.org/T302905) (owner: 10Ahmon Dancy) [19:10:23] +1 ^ [19:12:00] (03Merged) 10jenkins-bot: Don't trigger beta-mediawiki-config-update-eqiad [integration/config] - 10https://gerrit.wikimedia.org/r/767864 (https://phabricator.wikimedia.org/T302905) (owner: 10Ahmon Dancy) [19:13:03] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864 [19:13:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:23:05] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/767858 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [19:23:29] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10Jdforrester-WMF) [19:25:03] (03Merged) 10jenkins-bot: jjb: Update jobs using images derived from sury-php now it uses bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/767858 (https://phabricator.wikimedia.org/T278203) (owner: 10Jforrester) [19:28:59] (03PS1) 10Ahmon Dancy: progress reporter: Exclude 'in-flight' from 'left' count [tools/scap] - 10https://gerrit.wikimedia.org/r/767870 (https://phabricator.wikimedia.org/T302990) [19:30:52] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10brennen) [19:31:58] 10Release-Engineering-Team (Done by Feb 23 🧟), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [19:34:36] hashar: it seems there may be timeouts caused by the quibble issues [19:34:40] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/138874/console [19:34:46] 19:31:18 DEBUG:quibble.util:Waiting for Parallel npm install for projects with 'selenium-test' in package.json: 3470s elapsed, 60/61 completed [19:34:46] 19:31:28 Build timed out (after 60 minutes). Marking the build as failed. [19:35:06] also we should probaly make it print the output on failure as well as success [19:35:24] > 18:33:37 selenium-test command does not exist in project mediawiki/extensions/CodeEditor package.json, skipping npm install [19:35:35] I guess that one is normal [19:35:38] maybe that is due to the parallel npm install [19:35:43] although the output wrapper is getting confused [19:35:48] the timeout seems new though? [19:36:03] don't know if timeout is side-effect of the parallel logic [19:36:08] it didn't timeout before :) [19:36:17] 10Release-Engineering-Team (Done by Feb 23 🧟), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [19:36:25] the output portions are broken as well, but that can be fixed later [19:36:33] let me finish brushing my teeth and join back from laptop :) [19:38:26] hmm [19:39:09] the output is indeed a little messy [19:41:48] then we have been passing `--parallel-npm-install` since January 18th [19:43:08] What could go wrong! [19:43:48] sounds like a graph-based UI is in order to process logs from an operation like that. [19:45:28] (03CR) 10Zabe: [C: 03+2] progress reporter: Exclude 'in-flight' from 'left' count [tools/scap] - 10https://gerrit.wikimedia.org/r/767870 (https://phabricator.wikimedia.org/T302990) (owner: 10Ahmon Dancy) [19:46:08] (03Merged) 10jenkins-bot: progress reporter: Exclude 'in-flight' from 'left' count [tools/scap] - 10https://gerrit.wikimedia.org/r/767870 (https://phabricator.wikimedia.org/T302990) (owner: 10Ahmon Dancy) [19:46:12] something is off in that build log [19:46:20] I count 61 'Start: npm install' [19:46:26] and 61 'Finish: npm install' [19:52:01] Krinkle: I am tempted to ignore that one for now and blame it on cosmic ray :-\ [19:52:28] hashar: ack, it doesn't time out now after recheck [19:52:45] but surely the output should be enhanced [19:52:47] but non-UBN follow-up: fix collapse sections, and output stderr/stdout even if failed [19:53:00] the latter seems fairly urgent [19:53:06] and if we are busy looping on just a few loops we should probably dump the list of remaining tasks [19:53:19] CI exists to tell us about failures. Having it only say that it failed but not what is pretty bad, if that is indeed the case. Maybe it's only silent for timeouts. [19:53:47] what I don't get is that all tasks seem to have completed [19:54:21] at least based on the `<<< Finish: npm install in /path/to/x` messages [19:55:19] or the Parallel class in Quibble has some nasty off by one error [20:00:28] or it is off by one [20:20:22] (03PS1) 10Jforrester: [WIP] dockerfiles: [zuul-cloner] Migrate from stretch to buster [integration/config] - 10https://gerrit.wikimedia.org/r/767877 (https://phabricator.wikimedia.org/T278203) [21:25:07] 10Beta-Cluster-Infrastructure, 10Inuka-Team, 10Wikistories: Deploy Wikistories extension to beta cluster - https://phabricator.wikimedia.org/T303004 (10SBisson) [21:26:59] 10Beta-Cluster-Infrastructure, 10Inuka-Team, 10Wikistories: Deploy Wikistories extension to beta cluster - https://phabricator.wikimedia.org/T303004 (10SBisson) [21:33:22] (03PS1) 10Dduvall: Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/767896 (https://phabricator.wikimedia.org/T301417) [21:35:13] (03Abandoned) 10Dduvall: Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/767896 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [21:36:02] (03CR) 10Krinkle: [C: 03+2] "The depends-on change is "Included in: wmf/1.38.0-wmf.23 wmf/1.38.0-wmf.24", this matters since (afaik) we error on unknown options. Good " [tools/scap] - 10https://gerrit.wikimedia.org/r/761967 (owner: 10Ahmon Dancy) [21:37:13] (03Merged) 10jenkins-bot: Show the summary of the normal rebuildLocalisationCache.php operation [tools/scap] - 10https://gerrit.wikimedia.org/r/761967 (owner: 10Ahmon Dancy) [21:46:23] (03CR) 10Dduvall: "This change is ready for review." [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [21:47:36] (03PS5) 10Dduvall: Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) [21:48:39] dduvall: Maybe a question you can help answer... I'm working on nodejs runtime updates for Toolhub and am seeing some test slowness under the new nodejs12 image I'm updating us to. As a comparison I tried running the same tests with the "node:16-bullseye" official image and found that everything is faster, like a lot faster. [21:49:02] (03PS6) 10Dduvall: Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) [21:50:10] bd808: hmm, i'm afraid i have no idea why that would be [21:50:15] dduvall: so the question is would it be a horrible idea to be using a base image like node:16-bullseye in a pipelinelib project? That container would never be pushed or run in production, just an intermediate for compiling js [21:50:41] oh, I'm pretty sure it is that node v16 is just better written than node v12 :) [21:50:47] :) [21:51:18] well, there is currently a blubberoid policy in place that will reject any base image that's not from docker-registry.wikimedia.org [21:51:44] ummm... I don't think that is true [21:51:50] so, it won't work atm. personally i don't think there's much risk in allowing such base images in ci but we've have to discuss with others [21:52:19] I generated my local Dockerfile by sending things to blubberoid with curl [21:52:20] * dduvall tries [21:52:27] oh, ok then :) [21:53:09] hey, go for it then [21:54:18] you're catching me at a very hungry pre lunch moment, however. i should probably eat something before giving you advice or the go ahead [21:54:26] I'm going to put up a POC patch at least to see if it blows up when running the pipeline in CI [21:54:40] sounds good [21:54:49] yeah, I won't hold you to this discussion either :) Eat things [22:04:02] 10Phabricator, 10SRE-OnFire: Phabricator form request for creation of tasks tagged wikimedia-incident - https://phabricator.wikimedia.org/T303009 (10herron) [22:05:24] 10Phabricator, 10SRE-OnFire: Phabricator form request for creation of tasks tagged wikimedia-incident - https://phabricator.wikimedia.org/T303009 (10herron) [22:06:16] (03CR) 10jerkins-bot: [V: 04-1] Provide scap prep auto history browsing and replay [tools/scap] - 10https://gerrit.wikimedia.org/r/763864 (https://phabricator.wikimedia.org/T301417) (owner: 10Dduvall) [22:06:59] 10Phabricator, 10SRE-OnFire: Phabricator form request for creation of tasks tagged wikimedia-incident - https://phabricator.wikimedia.org/T303009 (10RhinosF1) I don't believe forms have granular edit permissions. If you want something you can edit easily on the fly, you can try phabulous.toolforge.org to gene... [22:12:35] 10Phabricator, 10SRE-OnFire: Phabricator form request for creation of tasks tagged wikimedia-incident - https://phabricator.wikimedia.org/T303009 (10herron) 05Open→03Resolved a:03herron Neat, thanks @RhinosF1 that should be good enough to get started with this [22:39:03] 10Phabricator, 10SRE-OnFire: Phabricator form request for creation of tasks tagged wikimedia-incident - https://phabricator.wikimedia.org/T303009 (10Aklapper) 05Resolved→03Declined No form was created; correcting task status [22:39:10] dduvall: blubber is happy to output Dockerfiles with external images, but CI seems to choke on using them (which didn't surprise me honestly) -- https://gerrit.wikimedia.org/r/c/wikimedia/toolhub/+/767902 [22:39:35] * bd808 will think about tilting at the newer nodejs base image windmill now