[01:36:38] (03CR) 10SBassett: Add new permissions for supplemental security tasks (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/787110 (https://phabricator.wikimedia.org/T307036) (owner: 10SBassett) [07:33:59] (03CR) 10Hashar: [C: 03+2] Gerrit v3.4.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/787000 (https://phabricator.wikimedia.org/T292759) (owner: 10Hashar) [07:34:24] (03Merged) 10jenkins-bot: Gerrit v3.4.4 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/787000 (https://phabricator.wikimedia.org/T292759) (owner: 10Hashar) [08:09:19] 10Release-Engineering-Team (Next), 10Gerrit (Gerrit 3.4), 10Patch-For-Review: Upgrade to Gerrit 3.4 - https://phabricator.wikimedia.org/T292759 (10hashar) There is no changes to `gerrit.config` after the upgrade on gerrit2001: ` Info: Applying configuration version '(1f0a039f04) Giuseppe Lavagetto - varnish:... [08:17:48] dwalden: I am killing the beta update jobs again :-\ [08:32:34] 10Release-Engineering-Team (Next), 10Gerrit (Gerrit 3.4), 10Patch-For-Review: Upgrade to Gerrit 3.4 - https://phabricator.wikimedia.org/T292759 (10hashar) 05Open→03Resolved Both Gerrit instances have been upgraded and the service seems to work properly. For any #regression we might find, we can file them... [08:39:21] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10jnuche) hi @elukey, thank you for taking care of this task. This release also has a fix for an issue in scap unrelated to `git-lfs`, so we would like to have it rolled out to eve... [08:43:49] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10elukey) @jnuche ack perfect! I'll coordinate with Service Ops for the rollout. I'll try to skip the ORES nodes for the moment since they are still on stretch. [09:03:00] !log Gerrit upgraded to 3.4.4 at roughly 8:00 UTC [09:03:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:28:30] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Infrastructure-Foundations, 10serviceops: New scap install-world command for self-install - https://phabricator.wikimedia.org/T307081 (10jnuche) [09:35:24] 10Release-Engineering-Team (Next), 10Scap: New scap install-world command for self-deploy - https://phabricator.wikimedia.org/T307081 (10jnuche) [09:44:18] 10Release-Engineering-Team, 10serviceops: helm-linter started failing on operations/deployment-charts today - https://phabricator.wikimedia.org/T307043 (10JMeybohm) I can reproduce that when running `rake run_locally['default']` on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/787058 but not... [10:14:18] 10Release-Engineering-Team (Next), 10Scap: Stage new scap release - https://phabricator.wikimedia.org/T307086 (10jnuche) [10:22:40] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10elukey) I tested `scap pull` on mwdebug* nodes, and only in codfw I noticed a very long step: ` 10:15:41 Started scap-cdb-rebuild 10:18:51 Finished scap-cdb-rebuild (duration: 0... [10:25:30] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10elukey) Deployed also to `restbase-dev` as highlighted on wikitech, all good. I'll wait some hours and then I'll roll it out to the rest of the nodes, excluding the Ores ones. [10:25:43] hi folks! [10:26:09] I have tested scap 4.7.1 on mwdebug2* and the scap-cdb-rebuild step took ~3 mins on each node, but subsequent runs were super fast [10:26:32] does it happen sometimes or it is worth to check it in more depth? [10:37:22] o/ is gerrit now refusing ssh-rsa keys? [10:37:49] Hello everybody! I want to deploy this patch: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/775320 What should I do next? [10:39:56] elukey: I don't know the details. I guess the cdb rebuild triggered because some localization files got altered. I don't know really :-] [10:40:28] StevenSun: I think this would be https://wikitech.wikimedia.org/wiki/Backport_windows#How_to_submit_a_patch_for_backport (tl/dr you schedule your patch in the deployment calendar and a deployer will deploy it for you) [10:40:28] dcausse: wanna pair it? I have updated Gerrit from 3.3 to 3.4 this morning. I am guessing it is related [10:40:37] hashar: sure [10:40:59] I'm getting "send_pubkey_test: no mutual signature algorithm" [10:44:17] david found out the old task T276486 [10:44:18] T276486: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper) - https://phabricator.wikimedia.org/T276486 [10:44:51] that's exactly it ^ [10:52:22] 10Gerrit, 10Upstream: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper) - https://phabricator.wikimedia.org/T276486 (10hashar) >>! In T276486#7817103, @hashar wrote: > Progress update, some changes got merged upstream: > * https://gerrit-review.googlesou... [10:58:40] fix: [10:58:42] Host gerrit.wikimedia.org [10:58:42] PubkeyAcceptedKeyTypes +ssh-rsa [10:58:51] in `~/.ssh/config` [10:59:03] which will be addressed with Gerrit 3.6 [11:00:00] 10Gerrit, 10Upstream: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper) - https://phabricator.wikimedia.org/T276486 (10hashar) 05Open→03Stalled Stalled and pending Gerrit 3.6 for a complete resolution. There is a workaround on the client side config... [11:06:31] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10jnuche) @elukey my guess is the long `scap-cdb-rebuild` step was caused by a change in the l10n files. It's most likely not an issue. @dancy maybe you can give some more insight. [11:46:02] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10hashar) From the scap logs https://logstash.wikimedia.org/goto/d454018867fa2a63300115ecaf60227c spanning the last 24 hours: | mwdebug2002 | 10:20:59 | Updated 0 CDB files(s) in... [11:46:35] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10hashar) Which leads me to the question: why mwdebug2001 and mwdebug2002 did not trigger a cdb rebuild when we ran the train yesterday. Maybe they are not part of the mediawiki in... [11:46:47] elukey: jnuche: about the cdb rebuilds, scap is fine [11:46:52] I commented on the task [11:47:05] somehow mwdebug2001 and mwdebug2002 did not get cdb rebuild yesterday as part of the train [11:47:29] my guess is the codfw hosts are not targets of the train [11:47:54] so when elukey ran `scap pull` on them they triggered the cdb rebuilds since they did not had the latest code [11:49:58] maybe the mwdebug codfw hosts were not in conftool [11:50:00] hashar: thanks for the comment and the details 👍 [11:50:55] sry, conftool? :) [11:51:06] etcd [11:51:13] a key value store in which mw app servers are listed [11:51:28] so SRE can pool/unpool servers in it [11:51:36] and puppet then generate /etc/dsh/groups/* files out of it [11:51:42] files that are used by scap to find the targets [11:51:55] etcd > puppet > dsh files > scap [11:52:00] something like that iirc [11:52:17] `dsh` is a distributed ssh which we used a LONG time ago [11:52:41] so you could do something such as `dsh -g mediawiki-installation git -C /srv/mediawiki pull` [11:52:50] and that would ssh to each of the hosts in parallel and run whatever command [11:53:03] yeah, I know about the dsh groups, didn't know about conftool though, thx [11:53:08] scap went to reuse those dsh config files as a source of deployment targets [11:53:15] yeah sorry :D [11:53:27] then [11:53:43] I don't get why cdb rebuild did not trigger on mwdebug codfw hosts [11:55:28] yeah, it's weird 🤔 [11:56:16] cause mwdebug2002 hasn't had any `Started scap-cdb-rebuild` log for the last 15 days [11:56:34] with the exception of the two triggered by elukey this morning when manually running `scap pull` [11:56:59] which sounds like a bug somewhere :] [12:02:36] they are in the list of targets at least: [12:02:36] jnuche@deploy1002:~$ egrep mwdebug200[12] /etc/dsh/group/mediawiki-installation [12:02:36] mwdebug2001.codfw.wmnet [12:02:36] mwdebug2002.codfw.wmnet [12:03:26] yeah [12:03:37] so I don't quite understand what is happening [12:03:44] or scap uses a different source [12:06:27] hashar: hi, i've fixed your avatar bug https://gerrit-review.googlesource.com/c/gerrit/+/313983 [12:06:49] it's because shared-styles is no longer imported into that module (thus it doesn't get :host[hidden] [12:06:56] paladox: you are awesome :] [12:07:13] I did witness a commit about shared-styles [12:07:20] then I don't understand anything about Polymer :-\ [12:07:54] almost finished the migration to lit so no more polymer apart from plugins & external modules :D [12:08:03] in Gerrit 3.4 GrAvatar is a PolymerElement [12:08:10] yeh [12:08:14] so I would expect the PolymerElement to have that style maybe? [12:08:18] no [12:08:23] or is it a built-in from Polymer library? [12:08:42] the libary doesn't export styles, you have to do it yourself [12:08:50] shared-styles is a gerrit custom style module [12:08:58] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10jnuche) @hashar thanks for all those details. mwdebug2001 and mwdebug2002 are indeed part of the targets, so that's a good question: why didn't the cdb rebuild trigger yesterda... [12:09:13] it was removed from that module to improve performance i believe [12:09:29] okk :] [12:09:54] in 3.4 polygerrit-ui/app/styles/shared-styles.ts has both `:host([hidden])` and `[hidden]` [12:10:00] yeh [12:10:16] so when shared-styles was dropped no more hiding elements with [hidden] [12:10:20] no idea what is the difference or whether `[hidden]` should be added as well [12:10:24] ok [12:10:29] you have to manually add the css for that [12:10:38] https://gerrit-review.googlesource.com/c/gerrit/+/336094 [12:10:47] and shared style also had an `!important` statement. But that is probably not needed inside a shadow dom [12:11:07] neat :] [12:12:16] 10Release-Engineering-Team (Next), 10Gerrit (Gerrit 3.4), 10Patch-For-Review: Upgrade to Gerrit 3.4 - https://phabricator.wikimedia.org/T292759 (10hashar) One follow up {T307072} [12:12:30] paladox: thank you very much for your assistance and the patches! [12:12:36] yw [12:12:52] dcausse: to reply to your question earlier, Paladox is indeed around :] [12:17:51] 10Gerrit: Syntaxhighlighting in Gerrit is broken on php here doc syntax - https://phabricator.wikimedia.org/T288304 (10Paladox) I *think* this may be fixed from gerrit 3.6. In gerrit 3.6 we redesigned how we did syntax highlighting. Still using highlight.js but syntax highlighting should be more reliable. [12:37:23] hashar, jnuche thanks for the feedback! IIUC it is ok for me to roll out scap 4.7.1 everywhere right? (atm it is only canary nodes) [12:41:14] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10jnuche) Interestingly, the only cdb rebuild happening on mwdebug2001 before today was on the 7th of Feb: https://logstash.wikimedia.org/goto/2504b0b3cbc067990f7197efda5381c2 Als... [12:42:22] elukey: yep, I think so [12:42:34] super thanks [12:45:32] 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: helm-linter started failing on operations/deployment-charts today - https://phabricator.wikimedia.org/T307043 (10Joe) 05Open→03Resolved a:03Joe [12:49:49] paladox: thanks! [12:52:26] hashar: the bug for avatars was fixed in 3.5 (i did this in august last although didn't realise it affected 3.4) [12:52:31] dcausse: yw [12:53:53] paladox: but somehow it is still affecting master ? [12:53:59] nope [12:54:04] master and 3.5 are fixed [12:54:15] ah good :) [12:54:24] https://github.com/GerritCodeReview/gerrit/blob/master/polygerrit-ui/app/elements/shared/gr-avatar/gr-avatar.ts#L40 [12:55:45] (03PS1) 10Hashar: jjb: update integration/quibble job to 1.4.5 image [integration/config] - 10https://gerrit.wikimedia.org/r/787492 (https://phabricator.wikimedia.org/T300340) [12:55:47] (03PS1) 10Hashar: jjb: update quibble-daily jobs to Quibble 1.4.5 image [integration/config] - 10https://gerrit.wikimedia.org/r/787493 (https://phabricator.wikimedia.org/T300340) [12:55:49] (03PS1) 10Hashar: jjb: update fundraising jobs to Quibble 1.4.5 image [integration/config] - 10https://gerrit.wikimedia.org/r/787494 (https://phabricator.wikimedia.org/T300340) [12:55:52] finally [12:56:52] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10hashar) [12:57:10] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10hashar) We first need to finish the deployment of Memcached support which is T300340 [12:58:38] (03CR) 10Hashar: [C: 04-1] "We first need to complete the 1.4.3 > 1.4.5 quibble upgrade with Memcached support. The task is T300340 and I have started a few changes h" [integration/config] - 10https://gerrit.wikimedia.org/r/777030 (https://phabricator.wikimedia.org/T303867) (owner: 10Jforrester) [13:19:43] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10elukey) new version rolled out everywhere :) [13:22:43] 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10jnuche) @elukey thanks! [13:22:52] elukey: thank you ! [13:23:06] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [debs/python-opensearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/787500 [13:23:12] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [debs/python-opensearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/787500 (owner: 10QChris) [13:23:22] (03PS1) 10QChris: Import done. Revoke import grants [debs/python-opensearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/787501 [13:23:24] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [debs/python-opensearch] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/787501 (owner: 10QChris) [13:39:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10bking) Opensearch is configured at `modules/opensearch/templat... [13:48:25] (03PS2) 10Jforrester: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. [integration/config] - 10https://gerrit.wikimedia.org/r/743995 (https://phabricator.wikimedia.org/T293509) [15:25:27] hashar: the change was merged [15:33:15] 10Release-Engineering-Team (Radar), 10User-brennen: Remove database name from normalized DBQueryError message - https://phabricator.wikimedia.org/T307118 (10brennen) [15:34:00] 10Release-Engineering-Team (Radar), 10User-brennen: Remove database name from normalized DBQueryError message - https://phabricator.wikimedia.org/T307118 (10brennen) a:03Ladsgroup [15:44:26] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2002.codfw.w... [15:59:51] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2003.codfw.w... [16:14:35] (03CR) 10Ahmon Dancy: "Just marking some comments as resolved." [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [16:18:33] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2002.codfw.wmnet... [16:19:56] (03PS1) 10Ahmon Dancy: make-container-image: Add Makefile target to pause/unpause deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787529 (https://phabricator.wikimedia.org/T299648) [16:20:43] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab-runner2004.codfw.w... [16:28:13] (03CR) 10Ahmon Dancy: [C: 03+2] make-container-image: Add Makefile target to pause/unpause deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787529 (https://phabricator.wikimedia.org/T299648) (owner: 10Ahmon Dancy) [16:28:53] (03Merged) 10jenkins-bot: make-container-image: Add Makefile target to pause/unpause deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787529 (https://phabricator.wikimedia.org/T299648) (owner: 10Ahmon Dancy) [16:33:08] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2003.codfw.wmnet... [16:53:34] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab-runner2004.codfw.wmnet... [17:00:32] hashar: paladox I was expecting the new Gerrit to have support for "has:attention:" (https://gerrit-review.googlesource.com/c/gerrit/+/313490) but it doesn't seem to be working.. is this because we're still using an old version or do we need to configure something? [17:01:04] That's in 3.5+ [17:01:07] we're on 3.4 [17:02:21] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org... [17:05:42] ah thanks paladox .. so disappointing :) [17:05:47] when will we get 3.5 [17:05:51] I am really excited about that feature [17:16:49] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2003.wikimedia.org... [17:25:09] (03CR) 10Ahmon Dancy: backport: compare fetched commits with backports (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [17:26:53] (03PS24) 10Jeena Huneidi: backport: compare fetched commits with backports [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) [17:27:21] (03CR) 10Jeena Huneidi: backport: compare fetched commits with backports (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [17:28:22] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10Papaul) [17:30:34] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with... [17:30:46] (03CR) 10Ahmon Dancy: "Initial testing looks good. I'm going to extend the tests I'm working on to cover submodules and merge commits... and eventually a test f" [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [17:31:39] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host gitlab2002.wikimedia.org... [17:44:20] (03CR) 10Ahmon Dancy: backport: compare fetched commits with backports (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [17:47:09] Jdlrobson: for Gerrit 3.5 , hopefully by end of May :) [17:48:02] hashar: woopee can't wait [18:04:26] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2003.wikimedia.org with... [18:06:24] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host gitlab2002.wikimedia.org with... [18:15:04] 10Phabricator: [Feature Request] Enable "Columns" query field in Herald rules - https://phabricator.wikimedia.org/T307130 (10MBinder_WMF) [18:15:23] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10Papaul) [18:16:11] (03PS1) 10Ahmon Dancy: Make sure update-repos works regardless of the current working directory [tools/train-dev] - 10https://gerrit.wikimedia.org/r/787542 [18:16:13] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10Papaul) 05In progress→03Resolved This is complete [18:17:34] (03CR) 10Ahmon Dancy: [C: 03+2] Make sure update-repos works regardless of the current working directory [tools/train-dev] - 10https://gerrit.wikimedia.org/r/787542 (owner: 10Ahmon Dancy) [18:17:56] (03Merged) 10jenkins-bot: Make sure update-repos works regardless of the current working directory [tools/train-dev] - 10https://gerrit.wikimedia.org/r/787542 (owner: 10Ahmon Dancy) [18:20:56] (03PS25) 10Jeena Huneidi: backport: compare fetched commits with backports [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) [18:21:23] (03CR) 10Jeena Huneidi: backport: compare fetched commits with backports (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [18:50:43] !log adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string [18:50:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:57:16] !log finished editing mediawiki-new-errors [18:57:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:47:15] 10Release-Engineering-Team (Radar), 10Performance-Team, 10Wikimedia-Rdbms, 10Patch-For-Review, 10User-brennen: Remove database name from normalized DBQueryError message - https://phabricator.wikimedia.org/T307118 (10Ladsgroup) [20:02:24] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:05:04] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:06:27] 10Gerrit, 10Release-Engineering-Team: Gerrit replication after a restart takes roughly 5 hours - https://phabricator.wikimedia.org/T307137 (10hashar) [20:06:40] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:19:35] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[3|4] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Cmjohnson) [20:27:41] (03CR) 10Jforrester: [C: 03+2] Zuul: Add PHP81 as voting for libraries, PHP extensions etc. [integration/config] - 10https://gerrit.wikimedia.org/r/743995 (https://phabricator.wikimedia.org/T293509) (owner: 10Jforrester) [20:29:50] (03Merged) 10jenkins-bot: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. [integration/config] - 10https://gerrit.wikimedia.org/r/743995 (https://phabricator.wikimedia.org/T293509) (owner: 10Jforrester) [20:31:25] !log Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for T293509 [20:31:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:31:27] T293509: Add PHP 8.1 for PHP extensions CI - https://phabricator.wikimedia.org/T293509 [20:40:50] (03PS3) 10Jforrester: [WIP] List VisualEditor's core and MediaWiki documentation separately [integration/docroot] - 10https://gerrit.wikimedia.org/r/131323 [20:42:54] 10Continuous-Integration-Config: Add PHP 8.1 for PHP extensions CI - https://phabricator.wikimedia.org/T293509 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [20:46:01] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:52:01] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:52:39] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:55:40] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10MarcoAurelio) 05Open→03In progress a:03MarcoAurelio Hello @Jasonkhanlar - I am happy to create a Component project for you if you're willing to use Phabricator for bug tracking re extensi... [20:55:45] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) 05In progress→03Open a:05MarcoAurelio→03None [20:56:01] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) [20:57:27] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10MarcoAurelio) 05Open→03In progress a:03MarcoAurelio Re-adding status, assignee and subtype. [20:57:31] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) Oops! I am sorry for removing @MarcoAurelio as assignee of this task! That was not intentional! I think it happened because I was editing at same time as assigned. My apologies! [20:58:48] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10MarcoAurelio) >>! In T307136#7889308, @Jasonkhanlar wrote: > Oops! I am sorry for removing @MarcoAurelio as assignee of this task! That was not intentional! I think it happened because I was ed... [21:05:13] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10MarcoAurelio) a:05MarcoAurelio→03None Apologies from me now. I forgot [[ https://phabricator.wikimedia.org/project/manage/1776/#71827 | I removed myself ]] from #acl_project-admins a while... [21:05:55] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) > "if you're willing to use Phabricator for bug tracking re extension PhotoSwipe." Yep! Thanks! >>! In T307136#7889319, @MarcoAurelio wrote: > Apologies from me now. I forgot [[... [21:11:03] 10GitLab, 10serviceops: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [21:27:31] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10brennen) 05Open→03Resolved Things generally quiet on all wikis since deploy. Optimistically... [21:28:38] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Peachey88) 05In progress→03Open [21:30:40] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Peachey88) [21:41:16] (03PS1) 10Ahmon Dancy: Improve the pause message for pause-deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787563 [21:41:18] (03PS1) 10Ahmon Dancy: Add update-mediawiki-release-values Makefile target and utility [tools/release] - 10https://gerrit.wikimedia.org/r/787564 [21:42:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Tgr) This is really getting frustrating for the wmf branches. E.g. [[https://gerrit.... [21:42:33] (03CR) 10Ahmon Dancy: [C: 03+2] Improve the pause message for pause-deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787563 (owner: 10Ahmon Dancy) [21:43:14] (03Merged) 10jenkins-bot: Improve the pause message for pause-deploy-mwdebug [tools/release] - 10https://gerrit.wikimedia.org/r/787563 (owner: 10Ahmon Dancy) [21:44:39] (03PS2) 10Ahmon Dancy: Add update-mediawiki-release-values Makefile target and utility [tools/release] - 10https://gerrit.wikimedia.org/r/787564 [21:45:52] (03CR) 10Ahmon Dancy: [C: 03+2] Add update-mediawiki-release-values Makefile target and utility [tools/release] - 10https://gerrit.wikimedia.org/r/787564 (owner: 10Ahmon Dancy) [21:46:37] (03Merged) 10jenkins-bot: Add update-mediawiki-release-values Makefile target and utility [tools/release] - 10https://gerrit.wikimedia.org/r/787564 (owner: 10Ahmon Dancy) [21:52:55] (03PS1) 10Ahmon Dancy: update-mediawiki-release-values -> update-mediawiki-releases-values [tools/release] - 10https://gerrit.wikimedia.org/r/787565 [21:53:21] (03CR) 10Ahmon Dancy: [C: 03+2] update-mediawiki-release-values -> update-mediawiki-releases-values [tools/release] - 10https://gerrit.wikimedia.org/r/787565 (owner: 10Ahmon Dancy) [21:53:58] (03Merged) 10jenkins-bot: update-mediawiki-release-values -> update-mediawiki-releases-values [tools/release] - 10https://gerrit.wikimedia.org/r/787565 (owner: 10Ahmon Dancy) [22:33:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Jdlrobson) I don't think we should run Selenium on anything other than the master br... [22:38:37] 10Phabricator (Upstream): [Feature Request] Enable "Columns" query field in Herald rules - https://phabricator.wikimedia.org/T307130 (10Aklapper) p:05Triage→03Low [22:42:11] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Aklapper) @Jasonkhanlar: Hi, do you plan to disable https://github.com/jasonkhanlar/mediawiki-extensions-PhotoSwipe/issues ? [23:05:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security Team AppSec, 10Security-Team, and 3 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10sbassett) **Update:** I've emailed #wmf-legal requesting gu... [23:05:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security Team AppSec, 10Security-Team, and 3 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10sbassett) a:03sbassett [23:05:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security Team AppSec, 10Security-Team, and 4 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10sbassett) [23:10:09] (03CR) 10Reedy: [C: 04-1] Add new permissions for supplemental security tasks (034 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/787110 (https://phabricator.wikimedia.org/T307036) (owner: 10SBassett) [23:21:06] hi releng! We're seeing all the fundraising CiviCRM tests fail lately [23:21:18] perhaps because of an update to the CI environment [23:21:30] e.g.: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/7922/console [23:22:19] I see a block of text I'm not sure was there before, on the mariadb install. Says it creates root and nobody all-priv accounts, but they depend on being the unix users root and nobody to connect [23:22:48] then down a little while where we try to connect as root and create DBs, it logs this: [23:22:52] ERROR 1698 (28000): Access denied for user 'root'@'localhost' [23:23:37] How should we update the crm unit tests so they can keep creating those DBs? [23:25:15] 10Project-Admins: Create project tag for mediawiki/extensions/PhotoSwipe - https://phabricator.wikimedia.org/T307136 (10Jasonkhanlar) >>! In T307136#7889737, @Aklapper wrote: > @Jasonkhanlar: Hi, do you plan to disable https://github.com/jasonkhanlar/mediawiki-extensions-PhotoSwipe/issues ? I did not have any p... [23:26:24] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 3 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10Jdlrobson) a:03bwang [23:46:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security Team AppSec, 10Security-Team, and 4 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10bd808) >>! In T304737#7846559, @sbassett wrote: > Given tha...