[01:48:58] (03PS1) 10Krinkle: doc: Update php-cssjanus URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/877290 [01:49:13] (03CR) 10Krinkle: [C: 03+2] doc: Update php-cssjanus URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/877290 (owner: 10Krinkle) [01:49:52] (03Merged) 10jenkins-bot: doc: Update php-cssjanus URLs [integration/docroot] - 10https://gerrit.wikimedia.org/r/877290 (owner: 10Krinkle) [02:05:21] (03PS2) 10Ejegg: Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [02:06:09] (03PS3) 10Ejegg: Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [02:06:37] (03CR) 10Ejegg: "PS2+3 fixed spacing" [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [11:29:46] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10kostajh) [12:37:05] 10Release-Engineering-Team (Onboarding 🚀), 10Scap, 10Sustainability (Incident Followup), 10User-brennen: scap's logstash_checker.py is blissfully unaware of any logstash indexing latency - https://phabricator.wikimedia.org/T255197 (10LSobanski) [12:50:11] 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10LSobanski) @hashar As the original requester (T307349#7895775), could you help clarify what's needed here? [15:23:14] GitLab needs a short maintenance break (around 5 to 10 minutes) at 16:OO UTC (in 40 minutes) [15:26:31] (03PS4) 10Jforrester: Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) [15:26:45] (03CR) 10Jforrester: [C: 03+2] Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [15:27:57] (03Merged) 10jenkins-bot: Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [15:28:13] !log Zuul: Add experimental jobs for fundraising/REL1_39 for T326372 [15:28:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:15] T326372: Update CI config for new fundraising/REL1_39 branch of mediawiki/core under php7.4 - https://phabricator.wikimedia.org/T326372 [15:58:13] (03PS1) 10Jforrester: Zuul: Skip quibble-donationinterface-REL1_39-php74-docker on deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/878119 (https://phabricator.wikimedia.org/T326372) [15:58:15] (03PS1) 10Jforrester: [DNM] Zuul: Make fundraising/REL1_39 jobs voting [integration/config] - 10https://gerrit.wikimedia.org/r/878120 (https://phabricator.wikimedia.org/T326372) [15:58:17] (03PS1) 10Jforrester: [DNM] Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [integration/config] - 10https://gerrit.wikimedia.org/r/878121 (https://phabricator.wikimedia.org/T326372) [15:58:23] (03PS1) 10Jforrester: [DNM] Zuul: [wikimedia/fundraising/SmashPig] Drop PHP73 testing [integration/config] - 10https://gerrit.wikimedia.org/r/878122 (https://phabricator.wikimedia.org/T326372) [15:58:27] (03PS1) 10Jforrester: [DNM] jjb: Drop fundraising/REL1_35 jobs, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/878123 (https://phabricator.wikimedia.org/T326372) [15:59:36] (03CR) 10CI reject: [V: 04-1] [DNM] Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [integration/config] - 10https://gerrit.wikimedia.org/r/878121 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [15:59:38] (03CR) 10CI reject: [V: 04-1] [DNM] Zuul: [wikimedia/fundraising/SmashPig] Drop PHP73 testing [integration/config] - 10https://gerrit.wikimedia.org/r/878122 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [15:59:44] (03CR) 10CI reject: [V: 04-1] [DNM] jjb: Drop fundraising/REL1_35 jobs, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/878123 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:25:56] (03CR) 10Jforrester: [C: 03+2] Zuul: Skip quibble-donationinterface-REL1_39-php74-docker on deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/878119 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:27:11] (03Merged) 10jenkins-bot: Zuul: Skip quibble-donationinterface-REL1_39-php74-docker on deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/878119 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:28:30] (03PS2) 10Jforrester: [DNM] Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [integration/config] - 10https://gerrit.wikimedia.org/r/878121 (https://phabricator.wikimedia.org/T326372) [16:28:32] (03PS2) 10Jforrester: [DNM] Zuul: [wikimedia/fundraising/SmashPig] Drop PHP73 testing [integration/config] - 10https://gerrit.wikimedia.org/r/878122 (https://phabricator.wikimedia.org/T326372) [16:28:34] (03PS2) 10Jforrester: [DNM] jjb: Drop fundraising/REL1_35 jobs, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/878123 (https://phabricator.wikimedia.org/T326372) [16:29:37] (03CR) 10CI reject: [V: 04-1] [DNM] Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [integration/config] - 10https://gerrit.wikimedia.org/r/878121 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:29:39] (03CR) 10CI reject: [V: 04-1] [DNM] Zuul: [wikimedia/fundraising/SmashPig] Drop PHP73 testing [integration/config] - 10https://gerrit.wikimedia.org/r/878122 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:29:43] (03CR) 10CI reject: [V: 04-1] [DNM] jjb: Drop fundraising/REL1_35 jobs, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/878123 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:34:31] (03PS2) 10Jforrester: [DNM] Zuul: Make fundraising/REL1_39 jobs voting [integration/config] - 10https://gerrit.wikimedia.org/r/878120 (https://phabricator.wikimedia.org/T326372) [16:34:33] (03PS3) 10Jforrester: [DNM] Zuul: Drop fundraising/REL1_35 jobs, everything has moved to REL1_39 now [integration/config] - 10https://gerrit.wikimedia.org/r/878121 (https://phabricator.wikimedia.org/T326372) [16:34:35] (03PS3) 10Jforrester: [DNM] Zuul: [wikimedia/fundraising/SmashPig] Drop PHP73 testing [integration/config] - 10https://gerrit.wikimedia.org/r/878122 (https://phabricator.wikimedia.org/T326372) [16:34:37] (03PS3) 10Jforrester: [DNM] jjb: Drop fundraising/REL1_35 jobs, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/878123 (https://phabricator.wikimedia.org/T326372) [16:34:39] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/DonationInterface] Add REL1_39 exp. job [integration/config] - 10https://gerrit.wikimedia.org/r/878130 (https://phabricator.wikimedia.org/T326372) [16:39:35] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/DonationInterface] Add REL1_39 exp. job [integration/config] - 10https://gerrit.wikimedia.org/r/878130 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:40:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DonationInterface] Add REL1_39 exp. job [integration/config] - 10https://gerrit.wikimedia.org/r/878130 (https://phabricator.wikimedia.org/T326372) (owner: 10Jforrester) [16:46:16] (03PS1) 10Jforrester: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf [integration/config] - 10https://gerrit.wikimedia.org/r/878132 (https://phabricator.wikimedia.org/T326653) [16:48:06] (03CR) 10Jforrester: [C: 03+2] Zuul: Fix extension-quibble to run vendor not composer quibble on wmf [integration/config] - 10https://gerrit.wikimedia.org/r/878132 (https://phabricator.wikimedia.org/T326653) (owner: 10Jforrester) [16:49:15] (03Merged) 10jenkins-bot: Zuul: Fix extension-quibble to run vendor not composer quibble on wmf [integration/config] - 10https://gerrit.wikimedia.org/r/878132 (https://phabricator.wikimedia.org/T326653) (owner: 10Jforrester) [16:50:02] !log Zuul: Fix extension-quibble to run vendor not composer quibble on wmf for T326653 [16:50:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:05] T326653: PHPStan failing on Wikibase backports to wmf branches - https://phabricator.wikimedia.org/T326653 [16:50:41] thanks James_F! [16:50:54] I uploaded a new patch set of the Wikibase backport removing my workaround, let’s see if it works [16:50:56] Lucas_WMDE: You're welcome! Sorry for my breaking it in the first place. :-( [16:55:21] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Update CI config for new fundraising/REL1_39 branch of mediawiki/core under php7.4 - https://phabricator.wikimedia.org/T326372 (10Jdforrester-WMF) OK, the new PHP74/RE... [17:33:22] hello hello I'm having a scap issue, hopefully it's not too bad... [17:33:22] It boils down to https://www.irccloud.com/pastebin/kI4IpTV0/ [17:33:54] Scap deploys from deploy1002 used to work, and I don't think anything specific changed on that server since, but yet... [17:34:05] anything I should check? [17:39:50] dancy: ^ how does the "scap" python module get installed? that netbox-dev host has scap but not the python module "scap" [17:40:18] after investigation with Riccardo, it might be due to https://phabricator.wikimedia.org/T303559 [17:40:32] but I can't find any doc on Wikitech [17:40:37] I am wondering if you need a deploy of scap to netbox-dev* [17:40:51] scap deploy of scap has list of targe thosts [17:40:58] maybe that is missing netbox-dev2002 [17:41:21] mutante: where do you see that list? [17:42:11] XioNoX: in hieradata. common/scap/dsh.yaml:scap::dsh::groups [17:42:27] is it safe to mess with it myself? [17:42:32] XioNoX: and then there is the question if "scap::target" is somewhere in puppet code applied on that host / role [17:43:26] XioNoX: I would say it's safe as long as you stay in your section.. like in this case for "homer", is that right? [17:43:30] mutante: looks like it's there https://github.com/wikimedia/puppet/blob/production/hieradata/common/scap/dsh.yaml#L123 [17:44:27] XioNoX: right.. but I think there must also be a list of which hosts get scap itself deployed to it [17:44:30] oh but that was tthere since a while [17:44:37] I see [17:44:40] hmmm... [17:44:59] let me see if your /usr/bin/scap is different from the one on deploy1002 [17:45:48] I think you're onto something that scap is now deployed with scap, and maybe that host got missed [17:46:02] yea, so just like we have the host lists here for tools [17:46:10] where is the list for meta-scap [17:46:23] is it "scap_masters" or not [17:46:26] that I am not sure yet [17:46:36] and where is the doc? 👀 [17:47:32] so..there is a python module called "scap" [17:47:40] and it cant find that ..let's get back to that [17:47:49] question is how did that module get on deployment hosts [17:47:54] as opposed to just /usr/bin/scap [17:48:06] do you have time to have this solved via ticket? [17:48:16] I have to step away in a few minutes, I'll resume my search tommorrow, let me know if you find something [17:48:23] * dancy reads [17:48:36] dancy: ModuleNotFoundError: No module named 'scap' [17:49:08] the actual /usr/bin/scap file exists but not the module [17:50:13] wondering how modules get installed..hopes the answer is not 'pip install' [17:50:31] ok. I have answers! [17:50:37] :)) [17:51:22] So, yes, netbox-dev2002 needs to be in the list of "scap targets", then we can run `scap install-world` to deploy scap to that host. [17:52:37] What does `dpkg -S /usr/bin/scap` return on that host? [17:52:55] that is "add a scap::target" snippet in profile::netbox or one of the other profiles included in role(netbox::standalone) [17:52:59] right [17:53:14] dancy: dpkg-query: no path found matching pattern /usr/bin/scap [17:53:27] nod. okay.. I wanted to be sure that there's no lingering scap deb package. [17:53:41] so /usr/bin/scap is some remnant as it stands. [17:54:24] or.. it's a stub waiting for that install-world to happen to this host.. I'd need to see the whole file. [17:55:06] in fact, once it gets the scap::target snippet, it will probably bootstrap itself. [17:55:13] let's find out. :-) [17:55:25] XioNoX: so yea, in the past it worked because a scap deployment host did not have to be a target for scap. but now it needs to be a scap::target. so the fix is 'grep -r scap::target *" in puppet repo and copy from one of the existing ones and put it in one of the profiles you have on netbox::standalone [17:56:07] ok, I will just upload a patch so we can compile it [17:57:01] I really have to go, will catch up asap, but thanks both of you! [17:58:10] Farewll! [17:58:16] +e [17:59:42] cya, yw [18:00:09] so "scap::target { 'scap': [18:00:19] deploY_user => 'scap', ? :p [18:00:27] if it's just for scap itself [18:06:07] hmm... [18:06:55] [netbox-dev2002:~] $ id scap [18:06:55] uid=919(scap) gid=919(scap) groups=919(scap) [18:07:46] I guess it depends on what XioNoX intends to deploy to that target. [18:08:01] "it" being the value of deploy_user. [18:08:27] I think it's "netbox" [18:08:36] but also you want to deploy scap itself [18:08:39] first [18:09:03] lemme actually look at the defintion. :-) [18:09:36] meanwhile https://gerrit.wikimedia.org/r/878167 but fully expecting it to continue there [18:10:22] scap-on-scap is a special case, it doesn't work that simply [18:11:42] won't work anyways. "Duplicate declaration: File[/var/lib/scap] is already declared" [18:11:59] yep, acknowledeged :P [18:12:29] so the scap installation happens in modules/scap/manifests/init.pp (which is included via P::netbox -> netbox -> service::uwsgi -> scap::target -> scap) [18:13:30] mutante: so is /usr/bin/scap just a regular file and not a symlink? [18:14:16] taavi: it's a link /usr/bin/scap: symbolic link to /var/lib/scap/scap/bin/scap [18:14:33] ok, second question: is that a buster or bullseye box? [18:14:45] it's bullseye [18:15:49] ok, I suspect I know what's going on [18:16:46] the new scap-on-scap mechanism relies on installing scap and its dependencies into a venv, and then rsyncing that from deploy1002 to the deployment target hosts [18:17:54] the problem is that python virtual environments are tied to the python interpreter version that they're created on, so if you create a virtualenv on python 3.7 (buster) it's not going to work on 3.9 (bullseye), this is a very frequent problem we've seen on Toolforge [18:18:12] dancy: so you might need to rework the scap-on-scap mechanism to account for that [18:20:01] thanks for merging those experimental jobs for fundraising/REL1_39 James_F ! [18:22:56] mutante. Can we do a screen share? [18:24:53] btw, would this be our first bullseye target? [18:25:36] dancy: yes, but can't right now [18:34:25] taavi: That does make a lot of sense. [18:38:22] dancy: did you still want to screen share or .. we already know now? and thanks taavi for that [18:38:49] yeah, at least if someone can confirm that this is the first bullseye target. [18:39:10] I mean, yeah, taavi's explanation tracks. [18:39:27] tries to find that out.. brb [18:41:10] mutante: try this cumin query: `"P{C:scap} and P{F:lsbdistcodename = bullseye}"` [18:41:34] yea, I am already doing something similar [18:41:51] there are more hosts that are bullseye and are scap::target [18:42:04] doing with C:scap vs R:scap::target too [18:42:22] Let's try running scap on one of those [18:42:56] 27 hosts will be targeted: [18:42:56] an-tool[1005,1007,1011].eqiad.wmnet,build2001.codfw.wmnet,clouddumps[1001-1002].wikimedia.org,logstash[2023-2025,2030-2032].codfw.wmnet,logstash[1023-1025,1030-1032].eqiad.wmnet,netbox2002.codfw.wmnet,netbox1002.eqiad.wmnet,netbox-dev2002.codfw.wmnet,netmon[1003,2002].wikimedia.org,webperf[2003-2004].codfw.wmnet,webperf[1003-1004].eqiad.wmnet [18:43:20] [webperf1003:~] $ scap [18:43:25] ModuleNotFoundError: No module named 'scap' [18:43:33] same [18:43:50] OK. that settles it. I'll file a bug. Thanks taavi and mutante! [18:43:55] [cumin2002:~] $ sudo cumin "P{C:scap} and P{F:lsbdistcodename = bullseye}" 'scap version' [18:44:06] 100.0% (27/27) of nodes failed to execute command 'scap version': [18:44:21] thanks as well, both [19:06:49] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) [19:06:56] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) [19:07:02] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [19:09:28] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) [19:31:31] mutante: What's the output of `ls -l /var/lib/scap/scap/lib/` on that netmon host? [19:33:13] dancy: 4.0K drwxr-xr-x 3 scap scap 4.0K Jan 4 19:07 python3.7 [19:42:28] I ran `scap install-world` just for netbox-dev2002.codfw.wmnet. Any change in behavior? [19:47:16] dancy: yes, looks like you fixed it [19:47:18] [netbox-dev2002:~] $ scap [19:47:18] MUST provide subcommand, run with --help for a list [19:47:28] [netbox-dev2002:~] $ scap version [19:47:28] 4.32.0-1 [19:48:50] Interesting. So there were already hacks in `install-world` to deal w/ differing versions of python3.. but it didn't seem to have taken effect on that host (and apparently some others) until now. I don't understand that. I've deployed new versions of scap at least two times last week. [19:50:30] I'm back, is there a tl;dr? [19:50:39] or task? [19:50:49] Task is https://phabricator.wikimedia.org/T326668 [19:50:59] I'm adding some last notes now [19:51:05] but please try your deployment again. [19:51:18] I see! [19:53:10] thx for the investigation! [19:53:21] I'm getting an error but it might be because of my changes [19:53:28] deploy-local failed: {'exitcode': 1, 'stdout': '', 'stderr': "Submodule 'netbox' (http://deploy1002.eqiad.wmnet/netbox/deploy/.git/modules/netbox) registered for path 'src'\nCloning into '/srv/deployment/netbox/deploy-cache/revs/6542b6a216ab743c55f8d60723ce6487b36d6fae/src'...\nerror: Unable to find 331958629f13a63b4ef3f8889625fe361952a9e4 under http://deploy1002.eqiad.wmnet/netbox/deploy/.git/modules/netbox\nCannot [19:53:28] obtain needed object 331958629f13a63b4ef3f8889625fe361952a9e4\nerror: fetch failed.\nFetched in submodule path 'src', but it did not contain 331958629f13a63b4ef3f8889625fe361952a9e4. Direct fetching of that commit failed.\n"} [19:54:06] nod.. progress. [19:55:01] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) ` netbox-dev2002$ ls -l /var/lib/scap/scap/lib/ 4.0K drwxr-xr-x 3 scap scap 4.0K Jan 4 19:07 python3.7 ` `scap install-world` already has hacks in place to deal with different pytho... [19:58:26] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10dancy) `webperf1003.eqiad.wmnet` has this problem. Here's how to test and investigate: From deploy1002: ` $ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i/etc/keyholder.d/scap scap@webper... [20:03:05] fixed that one, now different error [20:05:58] dancy: could those errors be related to a new scap behavior https://phabricator.wikimedia.org/P42973 ? [20:06:23] those files exist [20:06:29] netbox-dev2002:~$ ls -al /srv/deployment/netbox/deploy/src/netbox/manage.py [20:06:29] -rwxr-xr-x 1 netbox netbox 249 Oct 13 11:25 /srv/deployment/netbox/deploy/src/netbox/manage.py [20:06:42] and we didn't have "WARNING: Ignoring invalid distribution" before [20:22:04] XioNoX: is any of this related to the cr eqsin incident, btw? [20:22:11] like is that why you want to deploy? [20:30:56] mutante: nop, different project [20:32:28] XioNoX: ACK, so eqsin stays depooled and the interface on eqsin router is still down, afaict. but we can't keep the incident open [20:32:39] or it will keep nagging and eventually p.age again [20:33:06] while I agree that "monitoring" is the right state.. technically [20:41:26] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10Volans) Python virtual environments should not be copied, they are inherently non-portable, even on the same system. See for example the warning box on the venv documentation: https://docs.p... [20:47:08] XioNoX: I'm not certain, but it seems possible. Volans just highlighted some big problems w/ the existing hack. I'll discuss alternate solutions with my team this week. [20:49:03] ok, thanks! [21:03:05] 10Phabricator: Change your username yourself on Phabricator - https://phabricator.wikimedia.org/T96110 (10BEEDELLROKEJULIANLOCKHART) @John_Cummings, yeah, https://phabricator.wikimedia.org/T96110#4872946 explains that all data associated with the previous username should be transferred to the new username. Is th... [21:05:46] 10Phabricator: Change your username yourself on Phabricator - https://phabricator.wikimedia.org/T96110 (10Aklapper) @BEEDELLROKEJULIANLOCKHART: Please bring up general questions on https://www.mediawiki.org/wiki/Talk:Phabricator/Help instead - thanks. [21:05:48] 10Phabricator: Change your username yourself on Phabricator - https://phabricator.wikimedia.org/T96110 (10Dzahn) Could you guys please not reuse a ticket from 2015 for a new request in 2022? I am sure it can be solved (unlike in 2015, that's the good news), but please make this a separate thing. Thank you! [21:14:57] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/27 [21:14:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:21:28] 10Project-Admins, 10Jornadas WMES: Create milestone for Wikimedia España for WMES meeting 2023 - https://phabricator.wikimedia.org/T326532 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! https://phabricator.wikimedia.org/project/view/6335/ [23:21:46] mutante: Are all of the afflicted scap nodes Linux/x86_64 ? [23:23:19] I would've thought so... [23:24:03] Nod. I will proceed on that assumption unless I hear otherwise. [23:25:36] dancy: yes, verified via cumin. all 27 are x86_64 [23:25:47] not that I expected any other systems.. but yea [23:25:47] Thx! [23:42:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10matmarex) [23:57:06] (03PS1) 10Krinkle: doc: Fix handling of /cover/ in router.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/878220 [23:57:08] (03PS1) 10Krinkle: doc: Shrink and align mtime/xml extra links on coverage pager [integration/docroot] - 10https://gerrit.wikimedia.org/r/878221 [23:57:22] (03CR) 10Krinkle: [C: 03+2] doc: Fix handling of /cover/ in router.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/878220 (owner: 10Krinkle) [23:57:27] (03CR) 10Krinkle: [C: 03+2] doc: Shrink and align mtime/xml extra links on coverage pager [integration/docroot] - 10https://gerrit.wikimedia.org/r/878221 (owner: 10Krinkle) [23:58:05] (03Merged) 10jenkins-bot: doc: Fix handling of /cover/ in router.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/878220 (owner: 10Krinkle) [23:58:24] (03Merged) 10jenkins-bot: doc: Shrink and align mtime/xml extra links on coverage pager [integration/docroot] - 10https://gerrit.wikimedia.org/r/878221 (owner: 10Krinkle)