[05:42:24] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) >>! In T285232#7443847, @Joe wrote: > Sadly I found a problem with our current approach: any fi... [06:57:39] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/StickyTOC] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/733078 (owner: 10Zoranzoki21) [07:00:11] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/StickyTOC] Enable CI [integration/config] - 10https://gerrit.wikimedia.org/r/733078 (owner: 10Zoranzoki21) [07:16:10] (03CR) 10Hashar: "I have deployed the change." [integration/config] - 10https://gerrit.wikimedia.org/r/733078 (owner: 10Zoranzoki21) [07:33:07] (03CR) 10Hashar: [C: 03+1] "Looks like a nice trick ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/733019 (owner: 10Ahmon Dancy) [07:47:07] (03PS1) 10Zoranzoki21: parameter_functions: fix dependencies for StickyTOC [integration/config] - 10https://gerrit.wikimedia.org/r/734199 [07:47:22] (03PS2) 10Zoranzoki21: parameter_functions: fix dependencies for StickyTOC [integration/config] - 10https://gerrit.wikimedia.org/r/734199 [07:47:27] (03PS3) 10Zoranzoki21: parameter_functions: Fix dependencies for StickyTOC [integration/config] - 10https://gerrit.wikimedia.org/r/734199 [07:48:23] (03CR) 10Hashar: [C: 03+2] parameter_functions: Fix dependencies for StickyTOC [integration/config] - 10https://gerrit.wikimedia.org/r/734199 (owner: 10Zoranzoki21) [07:50:08] (03Merged) 10jenkins-bot: parameter_functions: Fix dependencies for StickyTOC [integration/config] - 10https://gerrit.wikimedia.org/r/734199 (owner: 10Zoranzoki21) [07:58:01] (03PS1) 10Hashar: parameter_functions: Chameleon > chameleon [integration/config] - 10https://gerrit.wikimedia.org/r/734201 [07:58:18] (03CR) 10Hashar: "It is all lower case: https://gerrit.wikimedia.org/r/c/integration/config/+/734201 :)" [integration/config] - 10https://gerrit.wikimedia.org/r/734199 (owner: 10Zoranzoki21) [07:58:49] (03CR) 10Hashar: [C: 03+2] parameter_functions: Chameleon > chameleon [integration/config] - 10https://gerrit.wikimedia.org/r/734201 (owner: 10Hashar) [08:00:32] (03Merged) 10jenkins-bot: parameter_functions: Chameleon > chameleon [integration/config] - 10https://gerrit.wikimedia.org/r/734201 (owner: 10Hashar) [08:57:08] 10Release-Engineering-Team, 10serviceops: Puppet failure on deploy-1002.devtools.eqiad1.wikimedia.cloud due to missing profile::kubernetes::deployment_server::user_defaults - https://phabricator.wikimedia.org/T294174 (10hashar) #beta-cluster-infrastructure has it set via Horizon: ` profile::kubernetes::deploym... [09:14:30] 10Release-Engineering-Team (Doing), 10Security-Team, 10ContentSecurityPolicy, 10GitLab (Administration, Settings & Policy), and 3 others: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10hashar) The CSP is in report-only mode on both gitlab.wikimedia.org and gitlab... [09:38:09] (03PS1) 10Hashar: Release Quibble 1.2.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/734211 (https://phabricator.wikimedia.org/T259456) [09:38:13] (03PS1) 10Hashar: changelog: begin new 1.2.1 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/734212 [09:38:37] (03CR) 10Hashar: "Ran:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/734211 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [10:05:23] 10Release-Engineering-Team (Doing), 10Security-Team, 10ContentSecurityPolicy, 10GitLab (Administration, Settings & Policy), and 3 others: Define a Content Security Policy for GitLab - https://phabricator.wikimedia.org/T285363 (10hashar) [12:16:55] Anyone with github access mind making master default on https://github.com/wikimedia/mediawiki-skins-WMAU/tree/master [12:22:20] done [12:37:27] Reedy: also what did you see at screen x [13:26:07] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10hashar) I have created a jenkins-bot account on lists.wikimedia.org and stored its credentials in releng secrets store. Whenever Jenkins emits an em... [13:30:08] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10hashar) I tried to create an account on mailman for releng@lists.wikimedia.org and that results in a server side error, I guess cause it is a list :D [13:38:12] hashar: what are you trying to do? [13:38:42] Spookreeeno: about hte lists madness? [13:38:58] I would like to have email notifications send by Jenkins to be From: releng@lists.wikimedia.org [13:39:08] but bunch of lists require to be a member [13:39:21] ah [13:40:53] hashar: hmm [13:45:08] Spookreeeno: maybe I will create a ci@wikimedia.org list instead ;) [13:46:03] hashar: i still have no idea why it would fail [13:46:24] but i don't know mailman well [13:47:23] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I identified at least two issues which prevent us from having a successful restore: One is... [13:47:50] Spookreeeno: I guess you can't subscribe a list to another list [13:48:28] well you sort of can [13:48:59] because mediawiki gets mw-announce [13:49:04] and same with cloud [13:49:15] but they don't send as themselves [14:04:33] I will check with others later tonight, I just need emails from releng@lists.wikimedia.org to be accepted on a few lists [14:12:27] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10hashar) So in the end, we need `releng@lists.wikimedia.org` to be allowed to post on the list, which in mailman 3 can be granted even if the email is... [14:22:34] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) >>! In T283076#7454868, @Jelto wrote: > So we have to make sure GitLab is not started by pup... [14:44:52] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Papaul) @Dzahn I need mw2253 and contint2001 down for me to reset the IDRAC befor... [14:50:04] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10hashar) [14:51:40] 10Gerrit, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work): Update gerrit submit type for discovery repositories in gerrit - https://phabricator.wikimedia.org/T255509 (10Gehel) 05Open→03Resolved [14:52:30] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul mw2253 is not a problem. done. it's shut down and downtimed. cont... [15:07:18] (03CR) 10Ahmon Dancy: [C: 03+2] beta-build-scap-deb/beta-publish-deb mods to aid automation [integration/config] - 10https://gerrit.wikimedia.org/r/733019 (owner: 10Ahmon Dancy) [15:09:10] (03Merged) 10jenkins-bot: beta-build-scap-deb/beta-publish-deb mods to aid automation [integration/config] - 10https://gerrit.wikimedia.org/r/733019 (owner: 10Ahmon Dancy) [15:23:16] (03CR) 10Ahmon Dancy: [C: 03+2] Mirror mediawiki/extensions/WikiLambda [tools/train-dev] - 10https://gerrit.wikimedia.org/r/733090 (owner: 10Dduvall) [15:23:47] (03Merged) 10jenkins-bot: Mirror mediawiki/extensions/WikiLambda [tools/train-dev] - 10https://gerrit.wikimedia.org/r/733090 (owner: 10Dduvall) [15:26:02] 10Release-Engineering-Team (Doing), 10GitLab (Project Migration), 10User-brennen: Early adoption signup for WMF GitLab - https://phabricator.wikimedia.org/T282842 (10BTullis) The #data-engineering team would like to use GitLab for a new requirement around Airflow. I have added some details of the requirement... [15:47:18] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10Zabe) [15:47:35] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10Zabe) [15:55:21] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10Reedy) Noting this CVE is Windows only; https://github.com/composer/composer/commit/ca5e2f8d505fd3bfac6f7c85b82f2740becbc0aa So it's probably low-ish... [15:58:55] 10Continuous-Integration-Infrastructure, 10Composer: Upgrade dockerfiles to use composer 2.1.9 per CVE-2021-41116 - https://phabricator.wikimedia.org/T294260 (10Reedy) p:05Triage→03Low [16:04:32] 06:47:50 <+hashar> Spookreeeno: I guess you can't subscribe a list to another list <-- yes you can :) [16:04:51] legoktm: good morning ;) [16:05:00] morning! [16:05:09] * legoktm is reading the ticket now [16:05:29] I tried creating an acocunt with releng@lists.wikimedia.org but the account gives me a server side error, I will fill it eventually [16:05:55] why do you need an account for that? [16:05:57] the idea is that Jenkins jobs send notifications using jenkins-bot@wikimedia.org which I wanna phase out in favor of our list email [16:06:26] so I wanted a mailman account for releng@ in order to become a member of the destination lists in order to have message accepted by those lists [16:06:30] (and disable email delivery ) [16:06:41] uh, don't create an account for the list, that won't work [16:07:02] you can just allow that email address to be accepted [16:07:58] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Majavah) [16:08:00] yup I found out that the email is listed as a non member and I can get the list owners to have the list accept those [16:08:12] will send some emails to those owners tomorrow ;) [16:09:14] that works, also "Message Acceptance" -> "Accept these non-members" [16:09:56] (03PS1) 10Dduvall: Mirror mediawiki/services/function-schemata [tools/train-dev] - 10https://gerrit.wikimedia.org/r/734340 [16:10:05] hashar: I can also change all those mailing lists for you if you want (using superuser permissions) [16:10:55] after reading the ticket, I don't understand why the From: part of the message is being changed [16:11:03] legoktm: that would be awesome! The idea I had was to go to each of the list in the table at https://phabricator.wikimedia.org/T151642 , browse to the non member list, find releng@lists.wikimedia.org and change its moderation to "accept" [16:11:14] or make the email a member of the list and disable email delivery [16:11:30] the aim is to get rid of jenkins-bot@wikimedia.org [16:11:40] ahh, and to handle bounces [16:11:41] got it [16:12:39] (03PS5) 10Dduvall: Simplify hack/test cycles by bind mounting project clones [tools/train-dev] - 10https://gerrit.wikimedia.org/r/732773 [16:13:34] and the reason is jenkins-bot is a google group with just me and tyler in it, which is not convenient. We felt like spamming the whole team would be better ;-] [16:15:49] hashar: {{done}} [16:15:59] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10Legoktm) >>! In T151642#7454987, @hashar wrote: > So in the end, we need `releng@lists.wikimedia.org` to be allowed to post on the list, which in mai... [16:16:45] legoktm: awesome!! It is way faster with the super user being around! [16:16:57] :)) [16:17:00] should I file a task to delete the "releng" mailman user I created earlier? [16:17:25] !log launching new runner-1006 instance under gitlab-runners [16:17:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:18:02] hashar: I don't think I have a button to delete the account, but if you log into it, you should be able to at https://lists.wikimedia.org/user-profile/delete [16:18:44] legoktm: don't worry I will file a task for later (account is broken due to some server side error which is a rabbit hole) [16:18:48] ack [16:18:58] and given there is no use for that account, it can wait ;) [16:19:53] 10Release-Engineering-Team (Doing): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10hashar) Thank you so much @Legoktm for proposing your help and fixing the rights. I will switch Jenkins to emit emails from releng@ tomorrow \o/ [16:32:18] 10Release-Engineering-Team (Done by Thu 04 Nov), 10GitLab (CI & Job Runners): Provide separate/larger volume for /var/lib/docker on GitLab runners - https://phabricator.wikimedia.org/T293835 (10dduvall) 05Stalled→03In progress Moving forward with runner re-provisioning. [16:32:31] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10CAS-SSO, 10GitLab (Auth & Access): Attempting to login to gitlab.wikimedia.org sometimes results in CAS 500 Internal Server Error - https://phabricator.wikimedia.org/T291964 (10jbond) Hi all i have updated idp.wikimedia.org today, could yo... [16:43:21] (03CR) 10Dduvall: [C: 03+2] Mirror mediawiki/services/function-schemata [tools/train-dev] - 10https://gerrit.wikimedia.org/r/734340 (owner: 10Dduvall) [16:44:06] (03Merged) 10jenkins-bot: Mirror mediawiki/services/function-schemata [tools/train-dev] - 10https://gerrit.wikimedia.org/r/734340 (owner: 10Dduvall) [16:49:54] hashar: we need to take down contint2001 at some point, for a DRAC firmware upgrade (will fix those flapping alerts on contint2001.mgmt). Should I make a ticket or mail to releng list to schedule downtime? or is it simpler than that because codfw-only [16:50:15] wasnt sure which route is actually more convenient for you as well [16:51:23] or I can just use releng tag on phab and avoid pinging a person, but here I did anyways, ironically [16:54:28] mutante: contint2001 is the primary for Jenkins CI. We could switch over but the runbook as some issues related to file permissions :-\ [16:54:50] hashar: oooh, of course, duh. yea, making ticket already :) [16:55:26] we should be making it way faster by simply dishing out the build history (but still keep the last build number) [16:57:22] https://phabricator.wikimedia.org/T294271 WIP [16:57:29] incorporated your reply [16:57:48] mutante: iirc the issue is the quick rsync puppet class uses a chroot and thus rsync is unable to do the uid / name translation. So we end up with crazy uid on the destination host [16:58:53] hm, I kind of remember that issue from the past, yes. but afaict that issue doesn't happen on other hosts where the UID is in sync on both hosts [16:59:11] or there might have been fixes to rsync class from the past [16:59:37] did we get the same UID on cont1001 and contint2001 at some point and adjusted it? [16:59:49] would have to look that up, but yea [17:00:13] we can make some tests first [17:01:06] 10Release-Engineering-Team, 10SRE, 10serviceops: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10Dzahn) [17:01:07] !log deleting bullseye runner-1006 and reverting to using buster due to some puppet provisioning issues [17:01:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:01:28] 10Release-Engineering-Team, 10SRE, 10serviceops: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10Dzahn) [17:01:38] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) [17:01:56] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Zabe) [17:02:47] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul Let's go ahead with mw2253. For contint2001 please consider it sta... [17:03:10] Project beta-scap-sync-world build #25079: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/25079/ [17:03:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10hashar) contint2001.wikimedia.org is indeed the primary for CI (Jenkins and Zuul). We could switch over to the other host but the runboo... [17:04:09] hashar: I am thinking for this case it is maybe easier to really just schedule downtime of CI for some minutes, in coordination with dcops-codfw and NOT actually switch it over [17:04:43] mutante: for the rsync/uid mixup I can't remember off hand but the task might have details [17:04:51] same here :) [17:04:55] yeah probably easier to just shutdown [17:05:01] well, but we fixed it in other cases [17:05:12] upgrade + reboot + fsck should not take that long [17:05:12] so "should work" (tm) [17:05:24] but no point in doing the whole switch for this [17:05:29] unless you want to test the procedure [17:05:29] with the risk that if the machine is bricked somehow, we gotta restore contint2001 on contint1001 and do a bunch of puppet changes [17:06:00] I would say for _this_ one just schedule that CI will be down. [17:06:10] while all the other comments about making it easier are of course true [17:06:20] and dont have to _not_ be done in addition [17:06:41] ah yeah https://www.mediawiki.org/wiki/Continuous_integration/Data_center_switch#rsync_data_and_states [17:06:47] which says "rsync over ssh as root" [17:07:08] and does not use the quickdata copy puppet class that set up a rsync daemon [17:07:38] there are plenty of other services using quickdatacopy that somehow dont run into the issue, though [17:07:41] guess I can make that reasonably faster by discarding old builds and only keep the file that tracks the last build # [17:07:44] but we have been there before, ack [17:07:48] yup [17:07:55] so depends on the risk we face when upgrading the DRAC [17:08:14] if it is know to always work fine I guess we don't have much risks [17:08:20] and tha tmakes the operation straightforward [17:08:24] i think it's just the risk of a hard reboot of the server [17:08:49] where "just" changes over time when servers get old [17:08:55] * mutante looksup in netbox [17:08:57] :]]] [17:09:14] the DRAC requires someone to be in the datacenter or can it be done remotely? [17:09:22] eh yea,, it's old [17:09:33] out of warranty and needs replacement [17:09:37] oh [17:10:13] I guess there is no hardware refresh ticket for this [17:10:20] because there were plans to move CI [17:10:44] guess we can look at replacing both if they are outdated. Last time I think we borrowed from a pool of spare machiens we had [17:10:50] I think they used to be elasticsearch servers [17:11:01] I think "pool of spares" isnt a thing anymore today [17:11:15] we would obey to whatever is the new standard :] [17:11:15] we need to ask dcops though [17:11:27] first there needs to be a request to replace them [17:11:50] so yea.. we could also say "dont touch this please" [17:12:01] and just downtime the Icinga mgmt alert, for contint2001.mgmt [17:12:11] and then work instead on replaceing the entire hardware [17:12:13] pretty sure we will have to refresh it, cause Jenkins is going to stay around for quite a while still [17:12:18] instead of fixing this minor annoyance [17:12:25] too many things run on it for us to phase it out in a short term [17:12:38] that's how it usually is, yea [17:12:41] yup [17:12:44] so downtime [17:12:50] drac + reboot + fsck [17:13:00] and that sounds the easiest path [17:13:15] well, unless we just say "dont fix it" [17:13:37] after all it's just like a log spam line [17:13:46] and we can say "let's replace it anyways' [17:13:52] and start that whole hw replacement ticket instead [17:14:11] well the drac upgrade seems a quick win and would get rid of that flappy alarm [17:14:15] and until that hardware arrives, the actual switch-over can be made easier [17:14:17] and we can do next a hardware refresh [17:14:27] ok, that is fine as well, alright [17:14:42] then let's just tell Papaul what time is ok, basically [17:15:00] or a time where all can be around with him in DC [17:15:08] anything in his morning, and we gotta pick a time that fit in https://wikitech.wikimedia.org/wiki/Deployments [17:15:17] *nod* [17:15:18] but maybe we have a no deploy week coming . thcipriani would know [17:16:09] that sounds good, lets do the details on the ticket then, +1 [17:16:19] then in the morning (for Texas) CI should be rather quiet and a downtime will have little impact [17:16:32] great, yep [17:16:35] feel free to copy, I am going to have dinner :] [17:16:41] ok, enjoy [17:16:49] thank you mutante ! [17:17:39] everything I know is (I hope) on https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar [17:19:45] Project beta-scap-sync-world build #25080: 04STILL FAILING in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/25080/ [17:21:49] 10Release-Engineering-Team, 10serviceops: contint2001 hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [17:22:12] 10Release-Engineering-Team, 10serviceops: contint2001 hardware refresh? - https://phabricator.wikimedia.org/T294276 (10Dzahn) [17:22:52] mutante: 2nd or 11th Nov is the next no deploy days [17:22:58] 2nd is Election Day though [17:23:17] There's eng prod offsite but no idea when that is [17:23:30] It will be next few weeks [17:25:16] added that about Nov days, thanks [17:25:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10Dzahn) < mutante> then let's just tell @Papaul what time is ok, basically < mutante> or a time where all can be around with him in DC <+... [17:27:39] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Papaul) [17:27:53] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Papaul) [17:28:16] Project beta-scap-sync-world build #25081: 04STILL FAILING in 6 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/25081/ [17:28:33] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Papaul) @Dzahn mw2253 done [17:28:44] The beta-scap-sync-world error is: [17:28:44] `sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-deploy01.deployment-prep.eqiad1.wikimedia.cloud::common/wikiversions*.{json,php} /srv/mediawiki (ran as mwdeploy@deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud) returned [255]: Permission denied (publickey).` [17:33:10] Yippee, build fixed! [17:33:11] Project beta-scap-sync-world build #25082: 09FIXED in 3 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/25082/ [17:41:39] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul. Thank you! - scap pulled - confirmed icinga green - repooled to... [17:59:44] (03CR) 10Legoktm: [DNM] Docker: [php74] Switch PHP 7.4 from Sury to Wikimedia package (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/732112 (https://phabricator.wikimedia.org/T293851) (owner: 10Jforrester) [18:13:25] (03PS1) 10Reedy: Temporarily remove Wikibase et al from Score dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/734370 (https://phabricator.wikimedia.org/T294238) [18:14:00] (03CR) 10Reedy: [C: 03+2] Temporarily remove Wikibase et al from Score dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/734370 (https://phabricator.wikimedia.org/T294238) (owner: 10Reedy) [18:16:03] (03Merged) 10jenkins-bot: Temporarily remove Wikibase et al from Score dependancies [integration/config] - 10https://gerrit.wikimedia.org/r/734370 (https://phabricator.wikimedia.org/T294238) (owner: 10Reedy) [18:16:34] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734370 https://phabricator.wikimedia.org/T294238 [18:16:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:24:28] (03CR) 10Ahmon Dancy: [C: 03+2] "This is a great improvement. Thank you!" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/732773 (owner: 10Dduvall) [18:24:53] (03Merged) 10jenkins-bot: Simplify hack/test cycles by bind mounting project clones [tools/train-dev] - 10https://gerrit.wikimedia.org/r/732773 (owner: 10Dduvall) [18:31:57] lol [18:32:02] WBMI is still in deps [18:32:52] I'm also seeing a few of these [18:32:52] 16:09:34 Syncing... [18:32:53] 16:09:34 rsync: change_dir "/castor-mw-ext-and-skins/REL1_35/mwgate-node12-docker" (in caches) failed: No such file or directory (2) [18:32:53] 16:09:34 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [Receiver=3.1.2] [18:32:53] 16:09:34 rsync: read error: Connection reset by peer (104) [18:40:58] (03PS1) 10Reedy: Also remove WikibaseCirrusSearch from score/REL1_35 [integration/config] - 10https://gerrit.wikimedia.org/r/734377 (https://phabricator.wikimedia.org/T294238) [18:41:39] (03CR) 10Reedy: [C: 03+2] Also remove WikibaseCirrusSearch from score/REL1_35 [integration/config] - 10https://gerrit.wikimedia.org/r/734377 (https://phabricator.wikimedia.org/T294238) (owner: 10Reedy) [18:43:28] (03Merged) 10jenkins-bot: Also remove WikibaseCirrusSearch from score/REL1_35 [integration/config] - 10https://gerrit.wikimedia.org/r/734377 (https://phabricator.wikimedia.org/T294238) (owner: 10Reedy) [18:45:44] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734377 T294238 [18:45:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:45:47] T294238: ApiUsageException: The file you submitted was empty. - https://phabricator.wikimedia.org/T294238 [18:51:30] Reedy: those rsync / castor are not an error. No cache have been stored for REL1_35, I might have nuked it recently [18:52:49] the jobs fail though [18:53:25] Reedy: any errors should be ignored, even if it explodes completely [18:53:36] must be failing for some other reason [18:54:14] 00:00:02.206 mkdir: cannot create directory ‘src’: Permission denied [18:54:20] https://integration.wikimedia.org/ci/job/mwgate-node12-docker/50718/console [18:55:22] performance on appservers just went down a bit [18:55:39] from 210 to 240ms or so [18:56:24] Reedy: I blame it on cosmic rays, that one does not make any sense to me :-\\ [18:58:02] no idea why Score ends up depending on mediawiki/extensions/WikibaseCirrusSearch without having Wikibase included [18:58:14] It's happened a few times [18:58:24] hashar: Because I didn't remove it too [18:59:40] ;] [19:00:13] In 1.31 WikibaseCirrusSearch didn't exist at all... So it was a seperate entry [19:19:11] (03PS5) 10Jforrester: Docker: [php74] Switch PHP 7.4 from Sury to Wikimedia package [integration/config] - 10https://gerrit.wikimedia.org/r/732112 (https://phabricator.wikimedia.org/T293851) [19:19:16] (03CR) 10Jforrester: [C: 03+2] Docker: [php74] Switch PHP 7.4 from Sury to Wikimedia package [integration/config] - 10https://gerrit.wikimedia.org/r/732112 (https://phabricator.wikimedia.org/T293851) (owner: 10Jforrester) [19:20:07] James_F: \o/ Chief CI Updates Officer, thx! [19:20:14] :-) [19:20:22] All praise to legoktm for packaging it. [19:20:42] :))) [19:21:00] (03Merged) 10jenkins-bot: Docker: [php74] Switch PHP 7.4 from Sury to Wikimedia package [integration/config] - 10https://gerrit.wikimedia.org/r/732112 (https://phabricator.wikimedia.org/T293851) (owner: 10Jforrester) [19:21:08] yay [19:21:43] !log Docker: Publishing new php74 and cascaded images with PHP 7.4 from Wikimedia package T293851 [19:21:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:21:46] T293851: Re-build all CI images of PHP 7.4 from sury's package to Wikimedia's one, to assure us that it will work - https://phabricator.wikimedia.org/T293851 [19:22:56] We should really get the php-compile images working again – see T276417 :-( [19:22:57] T276417: CI should validate that the pecl tarball contains all the necessary files to build the extension - https://phabricator.wikimedia.org/T276417 [19:23:57] :S [19:24:10] I'll just revert that after lunch [19:24:26] *hugs* [19:28:32] Oh, bother. [19:28:47] The images built locally but fail on the server. [19:28:57] (03PS1) 10Jforrester: jjb: Switch to PHP 7.4 images with Wikimedia not Sury package [integration/config] - 10https://gerrit.wikimedia.org/r/734382 (https://phabricator.wikimedia.org/T293851) [19:33:38] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) [19:33:57] 10Release-Engineering-Team, 10Release Pipeline (Blubber): "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10nnikkhoui) [19:34:26] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @Papaul Afraid this is a long story. just saw `mw2255.mgmt` alerting in Ic... [19:34:41] * James_F coughs. [19:34:41] (03PS1) 10Jforrester: Follow-up 1b6a0eda4: Actually commit my local changes, whoops [integration/config] - 10https://gerrit.wikimedia.org/r/734384 [19:34:47] (03CR) 10Jforrester: [C: 03+2] Follow-up 1b6a0eda4: Actually commit my local changes, whoops [integration/config] - 10https://gerrit.wikimedia.org/r/734384 (owner: 10Jforrester) [19:36:41] (03Merged) 10jenkins-bot: Follow-up 1b6a0eda4: Actually commit my local changes, whoops [integration/config] - 10https://gerrit.wikimedia.org/r/734384 (owner: 10Jforrester) [19:47:55] heh [19:48:13] And another failure. [19:48:15] * James_F sighs. [19:50:17] We're depending on php-imagick which doesn't have a php7.4 build. [19:51:21] https://phabricator.wikimedia.org/T200666 [19:51:31] this happened last time [19:51:39] Yeah. [19:51:42] cc: legoktm [19:51:51] legoktm: Can I whine at you or should I just skip it for now? [19:51:55] should we do the same again? [19:52:06] and add php-imagick [19:52:15] (I'm not sure why the mediawiki-phan images use php-imagick TBH.) [19:52:34] I can add it after lunch [19:54:00] (03PS1) 10Jforrester: Docker: [mediawiki-phan-php74] Switch php-apcu to php7.4-apcu and disable php-imagick [integration/config] - 10https://gerrit.wikimedia.org/r/734387 [19:54:14] For now I'll just drop it. [19:54:17] (03CR) 10Jforrester: [C: 03+2] Docker: [mediawiki-phan-php74] Switch php-apcu to php7.4-apcu and disable php-imagick [integration/config] - 10https://gerrit.wikimedia.org/r/734387 (owner: 10Jforrester) [19:55:48] (03CR) 10jerkins-bot: [V: 04-1] Docker: [mediawiki-phan-php74] Switch php-apcu to php7.4-apcu and disable php-imagick [integration/config] - 10https://gerrit.wikimedia.org/r/734387 (owner: 10Jforrester) [19:57:28] (03PS2) 10Jforrester: Docker: [mediawiki-phan-php74] Switch to php7.4-apcu, disable php-imagick [integration/config] - 10https://gerrit.wikimedia.org/r/734387 [20:02:59] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/734387 (owner: 10Jforrester) [20:03:38] (03PS1) 10Jforrester: Docker: [php80] Enable apcu, ast, imagick, and redis extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734388 [20:05:23] (03Merged) 10jenkins-bot: Docker: [mediawiki-phan-php74] Switch to php7.4-apcu, disable php-imagick [integration/config] - 10https://gerrit.wikimedia.org/r/734387 (owner: 10Jforrester) [20:12:43] (03PS2) 10Jforrester: jjb: Switch to PHP 7.4 images with Wikimedia not Sury package [integration/config] - 10https://gerrit.wikimedia.org/r/734382 (https://phabricator.wikimedia.org/T293851) [20:12:59] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/734382 (https://phabricator.wikimedia.org/T293851) (owner: 10Jforrester) [20:13:15] 10Continuous-Integration-Config: Also run PHP 7.4 jobs on wmf branch patches - https://phabricator.wikimedia.org/T293924 (10Jdforrester-WMF) [20:13:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Re-build all CI images of PHP 7.4 from sury's package to Wikimedia's one, to assure us that it will work - https://phabricator.wikimedia.org/T293851 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [20:14:56] (03Merged) 10jenkins-bot: jjb: Switch to PHP 7.4 images with Wikimedia not Sury package [integration/config] - 10https://gerrit.wikimedia.org/r/734382 (https://phabricator.wikimedia.org/T293851) (owner: 10Jforrester) [20:15:54] James_F: uploaded php-imagick [20:16:41] legoktm: ε> [20:20:13] (03PS1) 10Jforrester: Docker: [mediawiki-phan-php74] Enable php7.4-imagick, now it's available [integration/config] - 10https://gerrit.wikimedia.org/r/734394 [20:20:25] (03CR) 10Jforrester: [C: 03+2] Docker: [mediawiki-phan-php74] Enable php7.4-imagick, now it's available [integration/config] - 10https://gerrit.wikimedia.org/r/734394 (owner: 10Jforrester) [20:22:20] (03Merged) 10jenkins-bot: Docker: [mediawiki-phan-php74] Enable php7.4-imagick, now it's available [integration/config] - 10https://gerrit.wikimedia.org/r/734394 (owner: 10Jforrester) [20:24:03] What actually injects THING_SUBNAME into docker-setup-mwext-for-phan? It's failing for mwext-php74-phan-docker (but passing for mwext-php72-phan-docker)? [20:24:49] Ah, set_mw_dependencies() [20:25:02] Oh duh. [20:27:29] (03CR) 10Ahmon Dancy: WIP: mw is deployable to k8s in traindev (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:27:33] (03PS1) 10Legoktm: Revert "Have php*-compile images test pecl installation" [integration/config] - 10https://gerrit.wikimedia.org/r/734395 [20:28:04] mw_deps_jobs_starting_with = ( [20:28:04] 'mwext-php72-phan', [20:28:04] 'mwskin-php72-phan', [20:28:12] Indeed. [20:28:19] Oh, I didn't push. :-) [20:28:19] (03PS1) 10Jforrester: Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 [integration/config] - 10https://gerrit.wikimedia.org/r/734397 [20:28:50] (03CR) 10Legoktm: [C: 03+1] Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 [integration/config] - 10https://gerrit.wikimedia.org/r/734397 (owner: 10Jforrester) [20:28:55] Oh, it'll fail. [20:29:03] Fsking commit message validator. [20:29:05] I might kill it… [20:29:40] I put up a patch for the php-*-compile jobs: https://gerrit.wikimedia.org/r/c/integration/config/+/734395/ [20:29:53] I saw. Let me get php74-phan working first, then I [20:29:56] 'll review. [20:30:23] sneaks in an easteregg that makes commit-message validator always complain if anyone tries to remove it [20:30:25] (03CR) 10Jforrester: [C: 03+2] Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 [integration/config] - 10https://gerrit.wikimedia.org/r/734397 (owner: 10Jforrester) [20:30:30] mutante: :-) [20:32:07] (03Merged) 10jenkins-bot: Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 [integration/config] - 10https://gerrit.wikimedia.org/r/734397 (owner: 10Jforrester) [20:32:20] !log Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 [20:32:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:39:36] (03PS1) 10Jforrester: jjb: Make and use a php74 version of the phan setup builder [integration/config] - 10https://gerrit.wikimedia.org/r/734398 [20:41:20] Woo-hoo, https://integration.wikimedia.org/ci/job/mwext-php74-phan-docker/6/console actually passes. [20:43:42] whee [20:43:57] One sec. [20:46:36] (03PS2) 10Jforrester: jjb: Make and use php versions of the phan setup builder [integration/config] - 10https://gerrit.wikimedia.org/r/734398 [20:46:48] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/734398 (owner: 10Jforrester) [20:47:16] (03CR) 10Jforrester: [C: 03+2] Revert "Have php*-compile images test pecl installation" [integration/config] - 10https://gerrit.wikimedia.org/r/734395 (owner: 10Legoktm) [20:47:47] (03PS2) 10Jforrester: Revert "Have php*-compile images test pecl installation" [integration/config] - 10https://gerrit.wikimedia.org/r/734395 (https://phabricator.wikimedia.org/T276417) (owner: 10Legoktm) [20:47:57] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/734395 (https://phabricator.wikimedia.org/T276417) (owner: 10Legoktm) [20:48:36] (03Merged) 10jenkins-bot: jjb: Make and use php versions of the phan setup builder [integration/config] - 10https://gerrit.wikimedia.org/r/734398 (owner: 10Jforrester) [20:49:46] (03Merged) 10jenkins-bot: Revert "Have php*-compile images test pecl installation" [integration/config] - 10https://gerrit.wikimedia.org/r/734395 (https://phabricator.wikimedia.org/T276417) (owner: 10Legoktm) [20:50:30] !log Docker: Publishing php*-comile images without the PECL test so they work again. [20:50:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:22] (03PS2) 10Jforrester: Docker: [php80] Enable apcu, ast, imagick, and redis extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734388 [20:54:05] 10Continuous-Integration-Infrastructure: Re-build CI images with a PHP version with a fix for CVE-2021-21703 once that's in Wikimedia APT - https://phabricator.wikimedia.org/T294304 (10Jdforrester-WMF) [20:54:36] (03CR) 10Jforrester: [C: 03+2] Docker: [php80] Enable apcu, ast, imagick, and redis extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734388 (owner: 10Jforrester) [20:56:27] (03Merged) 10jenkins-bot: Docker: [php80] Enable apcu, ast, imagick, and redis extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734388 (owner: 10Jforrester) [20:58:36] (03PS1) 10Jforrester: jjb: Update php-compile images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/734403 [21:12:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops: schedule downtime for contint2001 - https://phabricator.wikimedia.org/T294271 (10Dzahn) also see T256422 - switch contint prod server back from contint2001 to contint1001 [21:18:27] (03PS1) 10Jforrester: jjb: Update jobs to use php80 images with various PHP extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734407 [21:19:15] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/734403 (owner: 10Jforrester) [21:20:55] (03Merged) 10jenkins-bot: jjb: Update php-compile images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/734403 (owner: 10Jforrester) [21:25:53] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/734407 (owner: 10Jforrester) [21:26:35] (03Abandoned) 10Jforrester: jjb: Bump jobs using php*-compile images [integration/config] - 10https://gerrit.wikimedia.org/r/668335 (owner: 10Legoktm) [21:28:11] (03Merged) 10jenkins-bot: jjb: Update jobs to use php80 images with various PHP extensions [integration/config] - 10https://gerrit.wikimedia.org/r/734407 (owner: 10Jforrester) [21:28:40] (03PS1) 10Jforrester: jjb: Bump users of tox-pywikibot image to 0.7.0-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/734409 [21:29:58] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/734409 (owner: 10Jforrester) [21:32:22] (03Merged) 10jenkins-bot: jjb: Bump users of tox-pywikibot image to 0.7.0-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/734409 (owner: 10Jforrester) [21:39:46] (03PS1) 10Jforrester: jjb: Update caster users from 0.2.1 (2018) to 0.2.4 (2019) [integration/config] - 10https://gerrit.wikimedia.org/r/734412 [21:40:50] (03CR) 10Jforrester: "Antoine, I'm not sure why this wasn't updated earlier other than it being a big change; do you know?" [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (owner: 10Jforrester) [21:48:33] (03PS1) 10Jforrester: Zuul: Don't define wmf branch jobs for extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/734415 [21:48:35] (03PS1) 10Jforrester: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) [21:49:36] (03CR) 10Jforrester: [C: 03+2] Zuul: Don't define wmf branch jobs for extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/734415 (owner: 10Jforrester) [21:50:36] (03CR) 10jerkins-bot: [V: 04-1] Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) (owner: 10Jforrester) [21:51:20] (03Merged) 10jenkins-bot: Zuul: Don't define wmf branch jobs for extension-quibble-composer-noselenium [integration/config] - 10https://gerrit.wikimedia.org/r/734415 (owner: 10Jforrester) [21:57:35] (03PS2) 10Jforrester: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) [21:57:38] (03PS1) 10Jforrester: jjb: Provide PHP 7.4 versions of wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/734417 [21:59:09] (03CR) 10jerkins-bot: [V: 04-1] jjb: Provide PHP 7.4 versions of wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/734417 (owner: 10Jforrester) [21:59:15] (03CR) 10jerkins-bot: [V: 04-1] Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) (owner: 10Jforrester) [22:39:02] (03PS8) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [22:39:28] (03CR) 10jerkins-bot: [V: 04-1] mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:42:53] (03PS9) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [22:45:24] (03CR) 10Jeena Huneidi: "We could move the values-traindev.yaml to the deployment-charts repo in another patchset (so as not to hold this up) if desired." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:47:14] (03CR) 10Jeena Huneidi: mw is deployable to k8s in traindev (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:54:00] 10Continuous-Integration-Infrastructure: Re-build CI images with a PHP version with a fix for CVE-2021-21703 once that's in Wikimedia APT - https://phabricator.wikimedia.org/T294304 (10Legoktm) Should be uploaded now: https://sal.toolforge.org/log/p0KouXwBa_6PSCT9gTC9 [23:27:23] !log fully provisioned runner-{1008,1011,1012,1013,1014,1015,1016,1017,1018,1019} instances for use as new gitlab runners and removed old instances (T293835) [23:27:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:27:26] T293835: Provide separate/larger volume for /var/lib/docker on GitLab runners - https://phabricator.wikimedia.org/T293835 [23:28:34] addshore: not for you [23:36:23] 10Release-Engineering-Team (Done by Thu 04 Nov), 10GitLab (CI & Job Runners): Provide separate/larger volume for /var/lib/docker on GitLab runners - https://phabricator.wikimedia.org/T293835 (10dduvall) 05In progress→03Resolved We now have 10 `g3.cores8.ram24.disk20.ephemeral40.4xiops` instances and each r... [23:40:50] (03PS10) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [23:41:08] (03CR) 10Jeena Huneidi: mw is deployable to k8s in traindev (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi)