[00:42:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10RLazarus) 05Stalled→03Open [00:42:45] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10RLazarus) Over to dcops! @Jclark-ctr Sorry to hand this off right as you're gone, but whenever you're back in the DC, these servers are offline and ready to be physically removed. [09:42:16] I'd like to have that https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/857672/ merged and built for this afternoon so we can work on the chart with j.ayme [09:42:55] I figured we can bring that in with rebuilding the PHP images and do a single rebuild session [09:42:57] thoughts? [09:56:12] <_joe_> ok [09:57:28] <_joe_> so, what is the new php package version? [09:59:10] It was effie on it, gimme a second to check [09:59:29] I also don't want to steal her work :D [09:59:39] <_joe_> anyways, the way to ensure you rebuild all the images in the dependency tree is to use docker-pkg update [09:59:43] <_joe_> as an example [09:59:47] I was proposing a litteral team-building session, just we built images :D [09:59:52] build* [09:59:55] <_joe_> docker-pkg -c config.yaml update --reason "New package version" --version "7.4.30-2" php7.4-cli images [10:01:05] That'll bump all dependant images right? [10:03:43] <_joe_> yes [10:03:53] <_joe_> you might want to edit their version in the changelog [10:04:07] <_joe_> because it assumes it's a security upgrade [10:22:45] the neww/fixed version is 7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u1 [10:23:39] or trim it down to 7.4.33-1+sec1 or so [10:24:06] 7.4.x is EOL upstream, so all further security fixes are going to be backports until we move to 8.x [10:31:19] <_joe_> moritzm: oh right 7.3 is in bullseye, not 7.4 [10:32:43] <_joe_> uh actually I see 7.4 there [10:34:31] <_joe_> moritzm: so I guess we should probably first move to bullseye, so icu upgrade, go back to using the shipped php packages? [10:34:46] <_joe_> before going to 8.1 I mean [10:40:25] yeah, that sounds like a good plan. I'd be happy to take care of the ICU-enabled backports again [10:43:13] <_joe_> oh this time we have the ability to fill the databases before we do the transition, too [10:45:33] even better :-) [10:45:53] Point of vocabulary, what do you mean by ICU upgrade? [10:46:23] it's the Unicode library used by PHP [10:46:29] and other parts of the stack [10:46:57] new ICU means newer versions of Unicode are supported and there's often changes to existing Unicode tables as well [10:47:43] the churn is mostly related to collation changes [10:48:13] and those changes can cause user-visible effects, e.g. in the way categories are sorted [10:48:16] <_joe_> moritzm: so when you have the new-icu packages built, we should add those to a different apt component [10:48:30] <_joe_> and use them to build the shellbox containers [10:48:45] _joe_: ack, IIRC that's how we did it for the passt ICU migrations [10:48:52] <_joe_> then we need a new shellbox deployment where we'll calculate the new collations [10:49:06] <_joe_> for some time, then we can transition ICU version [10:49:24] ack, sounds like a plan [10:49:25] <_joe_> this would all be easier if everything was in containers :P [10:49:43] <_joe_> actually, I can just build a bullseye php container right now [10:49:50] <_joe_> that will have the correct ICU version [10:50:08] claime: if you want to dig in the past how we did the last ICU transition: https://phabricator.wikimedia.org/T264991 [10:50:48] ack, bullseye will have ICU67 [10:51:51] 57-63 wasn't such a massive change compared to the previous one, maybe 63-67 is on the calmer side, we'll see [10:52:02] 10serviceops, 10Patch-For-Review: wikikube LIST secrets latency - https://phabricator.wikimedia.org/T323706 (10JMeybohm) Deployed to wikikube cluster this morning. I'll leave it running for couple of days with hopefully some deployments happening and will adapt the resource requests/limits after that. [10:52:19] and IIRC for 63 it was the first time enwiki had to be touched as well [10:53:44] <_joe_> yes [11:28:04] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Joe) >>! In T324200#8436279, @daniel wrote: > Note that we only need active purging if/when we emit cache control headers that tell th... [16:58:03] 10serviceops, 10API Platform, 10Foundational Technology Requests, 10Image-Suggestions, and 3 others: Public-facing API for image suggestions data - https://phabricator.wikimedia.org/T306349 (10DAbad) From a product perspective, ok with advancing with the extension solution. [17:49:54] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10Volans) Manually powered off `mw1312`: ` cumin1001 0 17:46:27 ~ $ sudo ipmitool -I lanplus -H "10.65.1.83" -U root -E chassis power off Unable to read password from environment Password: Ch... [18:15:23] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Decommission mw13[07-48] - https://phabricator.wikimedia.org/T306162 (10RLazarus) Thanks @Volans. DC ops, for mw1320, I wasn't able to manually shut it off -- please do just kill the power when you go in to unrack it. Thanks!