[05:20:06] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (Bonus Level 🕹ī¸): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10abi_) [08:40:51] last version of knative: "Bump min-version to k8s 1.23, so removing kind 1.22 testing" [08:40:54] :D :D :D [09:55:15] sounds like we schould start creating update tasks for 1.24+ :) [10:10:19] jayme: don't you want to have fun with dockershim? :D [10:11:29] next year, maybe :p [10:13:34] 10serviceops, 10CampaignEvents, 10Wikimedia-Site-requests, 10Campaign-Registration, and 2 others: Run the timezone update script periodically in prod and in beta - https://phabricator.wikimedia.org/T320403 (10Clement_Goubert) Sorry for the delay, it's been deployed. [10:35:31] elukey: can I possibly lure you into reviewing https://gerrit.wikimedia.org/r/861352 😇 [10:39:42] jayme: sure yes! It is urgent or ok during the next couple of days? [11:05:55] Morning. I'm also hoping for a review on spark-operator at some point this week, if possible: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/855674 I've adapted it to use the module templates. [12:45:34] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: tinyrgb is distributed via puppet - https://phabricator.wikimedia.org/T323775 (10MoritzMuehlenhoff) There's also a fourth option that comes to my mind: Debian already ships various ICC profiles, in two separate packages: https://tracker.de... [14:19:40] 10serviceops, 10Observability-Tracing: Package OpenTelemetry Collector as a .deb - https://phabricator.wikimedia.org/T320551 (10Clement_Goubert) [14:19:42] 10serviceops, 10Observability-Tracing: OpenTelemetry Collector puppetized and able to be deployed easily to arbitrary roles - https://phabricator.wikimedia.org/T320565 (10Clement_Goubert) 05In progress→03Resolved Rudimentary installation for otelcol-contrib as agent on puppetized servers is live and ready... [14:43:59] btullis: ...module templates? [14:45:27] OH sorry, this is in deployment-charts. i was looking at spark-operator and thinking it was in production-images and there was something new i needed to know about. nevermind, carry on! [14:47:24] ottomata: Yes, it's as a result of this work by _joe_: to make the deployment-charts templates nicer: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/837495 [15:17:40] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: tinyrgb is distributed via puppet - https://phabricator.wikimedia.org/T323775 (10Joe) The most obvious thing to me is to include the file in the thumbor docker image. It's ok to have a small binary that doesn't change much in it. [15:25:34] <_joe_> hnowlan: I am inclined to merge https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/860711 [15:25:46] <_joe_> unless it interferes with your current work [15:27:04] _joe_: I think that should be fine, thanks [15:27:24] <_joe_> hnowlan: we're still not serving traffic, right? [15:27:43] _joe_: nope, not yet [15:27:58] we're close once this batch of tests passes though [15:28:31] * _joe_ crosses fingers [15:31:40] <_joe_> hnowlan: no TLS on thumbor right now, correct? [15:37:58] <_joe_> hnowlan: I'm deploying your path fixes too [15:51:03] 10serviceops, 10ops-codfw: codfw: ManagementSSHDown for ores2009 and thumbor2004 - https://phabricator.wikimedia.org/T323925 (10Papaul) [15:51:16] 10serviceops, 10ops-codfw: codfw: ManagementSSHDown for ores2009 and thumbor2004 - https://phabricator.wikimedia.org/T323925 (10Papaul) p:05Triage→03High [16:01:52] _joe_: no TLS yep, thanks! [17:22:33] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10Platform Team Workboards (Platform Engineering Reliability): 3d2png failing in Kubernetes - https://phabricator.wikimedia.org/T323936 (10hnowlan) [17:36:30] 10serviceops, 10SRE, 10Wikibase Product Platform, 10Wikimedia-Apache-configuration: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10jijiki) @Silvan_WMDE sorry for not replying sooner, I will take a look at this when I find... [17:36:45] 10serviceops, 10SRE, 10Wikibase Product Platform, 10Wikimedia-Apache-configuration: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10jijiki) a:03jijiki [17:50:47] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10Platform Team Workboards (Platform Engineering Reliability): 3d2png failing in Kubernetes - https://phabricator.wikimedia.org/T323936 (10hnowlan) [18:23:43] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10Platform Team Workboards (Platform Engineering Reliability): 3d2png failing in Kubernetes - https://phabricator.wikimedia.org/T323936 (10hnowlan) [21:06:13] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:rack/setup/install arclamp1001.eqiad.wmnet - https://phabricator.wikimedia.org/T319433 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host arclamp1001.eqiad.wmnet with OS bullseye