[00:07:31] 06serviceops, 06SRE: Update grafana link for mediawiki-error-rate-$cluster in icinga check - https://phabricator.wikimedia.org/T281261#9728471 (10Dzahn) This was about https://gerrit.wikimedia.org/r/c/operations/puppet/+/668166/2/modules/profile/manifests/mediawiki/alerts.pp but now this seems all deleted fro... [00:09:09] 06serviceops, 06SRE: Update grafana link for mediawiki-error-rate-$cluster in icinga check - https://phabricator.wikimedia.org/T281261#9728472 (10Dzahn) p:05Triage→03Low Check was deleted in https://gerrit.wikimedia.org/r/c/operations/puppet/+/885288 by Giuseppe [08:02:11] 06serviceops, 07Kubernetes, 13Patch-For-Review: Fix rendering issue in modules.app.job when cronjobs are enabled and private values are defined - https://phabricator.wikimedia.org/T362954#9728820 (10JMeybohm) [08:03:15] 06serviceops, 07Kubernetes, 13Patch-For-Review: Fix rendering issue in modules.app.job when cronjobs are enabled and private values are defined - https://phabricator.wikimedia.org/T362954#9728819 (10JMeybohm) @jijiki I recall you had dug up some things around jobs/cronjobs as well, maybe you can take a look? [08:10:54] 06serviceops, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 07Kubernetes, 13Patch-For-Review: Fix rendering issue in modules.app.job when cronjobs are enabled and private values are defined - https://phabricator.wikimedia.org/T362954#9728825 (10brouberol) [08:11:27] 06serviceops, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 07Kubernetes, 13Patch-For-Review: Fix rendering issue in modules.app.job when cronjobs are enabled and private values are defined - https://phabricator.wikimedia.org/T362954#9728826 (10brouberol) [08:25:24] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9728841 (10JMeybohm) httpd-fcgi + dependent images seem to not have successfully rebuild on Monday. checking. [08:29:10] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9728843 (10JMeybohm) Successfully published image docker-registry.discovery.wmnet/httpd-fcgi:2.4.38-10-u5-20240415 Successfully published image... [09:07:42] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9729001 (10brouberol) [09:10:49] 06serviceops, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 07Kubernetes, 13Patch-For-Review: Fix rendering issue in modules.app.job when cronjobs are enabled and private values are defined - https://phabricator.wikimedia.org/T362954#9729004 (10Gehel) p:05Triage→03High [09:59:10] 06serviceops, 10ops-codfw, 06SRE: Degraded RAID on mw2382 - https://phabricator.wikimedia.org/T362938#9729097 (10Clement_Goubert) I suppose that can be hotswapped? Let us know if it can't, we'll drain and cordon the host for the disk swap. [10:24:08] 06serviceops, 06SRE: Update grafana link for mediawiki-error-rate-$cluster in icinga check - https://phabricator.wikimedia.org/T281261#9729176 (10Clement_Goubert) 05Open→03Invalid Yes, alert was moved to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/media... [12:07:21] 06serviceops: Provide nodejs20 base images for production - https://phabricator.wikimedia.org/T362681#9729331 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff That's not problem. We should just use the nodesource packages for this, we've been doing the same for "intermediate LTSes" before (e.g. node 16 or node 14)... [12:13:02] 06serviceops: Update cache.mrouter modules in deployment-charts - https://phabricator.wikimedia.org/T355237#9729340 (10JMeybohm) 05Resolved→03Open p:05Low→03High This breaks in CI when actually enabled. When mcrouter is enabled during scaffolding, it is not actually enabled in the chart because values.y... [12:28:11] 06serviceops: Provide nodejs20 base images for production - https://phabricator.wikimedia.org/T362681#9729369 (10Jdforrester-WMF) Thank you! [12:41:27] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507#9729396 (10JMeybohm) [12:42:01] 06serviceops, 10Prod-Kubernetes: Update all helm modules and charts to be compatible with the restricted PSS - https://phabricator.wikimedia.org/T362978 (10JMeybohm) 03NEW [12:42:06] 06serviceops, 10Prod-Kubernetes: Update all helm modules and charts to be compatible with the restricted PSS - https://phabricator.wikimedia.org/T362978#9729408 (10JMeybohm) p:05Triage→03High [12:44:09] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507#9729410 (10JMeybohm) [13:02:27] 06serviceops, 13Patch-For-Review: Update cache.mrouter modules in deployment-charts - https://phabricator.wikimedia.org/T355237#9729448 (10JMeybohm) I think I fixed all that because it was blocking me. But there is also two definitions of the mcrouter container: `{{- define "cache.mcrouter.container" -}}` an... [13:04:31] 06serviceops, 13Patch-For-Review: Update cache.mrouter modules in deployment-charts - https://phabricator.wikimedia.org/T355237#9729451 (10JMeybohm) p:05High→03Medium [13:06:17] 06serviceops, 10MW-on-K8s: Migrate mw-on-k8s base image from buster to bullseye - https://phabricator.wikimedia.org/T362981 (10Jdforrester-WMF) 03NEW [13:11:32] 06serviceops, 10MW-on-K8s: Migrate mw-on-k8s base image from buster to bullseye - https://phabricator.wikimedia.org/T362981#9729480 (10MoritzMuehlenhoff) All the custom PHP extensions are already fully rebuilt for the component/php74 for bullseye! And being kept updated, e.g. the recent PHP security fixes were... [13:14:14] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Migrate mw-on-k8s base image from buster to bullseye - https://phabricator.wikimedia.org/T362981#9729490 (10Jdforrester-WMF) >>! In T362981#9729480, @MoritzMuehlenhoff wrote: > All the custom PHP extensions are already fully rebuilt for the component/php74 for b... [13:14:26] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Migrate mw-on-k8s base image from buster to bullseye - https://phabricator.wikimedia.org/T362981#9729492 (10Jdforrester-WMF) [13:22:13] 06serviceops, 10Prod-Kubernetes, 13Patch-For-Review: Update all helm modules and charts to be compatible with the restricted PSS - https://phabricator.wikimedia.org/T362978#9729509 (10JMeybohm) [18:01:58] 06serviceops, 06SRE: Refactor memcached modules - https://phabricator.wikimedia.org/T284454#9730381 (10Dzahn) p:05Triage→03Low setting priority to low - to get it out of "untriaged incoming SRE tickets list". Just guessing based on the way the ticket is phrased and the age of it. Of course not trying to te... [21:02:18] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9730773 (10Ottomata) > I fear I read that task, the way it is written at least, differently.... [22:59:08] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9731003 (10Dzahn) > According to CodeSearch, affected images are: Codesearch didn't find itself ?:) https://gerrit.wikimedia.org/r/c/operation...