[06:53:46] 06serviceops, 06SRE: host rdb1014 is down - https://phabricator.wikimedia.org/T376961#10224623 (10LSobanski) [10:17:46] 06serviceops, 06SRE: host rdb1014 is down - https://phabricator.wikimedia.org/T376961#10225247 (10akosiaris) 05Open→03Resolved a:03akosiaris The host has some history of failure per {T370633} It is the passive failover for rdb1013, which means we have no degradation of anything right now. Nothing... [10:57:27] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Evaluate running a statsd-exporter in the mw-script namespace - https://phabricator.wikimedia.org/T376714#10225348 (10akosiaris) 05Open→03Resolved statsd-exporter deployment merged and deployed in both eqiad and codfw. It is addressable via the standard... [11:25:47] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10225458 (10JMeybohm) [12:56:12] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, 07Kubernetes: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132 (10JMeybohm) 03NEW [13:13:00] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, 07Kubernetes: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10225849 (10JMeybohm) [13:20:33] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, 07Kubernetes: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10225872 (10tappof) ` jayme │ tappof: I see the log format on disk changed :/ jayme │ updated the t... [13:26:36] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, 07Kubernetes: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10225902 (10JMeybohm) The new/containerd on disk log format is part of CRI definition (https://github.c... [13:50:47] 06serviceops, 10Sustainability (Incident Followup): Expand upon Kask/Sessionstore documentation - https://phabricator.wikimedia.org/T320398#10225997 (10hnowlan) a:05hnowlan→03None [14:05:22] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Sessionstore's discovery TLS cert will expire before end of May 2024 - https://phabricator.wikimedia.org/T363996#10226099 (10hnowlan) >>! In T363996#10220536, @elukey wrote: > @hnowlan if echostore turns out to work as expected (it sounds so from the othe... [14:40:00] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10226221 (10elukey) p:05Triage→03Medium [14:46:05] 06serviceops, 06Infrastructure-Foundations: Migrate Docker Registry's storage to S3/APU - https://phabricator.wikimedia.org/T376453#10226276 (10elukey) [14:46:13] 06serviceops, 06Infrastructure-Foundations: Migrate Docker Registry's storage to S3/APU - https://phabricator.wikimedia.org/T376453#10226277 (10elukey) p:05Triage→03Medium [14:56:59] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10226327 (10cmooney) >>! In T375845#10182322, @JMeybohm wrote: >> It might be possible though to migrate to a new IPPool that... [15:04:02] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10226362 (10JMeybohm) >>! In T375845#10226327, @cmooney wrote: > If we do this we probably need to allocate a new single pool,... [15:05:31] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running periodic jobs for mw on k8s - https://phabricator.wikimedia.org/T341555#10226369 (10Clement_Goubert) Since I will have to get rid of `mw-cli-wrapper` for launching scripts, there is as of yet no mechanism to *ensure* jobs do not get run on the seco... [16:29:15] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, and 2 others: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10226712 (10JMeybohm) There is an option in containerd to disable the container log line length limit (ab... [16:30:39] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, and 2 others: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10226718 (10JMeybohm)