[08:16:14] 06serviceops, 10MinT, 10Prod-Kubernetes, 06SRE, and 3 others: machinetranslation eqiad pods in state ContainerStatusUnknown - https://phabricator.wikimedia.org/T411058#11411996 (10Nikerabbit) [08:42:57] 06serviceops: WE6.2.6: ☂️ hcaptcha-proxy Production Readiness Review - https://phabricator.wikimedia.org/T410626#11412040 (10jijiki) [09:50:56] 06serviceops: Improve detection of kafka-main broker TLS certificate rotations - https://phabricator.wikimedia.org/T410552#11412206 (10Blake) I am not super excited about how this looks, but I think it's giving me results that are consistent with what we want. `label_replace(node_file_age_timestamp_seconds_tota... [11:45:51] 06serviceops, 13Patch-For-Review: Improve detection of kafka-main broker TLS certificate rotations - https://phabricator.wikimedia.org/T410552#11412512 (10Blake) Welp, turns out we can't use bool here, because the alert is checking for the presence of the metric, and if we use bool, there will always be a metr... [13:29:46] 06serviceops, 05WE4.2 Bot detection: hcaptcha extension, proxy: Define the backoff and retry strategies - https://phabricator.wikimedia.org/T411115#11412971 (10Raine) [14:13:02] 06serviceops, 06Traffic, 05WE4.2 Bot detection: hcaptcha-proxy health checks should also depool sites if their upstream is unreachable - https://phabricator.wikimedia.org/T411191 (10Raine) 03NEW [15:05:23] 06serviceops, 06Traffic, 05WE4.2 Bot detection: hcaptcha-proxy health checks should also depool sites if their upstream is unreachable - https://phabricator.wikimedia.org/T411191#11413343 (10ssingh) Yeah I think that makes sense if we want to exert control over upstream issues and how it reflects to the prox... [15:06:56] 06serviceops, 06Traffic, 05WE4.2 Bot detection: hcaptcha-proxy health checks should also depool sites if their upstream is unreachable - https://phabricator.wikimedia.org/T411191#11413347 (10ssingh) > And while there is that fallback mechanism to the old system, this is something to keep in mind. I mean Fan... [15:18:41] Hello again, one more changeprop change in prod incoming: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1212148/1/helmfile.d/services/changeprop/values-production.yaml [15:39:24] 06serviceops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: etcd in codfw burned all latency SLO error budget - https://phabricator.wikimedia.org/T345738#11413482 (10akosiaris) 05Open→03Resolved a:03akosiaris Resolving per last comment. 2 year old task anyway. [16:22:44] 06serviceops: Proof of Concept: SquareOne Dashboards - https://phabricator.wikimedia.org/T411202 (10jijiki) 03NEW [16:23:03] 06serviceops: Proof of Concept: SquareOne Dashboards - https://phabricator.wikimedia.org/T411202#11413719 (10jijiki) 05Open→03In progress p:05Triage→03Medium [16:36:28] 06serviceops: Draft Guided Dashboards Design Proposal - https://phabricator.wikimedia.org/T411204 (10jijiki) 03NEW [16:36:42] 06serviceops: Proof of Concept: SquareOne Dashboards - https://phabricator.wikimedia.org/T411202#11413779 (10jijiki) [16:36:43] 06serviceops: Draft Guided Dashboards Design Proposal - https://phabricator.wikimedia.org/T411204#11413778 (10jijiki) [16:37:13] 06serviceops: Proof of Concept: SquareOne Dashboards - https://phabricator.wikimedia.org/T411202#11413782 (10jijiki) [16:37:48] 06serviceops: Draft Guided Dashboards Design Proposal - https://phabricator.wikimedia.org/T411204#11413791 (10jijiki) 05Open→03In progress [16:37:49] 06serviceops, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team (Roadmap): Allow developers to disable their own OAuth clients - https://phabricator.wikimedia.org/T254190#11413789 (10bd808) [22:27:23] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11414385 (10Krinkle) [22:28:50] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11414388 (10Krinkle)