[09:02:02] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [09:03:22] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [12:07:32] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:28:10] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10JMeybohm) I tried to find a culprit but could not find anything obvious. The only thing... [14:50:26] 10serviceops, 10SRE, 10SecTeam-Processed, 10Security, 10Vuln-Misconfiguration: Helm Chart misconfigurations - https://phabricator.wikimedia.org/T355167 (10akosiaris) 05In progress→03Resolved a:03akosiaris I 'll resolve this, all patches have been merged. [14:54:02] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10Clement_Goubert) I don't think ipoid is alone in being affected. [[ https://logstash.wik... [15:10:01] 10serviceops, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Seems fine, most parsoid traffic is probably happening on the main cluster these days anyway. We just want to make sure that scandium doesn't... [15:11:28] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10JMeybohm) Yeah, although those are from a different index (`logstash-*` vs. `ecs-*`). No... [15:28:03] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10colewhite) Feels like index rollover, but these logs were found in a weekly-rotated inde... [16:06:14] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10JMeybohm) >>! In T357616#9546966, @colewhite wrote: > There were [[ https://logstash.wik... [16:46:36] 10serviceops, 10Observability-Logging, 10iPoid-Service: Logs from ipoid-production-daily-updates-28465260-qbbz9 are not visible in Logstash after Feb 14, 2024 @ 23:40:08.183 - https://phabricator.wikimedia.org/T357616 (10colewhite) >>! In T357616#9547141, @JMeybohm wrote: >>>! In T357616#9546966, @colewhite... [16:51:54] 10serviceops, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Also worth keeping in mind: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965608 -- in order to avoid inadventently getting e... [18:29:15] 10serviceops, 10Fundraising-Backlog, 10SRE, 10Wikimedia-Apache-configuration, and 2 others: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Scott_French) Thanks to @Dwisehaupt for preparing the config patch and @RLazarus for assistance deploying it. The change is now live and ap... [18:32:19] 10serviceops, 10Fundraising-Backlog, 10SRE, 10Wikimedia-Apache-configuration, 10fundraising-tech-ops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Scott_French) 05Open→03Resolved a:03Scott_French [19:01:22] 10serviceops, 10CirrusSearch, 10Discovery-Search (Current work), 10MW-1.42-notes (1.42.0-wmf.17; 2024-02-06), 10Patch-For-Review: High level of backend errors for CirrusSearch jobs in jobrunners - https://phabricator.wikimedia.org/T356526 (10EBernhardson) It seems the patch didn't actually make it into w... [19:43:55] 10serviceops, 10MW-on-K8s, 10SRE, 10Scap, 10Release-Engineering-Team (Now this 🫠): Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10dancy) p:05Triage→03Medium a:03dancy [20:29:21] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Wikimedia-production-error: RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic. - https://phabricator.wikimedia.org/T357713 (10thcipriani) >>! In T357713#9548631, @brennen wrote: > Not hard to imagine a scenari... [20:33:53] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Wikimedia-production-error: RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic. - https://phabricator.wikimedia.org/T357713 (10brennen) Going back to IRC logs from a bit ago, what I had in mind was {T355617}.... [22:36:35] 10serviceops, 10CirrusSearch, 10Discovery-Search (Current work), 10MW-1.42-notes (1.42.0-wmf.19; 2024-02-20), 10Patch-For-Review: High level of backend errors for CirrusSearch jobs in jobrunners - https://phabricator.wikimedia.org/T356526 (10EBernhardson) Was supposed to be in the backport window today,...