[00:05:40] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10414560 (10BPirkle) I've completed manual testing on various MW REST API endpoints using `mwdebu... [07:18:14] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414860 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2190.co... [08:00:39] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414897 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2190.codfw.... [08:01:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414899 (10ops-monitoring-bot) pool host wikikube-worker2190.codfw.wmnet by jelto@cumin1002 with reason: None [08:01:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414900 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [08:02:45] 06serviceops, 06collaboration-services, 06DC-Ops, 10ops-codfw, and 3 others: Comm Error: backplane 0 when reimaging wikikube-worker2190 - https://phabricator.wikimedia.org/T382420#10414901 (10Jelto) The host responses normally and a reimage worked. Thanks @Jhancock.wm for the quick help! [08:04:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414902 (10ops-monitoring-bot) depool host wikikube-worker[2061-2062].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [08:08:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414906 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [08:09:07] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414907 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2061.co... [08:09:10] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414908 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2062.co... [08:52:37] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414954 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2061.codfw.... [08:55:53] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414955 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2062.codfw.... [09:03:13] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414964 (10ops-monitoring-bot) pool host wikikube-worker[2061-2062].codfw.wmnet by jelto@cumin1002 with reason: None [09:03:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414965 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [09:06:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414966 (10ops-monitoring-bot) depool host wikikube-worker[2059-2060].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [09:08:12] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414968 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [09:09:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2060.co... [09:09:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10414982 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2059.co... [09:26:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10414998 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1296-1300].eqiad.wmnet [09:28:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10414999 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1301-1304].eqiad.wmnet [09:51:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415043 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2060.codfw.... [09:54:54] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415063 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2059.codfw.... [09:57:38] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415069 (10ops-monitoring-bot) pool host wikikube-worker[2059-2060].codfw.wmnet by jelto@cumin1002 with reason: None [09:57:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415070 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [09:58:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415083 (10ops-monitoring-bot) depool host wikikube-worker[2057-2058].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [09:59:48] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415086 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [10:00:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415087 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2057.co... [10:01:04] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415088 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2058.co... [10:16:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2058.codfw.... [10:18:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415123 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2058.co... [10:40:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415150 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2057.codfw.... [10:59:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415194 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2058.codfw.... [10:59:57] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415195 (10ops-monitoring-bot) pool host wikikube-worker[2057-2058].codfw.wmnet by jelto@cumin1002 with reason: None [11:00:00] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415196 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [11:00:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415197 (10ops-monitoring-bot) depool host wikikube-worker[2055-2056].codfw.wmnet by jelto@cumin1002 with reason: Reimage node to... [11:01:59] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415198 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host wikikube-wor... [11:02:46] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415201 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2056.co... [11:02:47] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415202 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2055.co... [12:31:05] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415363 (10BTullis) [13:00:56] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10415441 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker[1017-1020].eqiad.wmnet [13:15:27] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415473 (10BTullis) [13:21:37] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415499 (10BTullis) [14:16:42] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415691 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2055.codfw.... [14:17:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415693 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2055.co... [14:36:54] jayme: jelto: I merged https://gitlab.wikimedia.org/repos/sre/serviceops-kitchensink/-/merge_requests/21 to not reuse ids, lmk if it's all good if you want me to make more changes before leaving until the 6th [14:49:14] claime: thanks! The output looks good to me, the script suggests wikikube-worker2192+ to me. So it's not using the gaps from decommissioning. [14:49:14] But I wont start any new renaming today, just one or two more reimages. I'll also be back at the 6th and don't want to cause any trouble with unavailable nodes during the holidays :) [14:49:30] ack thanks for checking [14:52:40] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q2): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#10415796 (10fgiunchedi) I believe this is done for all intents and purposes, what do y... [14:57:11] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415806 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2056.codfw.... [14:57:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415819 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2056.co... [15:00:07] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q2): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#10415830 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert Th... [15:01:58] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415857 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2055.codfw.... [15:26:27] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10415933 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1301-1304].eqiad.wmnet} and (A:wiki... [15:39:19] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10415974 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2056.codfw.... [15:49:35] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10416022 (10ops-monitoring-bot) pool host wikikube-worker[2055-2056].codfw.wmnet by jelto@cumin1002 with reason: None [15:49:36] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate wikikube-codfw to containerd - https://phabricator.wikimedia.org/T377877#10416023 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worke... [15:50:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416025 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1017-1020].eqiad.wmnet} and (A:wiki... [16:00:16] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416078 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker1022.eqiad.wmnet [16:11:31] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416211 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1022].eqiad.wmnet} and (A:wikikube-... [16:15:20] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416230 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1296-1300].eqiad.wmnet} and (A:wiki... [16:16:17] 06serviceops, 10Observability-Logging: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10416242 (10dancy) [16:17:30] 06serviceops, 10Observability-Logging: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10416255 (10dancy) [16:30:32] 06serviceops, 10Observability-Logging: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10416354 (10Scott_French) Thanks for flagging, @dancy! This looks like a fairly simple typo. I'll post a patch to fix shortly. [16:48:12] 06serviceops, 10Observability-Logging, 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10416496 (10dancy) p:05Triage→03Medium [16:58:21] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416545 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker1022.eqiad.wmnet [17:00:41] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416549 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1022].eqiad.wmnet} and (A:wikikube-... [17:00:49] 06serviceops, 10Wikipedia-iOS-App-Backlog (iOS Release FY2024-25): Add new APNS server certificates to Trust Store - https://phabricator.wikimedia.org/T381808#10416554 (10Tsevener) @Seddon Per discussion in CT meeting yesterday with @Jgiannelos, SRE should do this work. They will also need to build and image... [17:21:34] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416660 (10ops-monitoring-bot) Started reimage of nodes in wikikube-eqiad cluster: * wikikube-worker1022.eqiad.wmnet [18:06:32] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Migrate wikikube-eqiad to containerd - https://phabricator.wikimedia.org/T377876#10416826 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.roll-reimage-nodes -- rolling reimage on P{wikikube-worker[1022].eqiad.wmnet} and (A:wikikube-... [18:54:37] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.12; 2025-01-14): PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10416941 (10Scott_French) Alright, I believe that should resolve the undefined array key issue: ` swfren... [19:30:18] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03): PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10417048 (10dancy) >>! In T382517#10416941, @Scott_French wrote: > Alright, I believe that should resolve... [19:30:40] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03): PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10417049 (10dancy) [19:41:46] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03): PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10417063 (10Scott_French) Thanks @dancy! So, after a bit of poking, I suspect the missing logs may be str... [22:17:34] 06serviceops, 10Observability-Logging, 10MW-1.44-notes (1.44.0-wmf.6; 2024-12-03), 13Patch-For-Review: PHP Warning seen by logspam-watch but not by mediawiki-errors logstash page - https://phabricator.wikimedia.org/T382517#10417470 (10Scott_French) One tricky aspect of deploying https://gerrit.wikimedia.or...