[01:30:01] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10487138 (10Eevans) [10:30:41] 06serviceops, 10MW-on-K8s, 10Observability-Logging: Unexpected utilization increase in udp_localhost-info kafka-logging topic - https://phabricator.wikimedia.org/T384233#10488036 (10fgiunchedi) good question, my understanding is that the problem is caused by messages with wacky timestamps that prevent re... [11:51:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10488314 (10kamila) [13:06:27] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488653 (10ops-monitoring-bot) depool host parse[1001-1006].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes [13:08:40] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes, and 2 others: Relabel codfw kubernetes nodes - https://phabricator.wikimedia.org/T383862#10488671 (10Jhancock.wm) a:03Jhancock.wm [13:09:57] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488674 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host parse[1001-1006].eqiad.wmnet... [13:11:16] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10488675 (10Jhancock.wm) [13:16:54] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488682 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1001 to wikikube-worker1142 completed: - pa... [13:20:34] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Error from mapnik internals when requesting a snapshot with an overlay - https://phabricator.wikimedia.org/T384285#10488701 (10elukey) 05Open→03Resolved a:03elukey Resolving since the issue... [13:20:59] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488705 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1002 to wikikube-worker1143 completed: - pa... [13:23:11] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Empty feature collection on geojson raises error - https://phabricator.wikimedia.org/T384435#10488716 (10elukey) 05Open→03Resolved This bug has been fixed as well in k8s staging, resolving. [13:25:32] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023#10488722 (10elukey) The updated library has been deployed to k8s staging and it works nicely. @Jgiannelos do yo... [13:25:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488723 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1003 to wikikube-worker1144 completed: - pa... [13:27:56] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Timeout when requests a map with geoshapes overlay - https://phabricator.wikimedia.org/T383710#10488726 (10elukey) 05Open→03Resolved Fixed in k8s staging. We decided to force Kartotherian to... [13:30:51] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488751 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1004 to wikikube-worker1145 completed: - parse1004 (**PASS**) -... [13:42:47] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488774 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1005 to wikikube-worker1146 completed: - parse1005 (**PASS**) -... [13:55:09] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488787 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1006 to wikikube-worker1147 completed: - parse1006 (**PASS**) -... [14:27:13] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488877 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1142.eqiad.wmnet with OS bookworm [14:27:19] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488878 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1143.eqiad.wmnet with OS bookworm [14:27:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488879 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1144.eqiad.wmnet with OS bookworm [14:27:38] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488881 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1145.eqiad.wmnet with OS bookworm [14:27:55] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1146.eqiad.wmnet with OS bookworm [14:28:08] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10488883 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1147.eqiad.wmnet with OS bookworm [14:42:54] 06serviceops, 13Patch-For-Review: Align mw-on-k8s alerts with PHP 8.1 migration - https://phabricator.wikimedia.org/T384532#10488929 (10jijiki) I completely agree that #2 is our best option in terms of value/effort, moving forward with that [15:03:54] 06serviceops, 06Content-Transform-Team, 06Content-Transform-Team-WIP, 10Maps (Kartotherian), 13Patch-For-Review: Difftesting between staging and production - https://phabricator.wikimedia.org/T384530#10489049 (10Jgiannelos) From a similar run but testing A/B between staging(eqiad) /prod(eqiad) since in t... [15:06:08] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489068 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1144.eqiad.wmnet with OS bookworm completed: - wik... [15:07:43] 06serviceops, 06Content-Transform-Team-WIP, 10Maps (Kartotherian), 13Patch-For-Review: Difftesting between staging and production - https://phabricator.wikimedia.org/T384530#10489069 (10ihurbain) [15:11:05] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489076 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1145.eqiad.wmnet with OS bookworm completed: - wik... [15:13:12] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10489081 (10AUgolnikova-WMF) @Eevans Yes, I'm the PM for the Structured Content team. I have a few questions about this migration: What is the benefit for us to... [15:13:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489083 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1146.eqiad.wmnet with OS bookworm completed: - wik... [15:15:19] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489090 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1143.eqiad.wmnet with OS bookworm completed: - wik... [15:15:34] 06serviceops, 06Content-Transform-Team-WIP, 10Maps (Kartotherian): Staging error: Snapshots with overlay map failed to render - https://phabricator.wikimedia.org/T384023#10489091 (10ihurbain) [15:18:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1147.eqiad.wmnet with OS bookworm completed: - wik... [15:23:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1142.eqiad.wmnet with OS bookworm completed: - wik... [15:29:13] 06serviceops, 06DC-Ops, 10ops-eqiad, 10Prod-Kubernetes, and 2 others: Relabel eqiad kubernetes nodes - https://phabricator.wikimedia.org/T383620#10489148 (10kamila) [15:31:32] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489178 (10ops-monitoring-bot) pool host wikikube-worker[1142-1147].eqiad.wmnet by kamila@cumin1002 with reason: None [15:35:10] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489198 (10ops-monitoring-bot) pool host wikikube-worker[1142-1147].eqiad.wmnet by kamila@cumin1002 with reason: None [15:36:38] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10489202 (10ops-monitoring-bot) pool host wikikube-worker[1142-1147].eqiad.wmnet by kamila@cumin1002 with reason: None [15:38:26] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10489207 (10VRiley-WMF) [15:50:35] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489374 (10Jhancock.wm) @Marostegui db2189 is moved, updated, and pinging! [15:51:34] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489378 (10Jhancock.wm) [16:07:18] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489449 (10Marostegui) >>! In T383709#10489374, @Jhancock.wm wrote: > @Marostegui db2189 is moved, updated,... [16:10:59] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10489466 (10VRiley-WMF) [16:19:56] 06serviceops, 10API Platform, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, and 2 others: Reading List REST Interface: reroute calls - https://phabricator.wikimedia.org/T348493#10489525 (10HCoplin-WMF) [16:35:50] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10489643 (10VRiley-WMF) 05Open→03Resolved [16:36:55] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489651 (10Jhancock.wm) @JMeybohm, what do you think of this schedule for getting these moved? wikikube-wor... [16:42:03] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489669 (10JMeybohm) >>! In T383709#10489651, @Jhancock.wm wrote: > @JMeybohm, what do you think of this sc... [17:15:34] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489809 (10Eevans) cassandra-dev2001 can be moved at your leisure (no coordination is needed). [17:18:33] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10489812 (10Jhancock.wm) [17:48:00] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10489959 (10Eevans) >>! In T368096#10489081, @AUgolnikova-WMF wrote: > @Eevans Yes, I'm the PM for the Structured Content team. I have a few questions about this... [18:09:34] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10490012 (10ops-monitoring-bot) pool host wikikube-worker[1142-1147].eqiad.wmnet by kamila@cumin1002 with reason: None [18:09:36] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10490013 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1142-1147].eqiad.wmnet completed: - w... [19:37:25] 06serviceops, 06collaboration-services, 06Data-Persistence, 06DC-Ops, and 2 others: Tracking List: Relocating servers to free up 10G switch space in codfw - https://phabricator.wikimedia.org/T383709#10490386 (10Jhancock.wm)