[07:30:39] 06serviceops, 10Wikidata, 10wmde-wikidata-tech, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 03Discovery-Search (Current work): Request permission to create 4 kafka topics in kafka-main (WDQS graph split) - https://phabricator.wikimedia.org/T367510#9997602 (10dcausse) >>! In T367510#9996901, @bking wro... [09:15:00] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9997733 (10Tchanders) [09:16:56] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9997744 (10JayCano) [09:20:31] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9997736 (10Tchanders) [09:45:10] 06serviceops, 10Observability-Metrics, 07Grafana, 07Kubernetes, 13Patch-For-Review: High cardinality metrics break queries/dashboards (envoy, istio, ...) - https://phabricator.wikimedia.org/T369607#9997886 (10JMeybohm) I have updated https://grafana-rw.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s which... [12:43:01] 06serviceops, 10Wikidata, 10wmde-wikidata-tech, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 03Discovery-Search (Current work): Request permission to create 4 kafka topics in kafka-main (WDQS graph split) - https://phabricator.wikimedia.org/T367510#9998316 (10Gehel) 05Open→03Resolved [13:03:59] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mw-mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9998376 (10jijiki) [13:08:56] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mw-mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9998395 (10jijiki) 05In progress→03Resolved Updated documentation on wikitech (see [[ https://wikitech.wikimedia.org/wiki/Mw-mcrouter | mw... [13:38:12] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#9998494 (10Clement_Goubert) I *tried* very hard to automate it [[ https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1054531 | with a cookbook ]], but the... [13:46:25] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998512 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2432 to wikikube-worker2035 completed: - mw2432 (**PASS*... [13:53:26] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998530 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2433 to wikikube-worker2036 completed: - mw2433 (**PASS*... [14:02:00] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998548 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2438 to wikikube-worker2037 completed: - mw2438 (**PASS*... [14:08:21] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998568 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by cgoubert@cumin1002 from mw2439 to wikikube-worker2038 completed: - mw2439 (**PASS*... [14:09:16] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998570 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2035.codfw.wmnet with OS bullseye [14:09:56] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998571 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2036.codfw.wmnet with OS bullseye [14:10:18] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998574 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS bullseye [14:10:50] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998579 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bullseye [14:20:15] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images running in Wikimedia production on Buster to either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#9998609 (10elukey) [14:36:54] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998684 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS bullseye executed with errors: - wi... [14:37:03] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998685 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bullseye executed with errors: - wi... [14:59:42] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998737 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS bullseye [15:09:29] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998784 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bullseye executed with errors: - wi... [15:10:31] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bullseye [15:24:56] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Move video transcoding to use Shellbox - https://phabricator.wikimedia.org/T356241#9998819 (10Clement_Goubert) [15:41:38] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998881 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2037.codfw.wmnet with OS bullseye completed: - wikikube-work... [15:52:56] 06serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9998923 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2038.codfw.wmnet with OS bullseye completed: - wikikube-work...