[02:35:44] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 07Grafana, 10SRE Observability (FY2024/2025-Q1): Gaps in Grafana graphs using Thanos - https://phabricator.wikimedia.org/T371885#10071919 (10lmata) [07:48:52] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 10observability, and 3 others: Upgrade Kafka to from 1.x to later version - https://phabricator.wikimedia.org/T300102#10072072 (10brouberol) The upgrade of kafka is currently a stretch OKR and will probably be a core OKR for next quarter. I'll definite... [10:38:24] 06serviceops, 10Page Content Service, 07Code-Health-Objective: Move PCS endpoints behind API Gateway - https://phabricator.wikimedia.org/T264670#10072688 (10Jgiannelos) After our latest restbase sunset meeting on Aug 13th, we decided we are going to change our strategy for PCS deprecation and instead of wait... [10:39:25] 06serviceops, 10Page Content Service, 07Code-Health-Objective: Route ptwiki mobile-html to the backing node service instead of RESTBase - https://phabricator.wikimedia.org/T372746 (10Jgiannelos) 03NEW [10:40:14] 06serviceops, 10Page Content Service, 07Code-Health-Objective: Use backing node service instead of RESTBase on pregeneration changeprop rules - https://phabricator.wikimedia.org/T372749 (10Jgiannelos) 03NEW [10:41:16] 06serviceops, 10Page Content Service, 07Code-Health-Objective: ptwiki: Use backing node service instead of RESTBase on pregeneration changeprop rules - https://phabricator.wikimedia.org/T372749#10072739 (10Jgiannelos) [10:41:38] 06serviceops, 10Page Content Service, 07Code-Health-Objective: ptwiki: Route mobile-html to the backing node service instead of RESTBase - https://phabricator.wikimedia.org/T372746#10072743 (10Jgiannelos) [11:26:54] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10072829 (10ayounsi) @Jclark-ctr @Clement_Goubert fixed the typo, so you should be good to go. We also got this alert: ` FAIL: cumin-check-aliases Syst... [12:38:45] 06serviceops, 10CX-cxserver, 06Language-Team, 10RESTBase Sunsetting: Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10073022 (10MSantos) [12:40:18] 06serviceops, 10CX-cxserver, 06Language-Team, 10RESTBase Sunsetting: Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10073020 (10MSantos) [12:49:04] 06serviceops, 10CX-cxserver, 06Language-Team, 10RESTBase Sunsetting: Switchover plan from RESTBase to REST Gateway for cxserver - https://phabricator.wikimedia.org/T372753#10073029 (10MSantos) I did an initial effort to get this list from the restbase specs, but I'm not sure I got everything right. Once th... [13:41:55] were the http stats for mw-jobrunner always empty on the RED dashboards? O_o https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&refresh=1m&var-site=All&var-deployment=mw-jobrunner&var-method=GET&var-code=200&var-handler=&var-service=mediawiki&from=now-2d&to=now [14:30:11] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073515 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1300.eqiad.wmnet with OS bull... [14:30:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073516 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1301.eqiad.wmnet with OS bull... [14:30:17] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073518 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1303.eqiad.wmnet with OS bull... [14:30:23] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073519 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1300.eqiad.wmnet with OS bullseye... [14:30:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073520 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1301.eqiad.wmnet with OS bullseye... [14:30:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073521 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1303.eqiad.wmnet with OS bullseye... [14:31:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073536 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1300.eqiad.wmnet with OS bull... [14:31:12] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1301.eqiad.wmnet with OS bull... [14:31:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073538 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1303.eqiad.wmnet with OS bull... [14:31:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073540 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1302.eqiad.wmnet with OS bull... [14:31:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073542 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1304.eqiad.wmnet with OS bull... [14:44:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10073649 (10VRiley-WMF) Worked with Dell on 1298. They have determined it will require a motherboard swap. Parts will be coming tomorrow. (Dell ticket 196... [16:00:19] hnowlan: err I don't think so [16:01:24] mw-jobrunner dashboard works [16:01:33] https://grafana.wikimedia.org/goto/ooQVh8jIg?orgId=1 [16:03:16] huh [16:05:56] hmm it uses mediawiki_http_requests_duration_count which iirc is a benthos mediated one [16:47:05] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800 (10RobH) 03NEW [16:47:36] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10074438 (10RobH) [16:49:33] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10074457 (10RobH) a:03jijiki @jijiki, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving the new ser... [16:49:34] 06serviceops: mc-misc200[12] implementation tracking - https://phabricator.wikimedia.org/T372802 (10RobH) 03NEW [16:57:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074502 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1300.eqiad.wmnet with OS bullseye... [16:57:41] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074503 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1301.eqiad.wmnet with OS bullseye... [16:57:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1302.eqiad.wmnet with OS bullseye... [16:57:56] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074505 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1303.eqiad.wmnet with OS bullseye... [16:58:04] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074507 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1304.eqiad.wmnet with OS bullseye... [16:59:03] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074509 (10Jclark-ctr) [16:59:49] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074512 (10Jclark-ctr) [17:00:38] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074514 (10Jclark-ctr) [17:02:16] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074521 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1286.eqiad.wmnet with OS bull... [17:02:17] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074522 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1285.eqiad.wmnet with OS bull... [17:50:22] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074735 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1285.eqiad.wmnet with OS bullseye... [17:50:26] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074736 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1286.eqiad.wmnet with OS bullseye... [17:50:55] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074738 (10Jclark-ctr) [17:52:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10074750 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1299.eqiad.wmnet with OS bull... [20:28:50] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images running in Wikimedia production on Buster to either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#10075509 (10thcipriani) Noting here for visibility: >>! In T372825#10075500, @thciprian...