[08:53:31] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: CPU 2 machine check error detected for rdb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T370633#10044349 (10jijiki) 05Open→03Resolved @Jclark-ctr Thank you! Closing for now and will reopen if the problem persists [10:16:56] 06serviceops: low rate of mw-memcached errors - https://phabricator.wikimedia.org/T371881 (10jijiki) 03NEW [10:19:22] 06serviceops: low rate of mw-memcached errors - https://phabricator.wikimedia.org/T371881#10044540 (10jijiki) p:05Triage→03Low a:03jijiki [11:23:49] 06serviceops: low rate of mw-memcached errors - https://phabricator.wikimedia.org/T371881#10044686 (10jijiki) [13:43:17] hi folks, as a sanity check does it track to you that statsd-exporter here is heavily cpu throttled? https://grafana.wikimedia.org/goto/5eSZyP9SR?orgId=1 [13:57:56] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 07Grafana: Gaps in Grafana graphs using Thanos - https://phabricator.wikimedia.org/T371885#10045150 (10fgiunchedi) Thank you for the detailed report @daniel ! Made it super easy to reproduce and investigate. I have played around with `rate()` interval and... [13:57:57] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 07Grafana: Gaps in Grafana graphs using Thanos - https://phabricator.wikimedia.org/T371885#10045151 (10fgiunchedi) [14:01:05] godog: does it happen often from the look of it? [14:01:49] to your knowledge at least [14:02:18] effie: I think I'm not following, what's "it" ? the throttling ? [14:05:38] sorry, I am not context switching very well [14:06:54] no worries [14:07:06] I meant, have you noticed that the statsd-exporter being generally throlltled? [14:07:47] looking the other deployments, it is probably happening in general and not only in mw-api-ext [14:08:29] yes indeed, it looks like it gets throttled everywhere [14:10:20] alright, if we still have an open task about statsd, do you mind adding a comment? if not, we could open a new task unless someone from the team can take a look now [14:10:57] effie: for sure, thank you will do [14:16:17] 06serviceops, 10MW-on-K8s, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q1): Create a per-release deployment of statsd-exporter for mw-on-k8s - https://phabricator.wikimedia.org/T365265#10045198 (10fgiunchedi) p:05Low→03High I'm bumping this task for visibility as it... [14:38:48] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10045289 (10Jhancock.wm) a:03Jhancock.wm [14:38:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Install (2) 960GB SSDs each in kafka-main20[06-10] - https://phabricator.wikimedia.org/T371423#10045294 (10Jhancock.wm) we can schedule this any time on Wednesday or Thursday this week (august 7th or 8th) or some time next week. Drives are set aside and ready to... [15:11:19] hey folks, for https://phabricator.wikimedia.org/T371132 I'd need to run the provision cookbook for some wikikube workers [15:11:32] it reboots them, so I'd need to drain first [15:11:49] ok I proceed with wikikube-worker2035 ? [15:22:01] (proceeding) [15:26:11] all right done, node uncordoned [15:26:15] tomorrow I'll do the others :) [18:52:04] 06serviceops, 10Cassandra: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10046492 (10Scott_French) Ah, these are good questions. So, taking a step back, we know that the code involved in serving the image-suggestions endpoints is identical between the two... [19:27:48] 06serviceops, 10Cassandra: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10046538 (10Eevans) >>! In T368096#10046492, @Scott_French wrote: > Ah, these are good questions. > > So, taking a step back, we know that the code involved in serving the image-sugg... [19:41:49] 06serviceops, 10Cassandra: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10046564 (10Eevans) @Cparle how hard would it be to create some functional tests for the extension? Something we could run (even manually/ad-hoc for the time-being) that would exerci... [22:45:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1270.eqiad.wmnet with OS bull... [22:46:21] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046982 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1272.eqiad.wmnet with OS bull... [22:46:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046983 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1273.eqiad.wmnet with OS bull... [22:47:18] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046984 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1274.eqiad.wmnet with OS bull... [22:47:35] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046985 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS bull... [22:47:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046986 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1276.eqiad.wmnet with OS bull... [22:47:50] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046987 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1277.eqiad.wmnet with OS bull... [22:48:06] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1278.eqiad.wmnet with OS bull... [23:02:28] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10046992 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1271.eqiad.wmnet with OS bull... [23:22:43] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047045 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1270.eqiad.wmnet with OS bullseye... [23:28:01] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047046 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1275.eqiad.wmnet with OS bullseye... [23:33:33] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047050 (10Jclark-ctr) [23:33:43] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047051 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1278.eqiad.wmnet with OS bullseye... [23:33:51] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047052 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1277.eqiad.wmnet with OS bullseye... [23:34:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047053 (10Jclark-ctr) [23:34:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047054 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1273.eqiad.wmnet with OS bullseye... [23:38:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047055 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1276.eqiad.wmnet with OS bullseye... [23:38:41] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047056 (10Jclark-ctr) [23:40:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047058 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1279.eqiad.wmnet with OS bull... [23:40:17] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047059 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1280.eqiad.wmnet with OS bull... [23:40:29] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047060 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1281.eqiad.wmnet with OS bull... [23:40:37] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047061 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1282.eqiad.wmnet with OS bull... [23:40:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047062 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1284.eqiad.wmnet with OS bull... [23:41:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047063 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1283.eqiad.wmnet with OS bull... [23:43:32] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047064 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1272.eqiad.wmnet with OS bullseye... [23:43:53] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047065 (10Jclark-ctr) [23:46:14] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047069 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1271.eqiad.wmnet with OS bullseye... [23:46:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047070 (10Jclark-ctr) [23:49:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047082 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1285.eqiad.wmnet with OS bull... [23:49:46] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047083 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host wikikube-worker1286.eqiad.wmnet with OS bull... [23:49:59] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047084 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host wikikube-worker1274.eqiad.wmnet with OS bullseye... [23:50:09] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10047085 (10Jclark-ctr)