[00:04:13] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:04:15] (DiskSpace) firing: (58) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.332% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:14:14] (DiskSpace) firing: (57) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.445% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:19:14] (DiskSpace) firing: (59) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.442% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:29:14] (DiskSpace) firing: (54) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.448% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [00:34:13] (DiskSpace) firing: (54) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.413% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:34:14] (DiskSpace) firing: (48) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 4.546% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:39:14] (DiskSpace) firing: (48) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 4.955% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:44:15] (DiskSpace) resolved: (48) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 4.955% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:46:28] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [04:04:27] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:36:13] (DiskSpace) firing: Disk space an-worker1152:9100:/var/lib/hadoop/data/c 5.496% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-worker1152 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:41:13] (DiskSpace) firing: (9) Disk space an-worker1152:9100:/var/lib/hadoop/data/b 5.618% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-worker1152 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:46:13] (DiskSpace) firing: (13) Disk space an-worker1114:9100:/var/lib/hadoop/data/k 5.484% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:51:13] (DiskSpace) firing: (25) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.739% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:56:13] (DiskSpace) firing: (26) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.706% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:01:13] (DiskSpace) firing: (28) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.674% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:11:13] (DiskSpace) firing: (29) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.576% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:16:13] (DiskSpace) firing: (31) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.556% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:26:13] (DiskSpace) firing: (33) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.505% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:36:13] (DiskSpace) firing: (34) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 5.217% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:36:54] (03PS6) 10TChin: Add iceberg version of aqs_hourly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) [05:46:13] (DiskSpace) firing: (35) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 4.999% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:51:13] (DiskSpace) firing: (36) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 4.985% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:11:47] (03CR) 10TChin: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [08:04:28] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:51:28] (DiskSpace) firing: (36) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.389% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:19:45] looking into an-worker1114 firing the alerts [10:21:46] stevemunene: Good stuff. It might be related to this conversation from yesterday. https://wikimedia.slack.com/archives/C02291Z9YQY/p1704302496823159 [10:22:33] i.e. large spark shuffles, creating big files in local appcache directories on the workers. Not certain, but worth knowing about. [10:29:15] ack, seen they auto clear once done [11:04:17] (KafkaReplicationFactorTooLow) firing: ... [11:04:17] Kafka topic eqiad.mediawiki.interwiki_link_hover replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=eqiad.mediawiki.interwiki_link_hover&viewPanel=40 - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [11:09:17] (KafkaReplicationFactorTooLow) resolved: ... [11:09:17] Kafka topic eqiad.mediawiki.interwiki_link_hover replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=eqiad.mediawiki.interwiki_link_hover&viewPanel=40 - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [11:58:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [12:03:52] 10Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817 (10phuedx) >>! In T353817#9433796, @Krinkle wrote: > 2. Volume. > > What's the traffic like?... [12:04:28] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:20] 10Data-Platform-SRE, 10Patch-For-Review: Bring an-mariadb100[12] into service - https://phabricator.wikimedia.org/T284150 (10BTullis) >>! In T284150#9348488, @jcrespo wrote: >> BTullis closed this task as Resolved. > > > What about? > >> [ ] Ensure backups are running on the right host(s) and with the lates... [12:22:31] 10Data-Platform-SRE, 10Patch-For-Review: Bring an-mariadb100[12] into service - https://phabricator.wikimedia.org/T284150 (10BTullis) [12:31:24] 10Data-Engineering, 10Product-Analytics, 10Wmfdata-Python: Retrieve host & port info when connecting to MariaDB replicas on the cluster - https://phabricator.wikimedia.org/T340472 (10BTullis) There are some example python scripts here, which show how to use DNS calls to look up the host and port number for a... [13:51:29] (DiskSpace) firing: (36) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 3.038% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:30:15] 10Data-Engineering, 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Capacity warnings on an-worker1127 - https://phabricator.wikimedia.org/T354291 (10BTullis) 05Open→03Resolved OK, resolving this ticket. We know the cause, we just need to be careful with overall capacity. [14:31:13] (DiskSpace) firing: (37) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.716% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:38:36] (03PS2) 10Phuedx: Add trv.wikisource and ab.wikibooks to Sqoop and pageview allowlists [analytics/refinery] - 10https://gerrit.wikimedia.org/r/972348 [14:39:33] (03CR) 10Phuedx: Add trv.wikisource and ab.wikibooks to Sqoop and pageview allowlists (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/972348 (owner: 10Phuedx) [14:46:13] (DiskSpace) firing: (40) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.575% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:46:54] 10Data-Engineering, 10Product-Analytics, 10Patch-For-Review: Propagate field descriptions from event schemas to Hive event tables - https://phabricator.wikimedia.org/T307040 (10Milimetric) Datahub allows you to add descriptions at sub-field level. We should at some point get to consensus about where we want... [14:51:13] (DiskSpace) firing: (40) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.505% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:56:13] (DiskSpace) firing: (41) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.415% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:57:14] 10Data-Engineering, 10Movement-Insights, 10Traffic, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10Milimetric) @Mayakp.wiki the patch to watch is: https://gerrit.wikimedia.org/r/c/operations/puppet/+/981352/. This has not yet been m... [15:01:13] (DiskSpace) firing: (40) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.36% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:06:13] (DiskSpace) firing: (41) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.36% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:10:07] (03Abandoned) 10Amire80: Add Arabic and Urdu [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/927231 (owner: 10Amire80) [15:11:13] (DiskSpace) firing: (42) Disk space an-worker1114:9100:/var/lib/hadoop/data/b 2.272% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:16:13] (DiskSpace) firing: (46) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.895% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:16:42] 10Data-Platform-SRE (2023/24 Q3 Milestone 1), 10Patch-For-Review: Create helmfile deployment files for superset and superset-next - https://phabricator.wikimedia.org/T353790 (10BTullis) a:03BTullis [15:21:14] (DiskSpace) firing: (51) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.794% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:26:51] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [15:28:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [15:36:13] (DiskSpace) firing: (54) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.722% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:38:48] 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Bring an-worker11[57-75] into service - https://phabricator.wikimedia.org/T353776 (10BTullis) [15:41:13] (DiskSpace) firing: (55) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.694% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:46:13] (DiskSpace) firing: (58) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.668% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:51:13] (DiskSpace) firing: (61) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.596% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:56:14] (DiskSpace) firing: (63) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.562% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:01:13] (DiskSpace) firing: (63) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.562% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:04:28] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:15] (DiskSpace) resolved: (63) Disk space an-worker1085:9100:/var/lib/hadoop/data/c 5.562% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:09:21] (HdfsCapacityRemainingPercent) resolved: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [17:49:35] (03CR) 10Xcollazo: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [20:04:28] (SystemdUnitFailed) firing: (2) wmf_auto_restart_prometheus-mysqld-exporter@staging.service Failed on dbstore1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:08:33] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) firing: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [20:08:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [20:37:39] 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Restart Search Platform-owned services for Java 8 / Java 11 security updates - https://phabricator.wikimedia.org/T350703 (10RKemper) [20:40:20] (03PS7) 10TChin: Add iceberg version of aqs_hourly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) [20:41:19] (03CR) 10TChin: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [20:51:05] 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Restart Search Platform-owned services for Java 8 / Java 11 security updates - https://phabricator.wikimedia.org/T350703 (10RKemper) @MoritzMuehlenhoff This should be all done. Let us know if you see any rogue java processes hanging around! [21:08:33] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) resolved: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [21:08:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [21:12:54] 10Analytics, 10Data-Engineering, 10Data-Engineering-Wikistats, 10Data Products: Wikistats - incorrect number of content articles for Latvian Wikipedia - https://phabricator.wikimedia.org/T354074 (10Milimetric) TL;DR; the data pipeline up to AQS seems fine, my guess is we're not filtering properly to exclud... [21:16:53] 10Data-Platform-SRE: Service implementation for elastic2087-2109 - https://phabricator.wikimedia.org/T353878 (10RKemper) `elastic2087` has joined the cluster as a bullseye host. I haven't officially pooled it yet. Next up (after pooling `elastic2087`) will be trying to add 1 bookworm host to the fleet and seein... [21:28:01] 10Analytics, 10Data-Engineering, 10Data-Engineering-Wikistats, 10Data Products: Wikistats - incorrect number of content articles for Latvian Wikipedia - https://phabricator.wikimedia.org/T354074 (10VirginiaPoundstone) p:05Triage→03High [21:29:51] 10Analytics, 10Data-Engineering, 10Data-Engineering-Wikistats, 10Data Products (Data Products Sprint 05): Wikistats - incorrect number of content articles for Latvian Wikipedia - https://phabricator.wikimedia.org/T354074 (10VirginiaPoundstone) [23:00:38] 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Ensure Elastic stack works on bookworm - https://phabricator.wikimedia.org/T353392 (10bking) a:03bking [23:06:46] 10Data-Platform-SRE (2023/24 Q3 Milestone 1), 10Patch-For-Review: Ensure Elastic stack works on bookworm - https://phabricator.wikimedia.org/T353392 (10bking)