[00:03:27] (MediawikiPageContentChangeEnrichTaskManagerNotRunning) resolved: ... [00:03:28] The mw-page-content-change-enrich Flink cluster in codfw has no registered TaskManagers - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=All - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichTaskManagerNotRunning [00:17:28] (MediawikiPageContentChangeEnrichJobManagerNotRunning) firing: ... [00:17:28] mw_page_content_change_enrich in codfw is not running - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichJobManagerNotRunning [00:19:42] (SystemdUnitFailed) firing: produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:21:12] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:22:27] (MediawikiPageContentChangeEnrichJobManagerNotRunning) resolved: ... [00:22:27] mw_page_content_change_enrich in codfw is not running - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichJobManagerNotRunning [00:30:10] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:34:42] (SystemdUnitFailed) resolved: produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:31] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 5.187% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:11:31] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 5.162% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:23:13] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE: [Data Platform] Deploy Spark History Service - https://phabricator.wikimedia.org/T330176 (10brouberol) [09:19:23] 10Data-Engineering, 10Movement-Insights, 10Traffic, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10Vgutierrez) VCL patch submitted by @Ottomata (https://gerrit.wikimedia.org/r/c/operations/puppet/+/981352) looks good to me, @elukey C... [09:27:52] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): ProbeDown - https://phabricator.wikimedia.org/T353065 (10Gehel) [09:28:05] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Run a spark job in test to make sure the history server can see the job data - https://phabricator.wikimedia.org/T352882 (10Gehel) a:05brouberol→03None [09:28:15] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Deploy the spark history services - https://phabricator.wikimedia.org/T352861 (10Gehel) a:05brouberol→03None [09:29:51] gehel: I was actually planning to work on these this week ^ [09:30:40] hello, I'm not sure if you "own" the geoIP db, but we're getting failures on puppetmaster for geoip_update_main.service that is getting 403s when trying to download the updates. (ownership guessed from previous tasks) [09:30:48] so I'm happy to have them assigned to me [09:31:21] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Discovery-Search (Current work), 10Patch-For-Review: Create dashboards for Search SLOs - https://phabricator.wikimedia.org/T338009 (10Gehel) [09:49:59] 10Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 05), 10Technical-Debt: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379 (10phuedx) >>! In T304379#9398410, @... [09:50:44] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Discovery-Search (Current work): Load Wikidata split graphs into test servers - https://phabricator.wikimedia.org/T350465 (10dcausse) Progress: - wdqs1024 (wikidata main): 6.6B triples loaded, processing chunk 885/1023 - wdqs1023 (scholarly articles): 6.3B triple... [10:11:56] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 5.072% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:52:04] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE: Document how to browse the History server locally - https://phabricator.wikimedia.org/T353232 (10brouberol) [11:23:45] (DiskSpace) resolved: Disk space an-test-worker1001:9100:/ 5.008% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:25:25] (KafkaReplicationFactorTooLow) firing: (200) Kafka topic EditorJourney replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [11:31:10] (KafkaReplicationFactorTooLow) resolved: (865) Kafka topic EditorJourney replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [11:31:13] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 5.007% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:56:32] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): Flaky test EventLoggingTest::testDispatch due to time tick within test ru - https://phabricator.wikimedia.org/T353241 (10Umherirrender) [11:58:15] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): Flaky test EventLoggingTest::testDispatch due to time tick within test run - https://phabricator.wikimedia.org/T353241 (10Umherirrender) [11:59:46] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch should - https://phabricator.wikimedia.org/T353243 (10Dreamy_Jazz) [12:00:13] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch fails due to timestamps of events being different by a second - https://phabricator.wikimedia.org/T353243 (10Dreamy_Jazz) [12:06:24] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): Flaky test EventLoggingTest::testDispatch due to time tick within test run - https://phabricator.wikimedia.org/T353241 (10Umherirrender) [12:06:32] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch fails due to timestamps of events being different by a second - https://phabricator.wikimedia.org/T353243 (10Umherirrender) [12:14:17] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Create a superset container image using the PipelineLib framework - https://phabricator.wikimedia.org/T352165 (10CodeReviewBot) btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/8 Test the use of the h... [12:28:21] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE: Document how to browse the History server locally - https://phabricator.wikimedia.org/T353232 (10brouberol) [12:34:30] 10Data-Engineering, 10Movement-Insights, 10Traffic, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10BTullis) Either approach seems fine to me and I don't have strong opinions on which is better. I have +1d the VCL change based on @Ott... [12:34:49] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Create a superset container image using the PipelineLib framework - https://phabricator.wikimedia.org/T352165 (10CodeReviewBot) btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/8 Test the use of the h... [12:49:10] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Run a spark job in test to make sure the history server can see the job data - https://phabricator.wikimedia.org/T352882 (10brouberol) After having merged and deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/980859, we ra... [12:49:18] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Run a spark job in test to make sure the history server can see the job data - https://phabricator.wikimedia.org/T352882 (10brouberol) a:03brouberol [12:49:42] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch fails due to time tick within test run - https://phabricator.wikimedia.org/T353243 (10Dreamy_Jazz) [12:50:08] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Run a spark job in test to make sure the history server can see the job data - https://phabricator.wikimedia.org/T352882 (10brouberol) After opening a couple of TCP ports/destinations, we were able to see the following logs from the h... [12:51:13] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch fails due to time tick within test run - https://phabricator.wikimedia.org/T353243 (10Dreamy_Jazz) [12:52:01] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10ci-test-error (WMF-deployed Build Failure): EventLoggingTest::testDispatch fails when time ticks within the test run - https://phabricator.wikimedia.org/T353243 (10Dreamy_Jazz) [13:03:08] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Create a superset container image using the PipelineLib framework - https://phabricator.wikimedia.org/T352165 (10CodeReviewBot) btullis opened https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/9 Configure the npm pro... [13:08:57] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Run a spark job in test to make sure the history server can see the job data - https://phabricator.wikimedia.org/T352882 (10brouberol) 05Open→03Resolved [13:09:00] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE: [Data Platform] Deploy Spark History Service - https://phabricator.wikimedia.org/T330176 (10brouberol) [13:11:09] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Create a superset container image using the PipelineLib framework - https://phabricator.wikimedia.org/T352165 (10CodeReviewBot) btullis merged https://gitlab.wikimedia.org/repos/data-engineering/superset/-/merge_requests/9 Configure the npm pro... [13:59:50] 10Data-Engineering, 10Movement-Insights, 10Traffic, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10Ottomata) > updating the varnishkafka JSON format Also, this would require schema changes in Hive. So ya let's go with VCL! [14:01:28] 10Data-Engineering, 10Movement-Insights, 10Traffic, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10JAllemandou) > So ya let's go with VCL! +1 [15:22:20] 10Data-Platform-SRE, 10Discovery-Search (Current work): Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10bking) a:03bking [15:31:13] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 4.873% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:48:18] 10Data-Engineering, 10Data Pipelines: [Iceberg] Migrate event_sanitized_iceberg to event_sanitized - https://phabricator.wikimedia.org/T311737 (10Ottomata) We should do {T225751} along the way, getting the semantics of these different databases correct. [16:01:05] 10Data-Platform-SRE, 10Discovery-Search: Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10bking) [16:10:13] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10Gehel) [16:11:13] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10observability, 10Epic: Change data platform-related IRC channels to improve communication - https://phabricator.wikimedia.org/T352783 (10Gehel) [16:17:03] 10Data-Engineering, 10Observability-Logging, 10Traffic: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Milimetric) >>! In T351117#9379025, @Fabfur wrote: > Hi @Milimetric sorry for the late reply, I'll try to answer to your question but consider we're still investig... [16:17:23] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10observability, 10Epic: Change data platform-related IRC channels to improve communication - https://phabricator.wikimedia.org/T352783 (10Gehel) We need to have communication and buy-in from the users of those channels. This is going to start early next year. [16:20:37] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): Expose 3 new dedicated WDQS endpoints - https://phabricator.wikimedia.org/T351650 (10Gehel) [16:21:42] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Create alerts for https://query.wikidata.org/bigdata/ldf - https://phabricator.wikimedia.org/T347355 (10bking) Packet captures from the wdqs1015 host are [[ https://phabricator.wikimedia.org/P54325 | here ]] [16:25:28] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Document how to browse the History server locally - https://phabricator.wikimedia.org/T353232 (10brouberol) [16:28:35] 10Data-Platform-SRE, 10Discovery-Search: Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10Gehel) p:05Triage→03Lowest [16:28:40] 10Data-Platform-SRE, 10Discovery-Search: Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10Gehel) p:05Lowest→03Medium [16:41:10] (03CR) 10Ottomata: [C: 03+2] Update list of scap targets to match where hdfs_tools is deployed [analytics/hdfs-tools/deploy] - 10https://gerrit.wikimedia.org/r/980405 (https://phabricator.wikimedia.org/T336045) (owner: 10Btullis) [16:41:17] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update list of scap targets to match where hdfs_tools is deployed [analytics/hdfs-tools/deploy] - 10https://gerrit.wikimedia.org/r/980405 (https://phabricator.wikimedia.org/T336045) (owner: 10Btullis) [16:43:53] (03Abandoned) 10Ottomata: [TEST] Add logging to test refine race condition [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/957683 (owner: 10Joal) [16:52:27] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10observability: Change data platform-related IRC channels to improve communication - https://phabricator.wikimedia.org/T352783 (10Gehel) [16:59:17] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] eventutilities-python should convert pyflink Instants to python DateTimes - https://phabricator.wikimedia.org/T349640 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requ... [17:00:33] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] eventutilities-python should convert pyflink Instants to python DateTimes - https://phabricator.wikimedia.org/T349640 (10Ottomata) I worked on this while I was on a train during my time off :) [[ https://gitlab.wikimedia.org/repos/da... [17:11:14] (DiskSpace) firing: (2) Disk space an-coord1001:9100:/ 5.704% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:11:31] ^ looking [17:13:11] !log executed `apt clean` on an-coord1001 to free up 7GB. [17:13:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:15:35] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, and 2 others: ORES hook integration with EventBus - https://phabricator.wikimedia.org/T201869 (10Ottomata) 05Open→03Declined I don't think there is any planned work around this. ORES has been deprecated... [17:16:14] (DiskSpace) firing: (2) Disk space an-coord1001:9100:/ 5.608% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:18:08] 10Analytics-Radar, 10Data-Engineering-Icebox, 10MediaWiki-Action-API, 10Patch-For-Review, 10Platform Team Initiatives (Modern Event Platform (TEC2)): Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321 (10Ottomata) Not related to Event Platform. R... [17:24:53] 10Data-Engineering, 10Event-Platform: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Ottomata) [17:24:55] 10Data-Engineering, 10Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10Ottomata) [17:29:23] 10Data-Engineering-Radar, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 05), 10Technical-Debt: Non-deterministic unit test "streamInSample() - session sampling resets" - https://phabricator.wikimedia.org/T304379 (10mpopov) Awesome! Okay, when I pla... [17:43:25] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10pfischer) [17:48:47] (03PS5) 10Ottomata: Fix convertToSchema to work with array of structs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/619034 (https://phabricator.wikimedia.org/T259924) [18:39:45] 10Data-Platform-SRE, 10Discovery-Search, 10GitLab (Project Migration): Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10Aklapper) [19:00:31] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Cirrus-streaming-updater test: validate relforge indices are correctly updated - https://phabricator.wikimedia.org/T350186 (10bking) @pfischer Wanted to ask you since Erik is out for the holidays...what is your level of confidence that the indices are created correct... [19:57:55] 10Analytics, 10Data-Engineering-Icebox, 10Tool-Pageviews: Allow users to query mediarequests using a file page link - https://phabricator.wikimedia.org/T244712 (10Dominicbm) Hi @mforns, sorry for late reply. I think I am not sure how the Commons Impact Metrics project is going to affect the existing AQS APIs... [20:06:28] 10Data-Engineering, 10Data-Catalog: Emit lineage information about Airflow jobs to DataHub - https://phabricator.wikimedia.org/T312566 (10Milimetric) Quick recap for anyone looking to implement lineage. **First**, a note regarding lineage as part of centralized configuration. I think this would be very usefu... [20:32:49] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 9 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10matmarex) [21:16:14] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 4.687% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:16:37] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 9 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10Tgr) > * **Language code**. If using Title::getPageViewLanguage to associate page language outside a... [21:31:17] (KafkaReplicationFactorTooLow) firing: (5) Kafka topic codfw.android.product_metrics.article_link_preview_interaction replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [21:31:53] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10CodeReviewBot) bking opened https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/54 elastic: update plugins version to 7.10.2-10... [21:36:17] (KafkaReplicationFactorTooLow) resolved: (5) Kafka topic codfw.android.product_metrics.article_link_preview_interaction replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [21:55:13] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10CodeReviewBot) bking merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/54 elastic: update plugins version to 7.10.2-10... [22:01:21] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10CodeReviewBot) bking opened https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/55 elasticsearch: add changelog entry [22:01:31] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Update relforge elasticsearch instance extra plugin - https://phabricator.wikimedia.org/T353270 (10CodeReviewBot) bking merged https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/55 elasticsearch: add changelog entry