[00:00:37] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Event Platform - Proof of Concept - Enriched Edit History Message Creation - https://phabricator.wikimedia.org/T302834 (10lbowmaker) 05Open→03Resolved Resolving - work is now being tracked under the event platform value stream: https://... [00:25:57] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:31:13] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:40:37] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [01:22:40] PROBLEM - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [03:06:59] (03CR) 10Gergő Tisza: [C: 03+1] helppanel: Add support for trynewtask dialog [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/893734 (https://phabricator.wikimedia.org/T330637) (owner: 10Kosta Harlan) [04:15:24] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [04:47:52] PROBLEM - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [06:56:07] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [07:00:31] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 10 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) m1-master and m2-master proxies failed over [07:01:11] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 10 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [07:07:39] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [07:47:37] !log restart test jobhistory to take in account: https://gerrit.wikimedia.org/r/c/operations/puppet/+/894481 [07:47:54] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:50:37] PROBLEM - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [07:56:19] !log restart prod jobhistory to take in account: https://gerrit.wikimedia.org/r/c/operations/puppet/+/894481 [07:56:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:14:57] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10MoritzMuehlenhoff) [08:32:26] (03CR) 10Gergő Tisza: [C: 03+2] helppanel: Add support for trynewtask dialog [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/893734 (https://phabricator.wikimedia.org/T330637) (owner: 10Kosta Harlan) [08:33:05] (03Merged) 10jenkins-bot: helppanel: Add support for trynewtask dialog [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/893734 (https://phabricator.wikimedia.org/T330637) (owner: 10Kosta Harlan) [08:58:37] !log restart test resourcemanager to take in account new dedicated exclude file [08:58:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:59:54] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [09:00:22] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [09:01:58] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [09:13:05] !log restart prod resourcemanager to take in account new dedicated exclude file [09:13:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:36:09] !log restart test hiveserver2: T303168 [09:36:12] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:36:12] T303168: Investigate trend of gradual hive server heap exhaustion - https://phabricator.wikimedia.org/T303168 [09:38:35] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 09): Investigate trend of gradual hive server heap exhaustion - https://phabricator.wikimedia.org/T303168 (10nfraison) What we can see from graph is that the leak in the OldGC is linked to the link of Meta... [09:47:29] !log restart namenode in an-test-master1001 to take in account new quota init threads setting [09:47:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:47:56] ^this is the current standby NN in test [09:52:31] !log failover namenode in test from an-test-master1002-eqiad-wmnet to an-test-master1001-eqiad-wmnet [09:52:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:53:40] !log restart namenode in an-test-master1002 to take in account new quota init threads setting [09:53:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:59:53] !log restart namenode in an-master1001 (standby in prod) to take in account new quota init threads setting [09:59:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:02:45] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Clement_Goubert) [10:25:24] !log failover namenode in prod from an-master1002-eqiad-wmnet to an-master1001-eqiad-wmnet [10:25:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:32:11] 10Data-Engineering, 10Event-Platform Value Stream, 10MediaWiki-extensions-WikimediaEvents, 10Product-Analytics, 10Technical-Debt: Decommission the EditorActivation instrument - https://phabricator.wikimedia.org/T330766 (10phuedx) >>! In T330766#8655569, @phuedx wrote: > @Mayakp.wiki also pointed out that... [10:36:16] !log restart namenode in an-master1002 to take in account new quota init threads setting [10:36:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:51:39] Starting build #118 for job analytics-refinery-maven-release-docker [10:58:08] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10cmooney) 05Open→03Resolved [11:05:21] Project analytics-refinery-maven-release-docker build #118: 09SUCCESS in 13 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/118/ [11:08:40] Starting build #77 for job analytics-refinery-update-jars-docker [11:08:58] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.13 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/895729 [11:08:58] Project analytics-refinery-update-jars-docker build #77: 09SUCCESS in 18 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/77/ [11:10:53] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [11:27:24] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [11:27:38] (03CR) 10Ottomata: [V: 03+2 C: 03+2] Add refinery-source jars for v0.2.13 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/895729 (owner: 10Maven-release-user) [11:45:17] RECOVERY - Hadoop HDFS Namenode FSImage Age on an-master1002 is OK: FILE_AGE OK: /srv/hadoop/name/current/VERSION is 90 seconds old and 217 bytes https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [11:54:07] PROBLEM - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [11:54:10] !log Deployed refinery using scap, then deployed onto hdfs [11:54:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:00:47] 10Data-Engineering, 10Data Pipelines, 10Patch-For-Review: Install conda-analytics on Airflow servers - https://phabricator.wikimedia.org/T331345 (10Ottomata) Done. [12:41:35] 10Data-Engineering, 10Event-Platform Value Stream: eventutilities-python should support using Kafka TLS ports - https://phabricator.wikimedia.org/T331526 (10Ottomata) [12:41:55] 10Data-Engineering, 10Event-Platform Value Stream: eventutilities-python should support using Kafka TLS ports - https://phabricator.wikimedia.org/T331526 (10Ottomata) [12:41:58] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment - https://phabricator.wikimedia.org/T328478 (10Ottomata) [14:03:01] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09), 10Patch-For-Review: Streaming services errors should be routed to an error event topic. - https://phabricator.wikimedia.org/T326536 (10JArguello-WMF) [14:21:54] 10Data-Engineering-Planning, 10Event-Platform Value Stream: EventStreamCatalog removes 'topic' table option if connector = upsert-kafka - https://phabricator.wikimedia.org/T330769 (10JArguello-WMF) [14:29:12] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Flink EventStreamCatalog should add watermark - https://phabricator.wikimedia.org/T330441 (10JArguello-WMF) [14:36:27] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [15:01:51] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10Ladsgroup) >>! In T329071#8662812, @elukey wrote: > Thanks a... [15:02:32] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09): Add event dt field to error event schema - https://phabricator.wikimedia.org/T330918 (10Ottomata) [15:05:28] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 09): eventutilities-python should support using Kafka TLS ports - https://phabricator.wikimedia.org/T331526 (10JArguello-WMF) [15:06:14] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) [15:08:59] PROBLEM - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [15:09:40] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) @calbon This is probably a task that Event Platform will do, unless you want @AikoCh... [15:10:52] 10Data-Engineering-Planning, 10Event-Platform Value Stream: jsonschema-tools tests - ensure that array items type is set - https://phabricator.wikimedia.org/T329515 (10JArguello-WMF) [15:11:41] 10Data-Engineering-Planning, 10Event-Platform Value Stream: jsonschema-tools tests - ensure that array items type is set - https://phabricator.wikimedia.org/T329515 (10JArguello-WMF) p:05Triage→03Medium [15:12:17] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10JArguello-WMF) p:05Triage→03High [15:14:24] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10Ottomata) [15:14:37] 10Data-Engineering-Planning, 10SRE-swift-storage, 10Event-Platform Value Stream (Sprint 10): Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10Ottomata) [15:14:54] 10Data-Engineering-Planning, 10SRE-swift-storage, 10Event-Platform Value Stream (Sprint 10): Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10JArguello-WMF) [15:16:57] 10Data-Engineering, 10Machine-Learning-Team, 10Research, 10Event-Platform Value Stream (Sprint 10): Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10JArguello-WMF) [15:18:37] 10Data-Engineering-Planning, 10Event-Platform Value Stream: EventStreamCatalog removes 'topic' table option if connector = upsert-kafka - https://phabricator.wikimedia.org/T330769 (10JArguello-WMF) a:03tchin [15:18:46] 10Data-Engineering-Planning, 10Event-Platform Value Stream: EventStreamCatalog removes 'topic' table option if connector = upsert-kafka - https://phabricator.wikimedia.org/T330769 (10Ottomata) a:05tchin→03None [15:21:53] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 10): Flink EventStreamCatalog should add watermark - https://phabricator.wikimedia.org/T330441 (10JArguello-WMF) [15:23:00] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10cmooney) [15:23:34] 10Data-Engineering, 10Event-Platform Value Stream, 10serviceops-radar: Store Flink HA metadata in Zookeeper for mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T331283 (10JArguello-WMF) p:05Triage→03High [15:23:45] 10Data-Engineering, 10serviceops-radar, 10Event-Platform Value Stream (Sprint 10): Store Flink HA metadata in Zookeeper for mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T331283 (10JArguello-WMF) [15:25:22] 10Data-Engineering-Planning, 10DBA, 10Data Pipelines, 10Infrastructure-Foundations, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10cmooney) [15:26:28] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 11): Refactor parameterization of eventutilities-python and mediawiki-event-enrichment - https://phabricator.wikimedia.org/T328478 (10JArguello-WMF) [15:38:04] 10Data-Engineering-Planning, 10Event-Platform Value Stream: EventStreamCatalog removes 'topic' table option if connector = upsert-kafka - https://phabricator.wikimedia.org/T330769 (10Ottomata) [15:40:49] 10Data-Engineering, 10Event-Platform Value Stream: EventStreamCatalog should not remove manually specified user options in CREATE TABLE STATEMENTS - https://phabricator.wikimedia.org/T331542 (10Ottomata) [15:41:18] 10Data-Engineering, 10Event-Platform Value Stream: EventStreamCatalog should not remove user specified options in CREATE TABLE statements - https://phabricator.wikimedia.org/T331542 (10Ottomata) [15:57:03] 10Data-Engineering, 10SRE, 10ops-eqiad: Degraded RAID on an-worker1132 - https://phabricator.wikimedia.org/T330971 (10Cmjohnson) 05Open→03Resolved The disk has been swapped and back online. I am resolving this task and creating a new one for the BBU. [15:58:05] RECOVERY - MegaRAID on an-worker1132 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [15:59:07] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09), 10Patch-For-Review: Streaming services errors should be routed to an error event topic. - https://phabricator.wikimedia.org/T326536 (10Ottomata) We have our first real error!!! `kafkacat -C -u -b kafka-jumbo1006.eqiad.wmnet:9092 -t eq... [15:59:28] 10Data-Engineering, 10SRE, 10ops-eqiad: Degraded RAID on an-worker1132 - https://phabricator.wikimedia.org/T330971 (10nfraison) Strangely since the change of disk everything is back to normal ` RECOVERY - MegaRAID on an-worker1132 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech... [16:13:49] ACKNOWLEDGEMENT - MegaRAID on an-worker1078 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough Btullis Filed T331544 to replace BBU https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [16:15:35] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10ntsako) 05Open→03Resolved [16:15:37] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10ntsako) [16:23:38] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 09): Investigate slownesses on an-worker1132 - https://phabricator.wikimedia.org/T330979 (10nfraison) Adding back node as all paritions are available and disks cache is back [16:24:01] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10elukey) >>! In T329071#8676678, @Ladsgroup wrote: >>>! In T3... [16:31:23] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10Ottomata) > wouldn't be able to populate the Mariadb table t... [16:37:02] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10diego) >The problem that I see with 1) is that we are alread... [16:37:33] 10Data-Engineering, 10Machine-Learning-Team, 10Research, 10Event-Platform Value Stream (Sprint 10): Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10achou) +1 having separate schemas for classification models, embeddings, and recommendat... [16:45:55] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10Ladsgroup) >>! In T329071#8676986, @elukey wrote: > The prob... [17:02:47] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10Ottomata) > much simpler via the suggested idea. I might be... [17:27:33] 10Data-Engineering-Planning, 10Edit-Review-Improvements-Integrated-Filters, 10Event-Platform Value Stream, 10Growth-Team, and 2 others: Integration of Revert Risk Scores to Recent Changes as a filter - https://phabricator.wikimedia.org/T329071 (10elukey) >>! In T329071#8677053, @Ladsgroup wrote: >>>! In T3... [17:27:37] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09), 10Patch-For-Review: Streaming services errors should be routed to an error event topic. - https://phabricator.wikimedia.org/T326536 (10Ottomata) I investigated a couple of the errors. One seems to be because the page was deleted before... [17:37:30] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10Mayakp.wiki) @ntsako / @JAnstee_WMF : can you provide the database name and table name of the production version of this table? [18:19:58] 10Data-Engineering, 10Machine-Learning-Team, 10Research, 10Event-Platform Value Stream (Sprint 10): Design event schema for ML scores/recommendations on current page state - https://phabricator.wikimedia.org/T331401 (10Ottomata) > Moreover, the explainer functionality is still in the exploratory phase in L... [21:25:35] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10JAnstee_WMF) @Mayakp.wiki This table is now in production in gdi.country_meta_data [21:44:42] RECOVERY - MegaRAID on an-worker1078 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [22:55:13] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10Mayakp.wiki) great, thanks @JAnstee_WMF should we use this new table for T330780 instead of `ntsako.country_meta_data` ?