[05:59:56] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9788813 (10daniel) >>! In T120242#9787880, @Ottomata wrote: > QQ: is there a corresponding status for replica inconsistency, when lag is close to 0? Are... [08:51:32] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9789261 (10Antoine_Quhen) [09:06:24] 10Data-Engineering (Q4 2024 April 1st - June 30th), 13Patch-For-Review: [Refine refactoring] Extract refine schema management into a dedicated tool - https://phabricator.wikimedia.org/T356762#9789320 (10Antoine_Quhen) I've encountered an issue with our current production Airflow setup where the scheduler is no... [09:12:25] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9789336 (10Ladsgroup) FWIW, for each section we have a master and a "candidate master". We always swap them when we need to do maintenance. So there is n... [09:31:47] (03CR) 10Gmodena: Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [09:46:19] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9789522 (10gmodena) Thi... [11:53:05] 10Quarry, 10ChangeProp, 06collaboration-services, 06Infrastructure-Foundations, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9789858 (10MoritzMuehlenhoff) Redict is now packaged in Debian: https://tracker.debian.org/pkg/re... [11:53:46] 10Quarry, 10ChangeProp, 06collaboration-services, 06Infrastructure-Foundations, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9789859 (10MoritzMuehlenhoff) [12:29:05] FIRING: KafkaReplicationFactorTooLow: ... [12:29:11] Kafka topic codfw.mediawiki.job.securePollUnarchiveElection replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=codfw.mediawiki.job.securePollUnarchiveElection&viewPanel=40 - ... [12:29:11] https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [12:34:05] RESOLVED: KafkaReplicationFactorTooLow: ... [12:34:05] Kafka topic codfw.mediawiki.job.securePollUnarchiveElection replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://grafana.wikimedia.org/d/000000234/kafka-by-topic?var-kafka_cluster=jumbo-eqiad&var-kafka_broker=All&var-topic=codfw.mediawiki.job.securePollUnarchiveElection&viewPanel=40 - ... [12:34:05] https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [13:45:00] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#9790310 (10JAllemandou) Hi @XiaoXiao-WMF , the impact of this issue is that the hive `wmf.mediawiki_wikitext_history` is currently not containing the `w... [15:35:35] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470#9790810 (10rook) 05Open→03Resolved [15:36:44] 10Quarry: remove buster systems - https://phabricator.wikimedia.org/T364753 (10rook) 03NEW [15:36:49] 10Quarry: remove buster systems - https://phabricator.wikimedia.org/T364753#9790830 (10rook) [15:36:52] 10Quarry: Shutdown quarry VMs - https://phabricator.wikimedia.org/T361470#9790831 (10rook) [16:24:16] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10GitLab (Pipeline Services Migration🐤): Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9791109 (10Snwachukwu) a:03Snwachukwu [18:07:34] elukey: around? [18:33:07] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator - https://phabricator.wikimedia.org/T361853#9791692 (10Ottomata) >... [18:55:30] 06Data-Engineering, 10EventStreams, 10Event-Platform: eventgate: eventstreams: services should use common logging schema - https://phabricator.wikimedia.org/T347498#9791832 (10Ottomata) @tchin, if we are able to get off of old service-runner, would your new framework take care of this? [19:00:54] 06Data-Engineering: Airflow DAG (hdfs_usage_weekly) failed with no details in the application log - https://phabricator.wikimedia.org/T364487#9791854 (10amastilovic) >>! In T364487#9786155, @BTullis wrote: > What's the longer-term location for the log4j properties file name? Presumably we don't want to leave the... [19:09:29] 06Data-Engineering, 10EventStreams, 10Event-Platform: eventgate: eventstreams: services should use common logging schema - https://phabricator.wikimedia.org/T347498#9791862 (10tchin) >>! In T347498#9791831, @Ottomata wrote: > @tchin, if we are able to get off of old service-runner, would your new framework t... [19:16:39] (03PS1) 10Snwachukwu: Refine DeequColumnAnalysis code [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1031049 [19:17:01] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Upgrade MediawikiHistory Checker to use AWS Deequ. 1. Update User history checker 2. Update Page history checker 3. Update Denormalized hist (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1024423 (https://phabricator.wikimedia.org/T361016) (owner: 10Snwachukwu) [19:46:45] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9792008 (10Ottomata) > So there is no real "source of truth". > So it is quite possible that we might even lose canonical data due to inconsistencies be... [19:48:08] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Eventually Consistent MediaWiki State Change Events - https://phabricator.wikimedia.org/T120242#9792011 (10Ottomata) > But not all our code is written to be defensive against that situation. @daniel what if somehow a page delete is missed in a repl... [19:49:58] 10Analytics-Canonical-Data, 06Movement-Insights, 06Product-Analytics: Create a structured list of Wikimedia projects' creation and closure dates - https://phabricator.wikimedia.org/T336999#9792016 (10nshahquinn-wmf) [19:51:32] 06Data-Engineering, 10EventStreams, 10Event-Platform: eventgate: eventstreams: services should use common logging schema - https://phabricator.wikimedia.org/T347498#9792019 (10Ottomata) 05Open→03Declined Awesome, I think we should just decline this task then and focus on getting rid of service-runner. [19:58:00] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9792043 (10Jdforrester-WMF) [22:45:11] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9792633 (10apaskulin)