[00:04:25] RECOVERY - Webrequests Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:04:25] RECOVERY - Webrequests Varnishkafka log producer on cp3072 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:07:21] RECOVERY - eventlogging Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:07:27] RECOVERY - Webrequests Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:07:27] RECOVERY - statsv Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:08:25] RECOVERY - eventlogging Varnishkafka log producer on cp3070 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:08:25] RECOVERY - statsv Varnishkafka log producer on cp3070 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:08:26] RECOVERY - Webrequests Varnishkafka log producer on cp3070 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:09:21] RECOVERY - Webrequests Varnishkafka log producer on cp3067 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:10:21] RECOVERY - eventlogging Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:10:21] RECOVERY - eventlogging Varnishkafka log producer on cp3072 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:10:27] RECOVERY - statsv Varnishkafka log producer on cp3072 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:10:27] RECOVERY - statsv Varnishkafka log producer on cp3068 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:13:37] RECOVERY - eventlogging Varnishkafka log producer on cp3067 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:13:37] RECOVERY - statsv Varnishkafka log producer on cp3067 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [00:39:43] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change: Design and merge the new tables of file tables - https://phabricator.wikimedia.org/T368113#9912257 (10Bugreporter) >deleted_files Note currently we do not use a table to store deleted pages. One of solutions in {T20493} represents deleted pages u... [01:54:02] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change: Design and merge the new tables of file tables - https://phabricator.wikimedia.org/T368113#9912404 (10Bugreporter) Per T28741#9912401, we may want a new table to stored normalized img_media_type, img_major_mime and img_minor_mime. [02:06:25] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change: Design and merge the new tables of file tables - https://phabricator.wikimedia.org/T368113#9912426 (10Bugreporter) Per parent task needed rows are: file table: * file_id * file_latest * file_name * file_type (normalized type) * file_delete filere... [10:16:33] 06Data-Engineering, 06Data-Platform-SRE: Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#9912923 (10BTullis) >>! In T368033#9911105, @bking wrote: > Just curious, how often do the DAGs change? I think I would go with option 2.2, at least until we've run Ceph long enough to b... [11:10:46] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913034 (10kamila) [11:12:59] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913036 (10kamila) Needs approval from one of: - Olja Dimitrjevic - Dan Andreescu - Will Doran - Andreas Hoelzl... [11:20:11] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913044 (10kamila) [11:20:48] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913045 (10kamila) @cwylo Can you please confirm that you have read the [[ https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Us... [11:21:55] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913049 (10kamila) [11:25:37] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9913062 (10kamila) @leila could you please confirm this access request? Thank you! [12:24:09] (03PS1) 10Btullis: Switch to using the new redacted replica MariaDB server [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1048428 (https://phabricator.wikimedia.org/T365453) [12:33:40] (03CR) 10Ottomata: [C:03+2] Make performer field optional in mediawiki/page/change/1.2.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1047549 (https://phabricator.wikimedia.org/T367923) (owner: 10Ottomata) [12:34:25] (03Merged) 10jenkins-bot: Make performer field optional in mediawiki/page/change/1.2.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1047549 (https://phabricator.wikimedia.org/T367923) (owner: 10Ottomata) [12:37:03] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 13Patch-For-Review: Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9913264 (10Ottomata) I've merged the new schema version and the Ev... [12:46:05] 14Analytics, 06Product-Analytics, 07Epic, 13Patch-For-Review: Data Lake incremental Data Updates - https://phabricator.wikimedia.org/T258511#9913293 (10Ottomata) 05Declined→03Open Being bold and reopening for posterity. [15:53:47] (03PS1) 10Xcollazo: Adding new btm.wikipedia project to pageviews and sqoop groups list. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1048496 (https://phabricator.wikimedia.org/T368038) [16:15:11] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change: Design and merge the new tables of file tables - https://phabricator.wikimedia.org/T368113#9913886 (10VirginiaPoundstone) CC @Ottomata @Milimetric @ahoelzl @lbowmaker @WDoranWMF [16:59:58] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9914019 (10mpopov) @Milimetric Once t... [18:38:26] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change: Design and merge the new tables of file tables - https://phabricator.wikimedia.org/T368113#9914184 (10Milimetric) This might affect some data we sqoop into HDFS and some of how we compute commons impact metrics or similar future metrics. We have... [19:24:13] 06Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505#9914254 (10Ottomata) [19:24:14] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9914255 (10Ottomata) [19:37:23] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9914281 (10Ottomata) Thomas and I had a working session today where we really tried to understand how we can get Refine in Airflow without breaking... [19:58:35] 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25), 13Patch-For-Review: Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9914320 (10Ottomata... [20:03:44] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9914331 (10Milimetric) We get a ton m... [20:08:36] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9914338 (10Ottomata) @gmodena I'm [[ https://phabricator.wikimedia.org/T346046#9903943 | considering a change ]] in EventBus for which... [20:12:14] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9914354 (10Milimetric) The simpler wa... [20:29:40] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Event Platform] Instrument EventBus with prometheus MW Statslib - https://phabricator.wikimedia.org/T363587#9914378 (10Ottomata) Hm, there are [[ https://codesearch.wmcloud.org/search/?q=EventBus&files=&excludeFiles=&repos= | more usages of E... [20:30:37] 10Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Integrate Refine workflow configuration into ESC - https://phabricator.wikimedia.org/T367134#9914386 (10Ottomata) @Antoine_Quhen @JAllemandou let us know what you think! [23:10:45] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products, 10MediaWiki-Special-pages: Statistics - https://phabricator.wikimedia.org/T365952#9914654 (10Filipovic_Zoran) @VirginiaPoundstone Yes.