[00:56:51] (03PS1) 10Neil P. Quinn-WMF: Begin sanitizing Wikistories streams [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832383 (https://phabricator.wikimedia.org/T312262) [00:58:23] (03PS2) 10Neil P. Quinn-WMF: Begin sanitizing Wikistories streams [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832383 (https://phabricator.wikimedia.org/T312262) [01:02:48] (03PS3) 10Neil P. Quinn-WMF: Begin sanitizing Wikistories streams [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832383 (https://phabricator.wikimedia.org/T312262) [01:05:52] (03PS4) 10Neil P. Quinn-WMF: Begin sanitizing Wikistories streams [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832383 (https://phabricator.wikimedia.org/T312262) [05:14:08] !log sudo -u analytics kerberos-run-command analytics refine_eventlogging_legacy --table_include_regex='wikipediaportal' --since='2022-09-13T23:00:00.000Z' --until='2022-09-15T00:00:00.000Z' [05:14:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [05:57:46] PROBLEM - MegaRAID on an-worker1146 is CRITICAL: CRITICAL: 12 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [07:08:10] 10Data-Engineering: Deploy new History snapshot for Wikistats Backend 2022-08 [Ops-week] - https://phabricator.wikimedia.org/T317848 (10Antoine_Quhen) [07:14:47] 10Data-Engineering, 10Patch-For-Review: Deploy new History snapshot for Wikistats Backend 2022-08 [Ops-week] - https://phabricator.wikimedia.org/T317848 (10Antoine_Quhen) The patch is ready. Now an SRE needs to follow the recipe here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS#Deploy_new_Histor... [07:24:47] (Copy from Slack) Hi, using August snapshot on wikistat requires SRE super-powers, plz: https://phabricator.wikimedia.org/T317848 [08:43:49] !log about to deploy analytics/refinery [08:43:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:51:59] 10Data-Engineering: Deploy new History snapshot for Wikistats Backend 2022-08 [Ops-week] - https://phabricator.wikimedia.org/T317848 (10BTullis) a:03BTullis I have deployed the new snapshot and roll-restarted the AQS service. [08:52:11] 10Data-Engineering: Deploy new History snapshot for Wikistats Backend 2022-08 [Ops-week] - https://phabricator.wikimedia.org/T317848 (10BTullis) 05Open→03Resolved [08:58:47] aqu: I'm having a go at building this new airflow package: https://gerrit.wikimedia.org/r/c/operations/debs/airflow/+/832292 but I have a python dependency conflict. Any idea how I might address it? [08:59:03] https://www.irccloud.com/pastebin/VCA5VybF/ [09:02:36] btullis: I'm looking [09:04:07] I can make a paste of the whole of the output of `./build_deb.sh`if that helps. I'm running it on build2001.codfw.wmnet against my branch of the airflow deb repo. [09:05:12] aqu: https://phabricator.wikimedia.org/P34771 [09:31:58] Thanks for the paste [10:14:25] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10KCVelaga_WMF) @JAnstee_WMF once the country-level metrics are verified, regions (sub-continent, continent and global) metrics can be accessed at ` SELECT geography_level, iso2_country_code,... [10:18:29] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10KCVelaga_WMF) regional levels metrics can be accessed at ` SELECT geography_level, iso2_country_code, country_name, continent, sub_continent, commons_monthly_bins, medi... [10:19:55] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) 05Resolved→03Open [10:19:57] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [10:20:09] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) opened the task for data QA check. [10:21:32] 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) [10:24:46] 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) Details for QA: **Transformation** ` WITH readership_input_metrics_hstack AS ( SELECT geography_level, iso2_country_code AS country_code,... [10:25:11] 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) a:05KCVelaga_WMF→03JAnstee_WMF [10:34:06] 10Data-Engineering, 10Observability-Alerting, 10Patch-For-Review, 10User-fgiunchedi: Migrate Kafka prometheus alerts from Icinga to Alertmanager - https://phabricator.wikimedia.org/T309010 (10fgiunchedi) [10:34:23] 10Data-Engineering, 10Observability-Alerting, 10Patch-For-Review, 10User-fgiunchedi: Migrate zookeeper prometheus checks from Icinga to Alertmanager - https://phabricator.wikimedia.org/T309012 (10fgiunchedi) [10:39:51] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) a:05KCVelaga_WMF→03JAnstee_WMF re-opening for data QA check [10:40:16] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) **Transformation** ` WITH editorship_input_metrics_hstack AS ( SELECT geography_level, iso2_country_code AS country_code, CO... [10:41:11] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) Data access: ` SELECT geography_level, country_code, country_name, sub_continent, continent, editor_presence, editor_growth, editorship FROM... [10:41:50] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) 05Resolved→03Open [10:41:52] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [10:42:08] 10Data-Engineering, 10Equity-Landscape: Editorship Output Rank Metrics - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) [12:27:30] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics-Platform, 10Epic: [EPIC] Deprecate mw.eventLog.logEvent() - https://phabricator.wikimedia.org/T317874 (10phuedx) [12:27:44] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics-Platform, 10Epic: [EPIC] Deprecate EventLogging::schemaValidate - https://phabricator.wikimedia.org/T317793 (10phuedx) [12:28:25] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics-Platform, 10Epic: [EPIC] Deprecate mw.eventLog.logEvent() - https://phabricator.wikimedia.org/T317874 (10phuedx) [12:29:03] (03CR) 10Joal: [C: 03+1] "LGTM - Waiting for someone else to give opinion :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832336 (https://phabricator.wikimedia.org/T317124) (owner: 10Xcollazo) [12:51:38] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Metrics-Platform, 10Epic: [EPIC] Deprecate mw.eventLog.logEvent() - https://phabricator.wikimedia.org/T317874 (10phuedx) [14:23:08] 10Analytics-Kanban, 10Data-Engineering, 10Patch-For-Review, 10User-MoritzMuehlenhoff: Automate kerberos credential creation and management to ease the creation of testing infrastructure - https://phabricator.wikimedia.org/T292389 (10MoritzMuehlenhoff) [16:41:33] 10Analytics, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0 documentation - https://phabricator.wikimedia.org/T288664 (10apaskulin) [16:43:28] 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10Milimetric) @BCornwall: part of the kerberos credential setup used to be that an email was sent to Hannah with instructions on how to reset a temporary password (https://wikitec... [16:45:27] (03PS1) 10Phuedx: Remove ChangesList* allowlist entries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832524 (https://phabricator.wikimedia.org/T317525) [16:56:35] 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BCornwall) Hi, @Milimetric. No need for a new email, I consider that to be part of the same issue. I'm assuming Hannah checked their spam folder? :) @BTullis when you kindly se... [17:08:08] 10Analytics, 10API Platform, 10Code-Health-Objective: Synchronize .gitignore files - https://phabricator.wikimedia.org/T315113 (10VirginiaPoundstone) [17:08:15] 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BTullis) >>! In T317545#8240048, @BCornwall wrote: > Hi, @Milimetric. No need for a new email, I consider that to be part of the same issue. I'm assuming Hannah checked their sp... [17:50:55] 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10Hokwelum) I'm sorry @Milimetric, @BCornwall, and @BTullis, I didn’t check my spam earlier but I just saw the mail now :-) [18:09:10] 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10Milimetric) Ok, cool. After kinit-ing successfully I noticed Hannah is now in analytics-admin but not analytics-privatedata-users. She needs both of those the way it's current... [19:54:50] 10Data-Engineering, 10Product-Analytics: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10Milimetric) I did not know this was left behind in the migration to #data-engineering. I recently talked to Olja and mentioned that I want to use this task to show... [19:56:02] 10Data-Engineering, 10Product-Analytics: Creation of canonical pageview dumps for users to download - https://phabricator.wikimedia.org/T251777 (10Milimetric) [19:56:41] 10Data-Engineering, 10Patch-For-Review: Migrate pagecounts-ez generation to hadoop - https://phabricator.wikimedia.org/T192474 (10Milimetric) 05Declined→03Open re-opening per T251777#8240584 [21:16:29] 10Data-Engineering, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BCornwall) Okay, @Milimetric, that should be solved. She's in analytics-privatedata-users now. Would you say that's everything needed? [23:01:15] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10JAnstee_WMF) @ntsako looking good for the 2021 counts - There is some slight difference which I believe is due to pulling avg vs sum and then taking average but I am not sure where I can see your cod... [23:03:42] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10JAnstee_WMF) @KCVelaga_WMF I get an access denied permission failure on superset, should I be running this elsewhere or is there a permission setting I need changed?