[00:07:09] 10Data-Engineering-Planning, 10Data Pipelines: Document known data quality issues on Wikistats - https://phabricator.wikimedia.org/T325256 (10odimitrijevic) It is possible to add annotations to the wikistats dashboards via meta. See https://meta.wikimedia.org/w/index.php?title=Config:Dashiki:Annotations/Wikist... [01:12:29] 10Data-Engineering, 10Equity-Landscape: Population input metrics - https://phabricator.wikimedia.org/T309279 (10JAnstee_WMF) @KCVelaga_WMF @ntsako As discussed, for now, we will use a mirrored extract from my supplementation mapping workbook. I created a mirror of only the build 1 data inputs for the 2020 and... [01:49:06] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:00:14] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:42:18] 10Data-Engineering, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) Response: https://lists.apache.org/thread/dont796lp84vfqnovolryw0y0470mqsv > The ap... [03:46:59] 10Data-Engineering, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) @akosiaris I don't love it, but I added a `engine: flink` label to the pods, and am... [06:13:10] (03PS1) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 [06:13:37] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (owner: 10Sharvaniharan) [06:15:08] (03PS2) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions Legacy schema:https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (https://phabricator.wikimedia.org/T324167) [06:15:32] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppSessions Legacy schema:https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [06:18:22] (03PS3) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 [06:18:50] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (owner: 10Sharvaniharan) [06:26:46] (03PS4) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 [06:27:13] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (owner: 10Sharvaniharan) [06:40:19] (03PS1) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881750 [06:41:03] (03Abandoned) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881748 (owner: 10Sharvaniharan) [06:42:13] (03PS2) 10Sharvaniharan: Android schema migration: MobileWikiAppSessions Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881750 (https://phabricator.wikimedia.org/T324167) [06:44:10] (03CR) 10Sharvaniharan: "Please take a look at the migrated schema and review the changes 😊 Thank you." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881750 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [09:37:22] 10Data-Engineering-Planning: Emit lineage information about Airflow jobs to DataHub - https://phabricator.wikimedia.org/T312566 (10Antoine_Quhen) This Gitlab MR https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/204 is: * adding the datahub dependency * adding the configurations in... [09:39:10] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07): Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow - https://phabricator.wikimedia.org/T309552 (10Antoine_Quhen) [09:39:20] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07): Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow - https://phabricator.wikimedia.org/T309552 (10Antoine_Quhen) 05Open→03Resolved [09:39:22] 10Data-Engineering-Kanban, 10Data-Engineering-Planning, 10Data Pipelines: Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10Antoine_Quhen) [12:27:32] 10Data-Engineering, 10Event-Platform Value Stream: [NEEDS GROOMING] Set PYTHONPATH and FLINK_CLASSPATH in Flink docker images. - https://phabricator.wikimedia.org/T327494 (10gmodena) [14:00:38] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Patch-For-Review, 10SecTeam-Processed, 10Vuln-VulnComponent: Upgrade Puppet code to make Airflow configuration files compatible with version 2.3.4 - https://phabricator.wikimedia.org/T315580 (10Antoine_Quhen) Here are the last modifications to... [14:13:17] (03CR) 10Awight: "This change is ready for review." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/880479 (https://phabricator.wikimedia.org/T326637) (owner: 10Awight) [15:37:21] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I always forget which file is the one that was manually created and needs manual review. It's only the current.yaml file, right? That look" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/880479 (https://phabricator.wikimedia.org/T326637) (owner: 10Awight) [16:21:56] 10Quarry: restart quarry processes on schedule - https://phabricator.wikimedia.org/T327522 (10rook) [16:22:24] 10Quarry: restart quarry processes on schedule - https://phabricator.wikimedia.org/T327522 (10rook) [16:22:27] 10Quarry, 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro, 10cloud-services-team (Kanban): [quarry] quarry-web-02 out of memory - https://phabricator.wikimedia.org/T324438 (10rook) [16:22:30] 10Quarry, 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro, 10cloud-services-team (Kanban): [quarry] worker-04 down - https://phabricator.wikimedia.org/T324402 (10rook) [17:11:59] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JTannerWMF) [17:12:25] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JTannerWMF) [18:13:10] (03PS1) 10Sharvaniharan: Android schema migration: MobileWikiAppCreateAccount [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881912 [18:13:58] (03PS2) 10Sharvaniharan: Android schema migration: MobileWikiAppCreateAccount Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppCreateAccount [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881912 (https://phabricator.wikimedia.org/T324167) [18:15:16] (03CR) 10Sharvaniharan: "Please take a look at the migrated schema and review the changes 😊 Thank you." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881912 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [18:43:41] 10Data-Engineering: Improve pageview automated traffic detection heuristics - https://phabricator.wikimedia.org/T280565 (10dr0ptp4kt) [18:43:44] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10dr0ptp4kt) [18:56:23] (03PS1) 10Sharvaniharan: Android schema migration: MobileWikiAppInstallReferrer [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881915 [18:57:11] (03PS2) 10Sharvaniharan: Android schema migration: MobileWikiAppInstallReferrer [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881915 (https://phabricator.wikimedia.org/T324167) [18:58:13] (03CR) 10Sharvaniharan: "Please take a look at the migrated schema and review the changes 😊 Thank you." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881915 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [19:01:46] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JMinor) Historically we used two systems which worked fairly well. I am not sure why those have been removed over time... [19:20:46] (03PS1) 10Sharvaniharan: Android schema migration: MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 [19:21:10] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (owner: 10Sharvaniharan) [19:21:20] (03PS2) 10Sharvaniharan: Android schema migration: MobileWikiAppReadingLists Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (https://phabricator.wikimedia.org/T324167) [19:21:43] (03CR) 10CI reject: [V: 04-1] Android schema migration: MobileWikiAppReadingLists Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [19:22:03] (03PS3) 10Sharvaniharan: Android schema migration: MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 [19:22:55] (03PS4) 10Sharvaniharan: Android schema migration: MobileWikiAppReadingLists Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (https://phabricator.wikimedia.org/T324167) [19:23:44] (03CR) 10Sharvaniharan: "Please take a look at the migrated schema and review the changes 😊 Thank you." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [19:27:10] 10Data-Engineering, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) > 'engine' is the best I came up with, please bikeshed away (if you care?) :) I cha... [19:46:48] 10Data-Engineering, 10Event-Platform Value Stream, 10Product-Analytics: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [19:47:59] (03CR) 10Ottomata: [C: 03+1] Android schema migration: MobileWikiAppCreateAccount Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppCreateAccount [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881912 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [19:49:49] (03CR) 10Shay Nowick: [C: 03+1] Android schema migration: MobileWikiAppReadingLists Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppReadingLists [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881917 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [19:50:45] (03CR) 10Shay Nowick: [C: 03+1] Android schema migration: MobileWikiAppInstallReferrer [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881915 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [21:46:54] (03CR) 10Shay Nowick: [C: 03+1] Android schema migration: MobileWikiAppSessions Legacy schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppSessions [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/881750 (https://phabricator.wikimedia.org/T324167) (owner: 10Sharvaniharan) [21:57:01] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10Milimetric) Quick interlude to mention some specific ERC guidelines for communication. For example, from https://offi... [22:04:20] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10Milimetric) As far as the issue at hand, thank you to Adam for tagging it under the very well known and as yet unprior... [22:07:12] 10Data-Engineering-Planning, 10Data Pipelines: Document known data quality issues on Wikistats - https://phabricator.wikimedia.org/T325256 (10Milimetric) just to have this on record: the Wikistats annotation display system was thrown together quickly. It can be easily modified to include ranges, and it probab... [22:29:21] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JMinor) Amended comment for tone per @Milimetric in response to his interlude. Additionally, the issue is not why ind... [23:06:25] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JTannerWMF) Hi @Milimetric am a bit confused about: > thank you to Adam for tagging it under the very well known as y... [23:12:24] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, and 2 others: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10Jclark-ctr) [23:37:42] 10Data-Engineering, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Artificial views are being displayed in top read feed on apps - https://phabricator.wikimedia.org/T327525 (10JTannerWMF)