[01:21:41] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:22:23] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:31:09] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:33:45] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:51:11] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/826983 (https://phabricator.wikimedia.org/T316457) (owner: 10Gerrit maintenance bot) [07:54:43] !log rerun pageview-hourly-wf-2022-8-28-15 oozie workflow [07:54:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:04:49] !log Rerun refine_eventlogging_legacy failed hours [08:04:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:09:42] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10ntsako) a:05ntsako→03JAnstee_WMF [09:37:21] 10Data-Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10Eric_Luth_WMSE) I would be interested in seeing Swedish added to the tool. [10:17:18] (03CR) 10Kosta Harlan: [C: 03+2] "LGTM! Thank you" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/821711 (https://phabricator.wikimedia.org/T306018) (owner: 10Sergio Gimeno) [10:18:36] (03Merged) 10jenkins-bot: Instrument blocked account registration [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/821711 (https://phabricator.wikimedia.org/T306018) (owner: 10Sergio Gimeno) [10:33:55] 10Data-Engineering, 10Data-Engineering-Operations, 10SRE, 10SRE-Access-Requests: Access request to analytics system(s) for TThoabala - https://phabricator.wikimedia.org/T315409 (10Jelto) @gmodena @Tchanders Can you clarify if `analytics-privatedata-users` is the correct group here? Is this a similar Jupyte... [11:09:27] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): [Shared Event Platform] Mediawiki Stream Enrichment should consume the consolidated page-change stream. - https://phabricator.wikimedia.org/T311084 (10gmodena) [11:09:45] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): [Shared Event Platform] Mediawiki Stream Enrichment should consume the consolidated page-change stream. - https://phabricator.wikimedia.org/T311084 (10gmodena) [11:31:53] (03CR) 10Vivian Rook: [C: 03+2] update XlsxWriter plugin [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/826948 (https://phabricator.wikimedia.org/T314706) (owner: 10Vivian Rook) [11:36:26] (03CR) 10Joal: "Two things, then ready to be merged!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) (owner: 10Snwachukwu) [11:36:53] (03Merged) 10jenkins-bot: update XlsxWriter plugin [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/826948 (https://phabricator.wikimedia.org/T314706) (owner: 10Vivian Rook) [11:44:52] 10Quarry, 10Patch-For-Review: "Download data -> Excel XLSX" corrupted - https://phabricator.wikimedia.org/T314706 (10rook) That's merged and deployed, but I don't think it helped. I'm still seeing the file cut off a little after 2k lines. A CSV seems to get the whole thing. I'll make some time to dig around in... [11:53:30] (03CR) 10Joal: [WIP]Repurpose Refinery-Tool module to contain codes reused across other modules. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) (owner: 10Snwachukwu) [11:57:05] 10Quarry, 10Patch-For-Review: "Download data -> Excel XLSX" corrupted: Cut off after ~2k lines - https://phabricator.wikimedia.org/T314706 (10Aklapper) [12:06:07] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Assess what is required for the enrichment pipline to run on k8 - https://phabricator.wikimedia.org/T315428 (10gmodena) [12:14:39] (03PS4) 10Snwachukwu: Repurpose Refinery-Tool module to contain codes reused across other modules. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) [12:28:45] (03CR) 10Snwachukwu: Repurpose Refinery-Tool module to contain codes reused across other modules. (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) (owner: 10Snwachukwu) [12:31:27] 10Data-Engineering, 10Event-Platform Value Stream: Create a shared flink docker image - https://phabricator.wikimedia.org/T316519 (10dcausse) [12:35:21] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Assess what is required for the enrichment pipline to run on k8 - https://phabricator.wikimedia.org/T315428 (10gmodena) == Spike summary I explored with adjusting the [k8 workshop](https://wikitech.wikimedia.org/wiki/Kubernetes/... [12:39:53] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10ayounsi) [12:58:02] 10Data-Engineering, 10Event-Platform Value Stream, 10Platform Engineering Roadmap Decision Making, 10Platform Team Workboards (S&F Workboard): Need for new event-type - `user_create` and `user_rename` - https://phabricator.wikimedia.org/T262205 (10roman-stolar) a:03roman-stolar [13:00:57] (03CR) 10Joal: [C: 03+2] "LGTM! MErging for tomorrow's deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) (owner: 10Snwachukwu) [13:10:55] (03Merged) 10jenkins-bot: Repurpose Refinery-Tool module to contain codes reused across other modules. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/826599 (https://phabricator.wikimedia.org/T316120) (owner: 10Snwachukwu) [13:27:18] 10Quarry: "Download data -> Excel XLSX" corrupted: Cut off after ~2k lines - https://phabricator.wikimedia.org/T314706 (10rook) There appears to be something about the metadata field with ArthurCovey.jpg https://quarry.wmcloud.org/query/66944 doesn't seem to load it as xlsx [14:10:14] 10Data-Engineering, 10Data³, 10Patch-For-Review: Audit JSON schemas for Gerrit events - https://phabricator.wikimedia.org/T311615 (10hashar) The auditing is done, the json schemas I have generated do validate events when send to a local EventGate. The pending change is https://gerrit.wikimedia.org/r/c/schema... [14:27:27] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Assess what is required for the enrichment pipeline to run on k8 - https://phabricator.wikimedia.org/T315428 (10gmodena) [14:54:22] 10Data-Engineering, 10Event-Platform Value Stream: Use RowTypeInfo to ensure better validation of the event data within the Mediawiki Stream Enrichment pipeline - https://phabricator.wikimedia.org/T316555 (10gmodena) [15:04:06] 10Data-Engineering, 10Data-Engineering-Operations, 10SRE, 10SRE-Access-Requests: Access request to analytics system(s) for TThoabala - https://phabricator.wikimedia.org/T315409 (10gmodena) Hey @Jelto - it's a notebook like the one described in https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter#... [15:16:54] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Assess what is required for the enrichment pipeline to run on k8 - https://phabricator.wikimedia.org/T315428 (10gmodena) [15:17:06] 10Data-Engineering, 10Discovery-Search (Current work): Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Gehel) [15:32:08] mforns: meeting? [15:32:29] 10Data-Engineering, 10Discovery-Search (Current work): Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Gehel) [16:16:09] (03PS1) 10Milimetric: Populate metric dropdown with all valid options [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/827527 [16:24:02] 10Quarry: "Download data -> Excel XLSX" corrupted: Cut off after ~2k lines - https://phabricator.wikimedia.org/T314706 (10rook) a:03rook [16:32:03] mforns: do you have a minute about uniques/ [16:32:04] ? [16:44:45] !log killed mediawiki-history-dumps oozie after migration to airflow [16:44:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:44:49] joal: yes! [16:44:56] mforns: batcave? [16:44:58] ok [17:19:39] 10Data-Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10Isaac) [17:27:42] (03PS1) 10Vivian Rook: strip invalid utf-8 chars for xlsx [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/827538 (https://phabricator.wikimedia.org/T314706) [17:32:24] (03CR) 10CI reject: [V: 04-1] strip invalid utf-8 chars for xlsx [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/827538 (https://phabricator.wikimedia.org/T314706) (owner: 10Vivian Rook) [17:33:42] (03PS2) 10Vivian Rook: strip invalid utf-8 chars for xlsx [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/827538 (https://phabricator.wikimedia.org/T314706) [17:37:42] 10Data-Engineering, 10Research: Update clickstream airflow code to support more languages - https://phabricator.wikimedia.org/T292476 (10Isaac) [17:38:12] 10Data-Engineering, 10Research: Update clickstream code to support more languages - https://phabricator.wikimedia.org/T292476 (10Isaac) [17:40:08] 10Data-Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10Isaac) @JAllemandou I think now would be a good time to revisit this task. To summarize, we last investigated about a year ago and at the time, the Oozie config was deemed a pret... [17:53:13] 10Quarry, 10Patch-For-Review: "Download data -> Excel XLSX" corrupted: Cut off after invalid character - https://phabricator.wikimedia.org/T314706 (10rook) [17:55:06] 10Quarry, 10Patch-For-Review: "Download data -> Excel XLSX" corrupted: Cut off after invalid character - https://phabricator.wikimedia.org/T314706 (10rook) @Novem_Linguae I've put https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/827538 out to test it, seems to work for me now. Is it working for you? It... [18:30:16] 10Analytics-Wikistats, 10Data Engineering Planning, 10Data Pipelines: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717 (10Nevmit) Hi @Milimetric and @EChetty, I guess the problem is still not fixed [20:47:05] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Decide on technical solution for page state stream backfill process - https://phabricator.wikimedia.org/T314389 (10tchin) Ok so to summarize: If the end goal is to have a table with the latest state of every page (plus another o... [21:03:50] 10Data-Engineering: Broken DAG Error when trying to import Gitlab .tgz file into airflow - https://phabricator.wikimedia.org/T316600 (10Htriedman) [21:29:28] (03CR) 10Novem Linguae: [C: 03+1] "Tested, fixes my bug in T314706. Algorithm looks fine. Thank you for taking the time to write 2 patches addressing this issue." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/827538 (https://phabricator.wikimedia.org/T314706) (owner: 10Vivian Rook) [21:31:27] 10Quarry, 10Patch-For-Review: "Download data -> Excel XLSX" corrupted: Cut off after invalid character - https://phabricator.wikimedia.org/T314706 (10Novem_Linguae) +1. Tested, fixes my bug. Algorithm looks fine. Thank you for taking the time to write 2 patches addressing this issue. Out of curiosity, any ide...