[00:06:28] <icinga-wm>	 RECOVERY - Check unit status of drop_event on an-launcher1002 is OK: OK: Status of the systemd unit drop_event https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[00:20:20] <icinga-wm>	 PROBLEM - Check unit status of refinery-drop-webrequest-raw-partitions on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-drop-webrequest-raw-partitions https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[00:20:34] <icinga-wm>	 PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[00:32:06] <icinga-wm>	 RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[01:39:52] <icinga-wm>	 PROBLEM - Check unit status of drop_event on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit drop_event https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:01:24] <icinga-wm>	 RECOVERY - Check unit status of drop-anomaly-detection on an-launcher1002 is OK: OK: Status of the systemd unit drop-anomaly-detection https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:01:28] <icinga-wm>	 RECOVERY - Check unit status of refinery-drop-banner-activity on an-launcher1002 is OK: OK: Status of the systemd unit refinery-drop-banner-activity https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[03:06:56] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10BPirkle)
[04:41:08] <icinga-wm>	 PROBLEM - Check unit status of monitor_refine_event_sanitized_main_immediate on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_sanitized_main_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[05:08:26] <wikibugs>	 (03PS1) 10Gergő Tisza: analytics/legacy/helppanel: Add new actions related to mid-edit signup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/828676 (https://phabricator.wikimedia.org/T310320)
[05:09:59] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Decide on technical solution for page state stream backfill process - https://phabricator.wikimedia.org/T314389 (10tchin)
[07:32:15] <wikibugs>	 10Data-Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10Aroraakhil) Thanks @Isaac for restarting conversation on this  thread!  @JAllemandou @EChetty, I would also like  to point out a specific research project that was carried out la...
[07:51:54] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM - Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/828633 (https://phabricator.wikimedia.org/T314578) (owner: 10CDanis)
[07:54:53] <wikibugs>	 10Data-Engineering, 10Patch-For-Review: Add the requestctl element of the x-analytics map to turnlio's webrequest_sampled_128 - https://phabricator.wikimedia.org/T314578 (10JAllemandou) Path merged, next deploy should be around Tuesday next week. I've asked for a rerun of the oozie jobs for the past 4 weeks.
[09:31:27] <wikibugs>	 (03PS2) 10Joal: Update cassandra hql loading file [analytics/refinery] - 10https://gerrit.wikimedia.org/r/828518 (https://phabricator.wikimedia.org/T311507)
[09:32:58] <wikibugs>	 (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/828518 (https://phabricator.wikimedia.org/T311507) (owner: 10Joal)
[09:34:30] <joal>	 Hey mforns - I'm doing a dry-run for drop_event right now
[09:34:50] <mforns>	 ok, it took more than 1 hour yesterday for me...
[09:37:25] <joal>	 np mforns - go on day off, I'll manage that - thank you again for running all the other instance!
[09:37:34] <mforns>	 :thumbsu
[09:37:45] <mforns>	 👍
[09:56:11] <wikibugs>	 (03PS4) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[10:02:26] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] Fix Array UDFs (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566 (owner: 10Nmaphophe)
[12:10:27] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01), 10Patch-For-Review: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017 (10JArguello-WMF)
[12:10:39] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01): [Shared Event Platform] Produce new mediawiki.page-change stream from MediaWiki EventBus - https://phabricator.wikimedia.org/T311129 (10JArguello-WMF)
[12:10:56] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01): Remove materialized .json files from event schema repositories - https://phabricator.wikimedia.org/T315674 (10JArguello-WMF)
[12:12:55] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01), 10Spike: [SPIKE] Assess what is required for the enrichment pipeline to run on k8 - https://phabricator.wikimedia.org/T315428 (10JArguello-WMF)
[12:13:32] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): Use RowTypeInfo to ensure better validation of the event data within the Mediawiki Stream Enrichment  pipeline - https://phabricator.wikimedia.org/T316555 (10JArguello-WMF) 05Open→03Resolved
[12:13:41] <wikibugs>	 10Data-Engineering, 10Epic, 10Event-Platform Value Stream (Sprint 00): Integrate Image Suggestions Feedback with Cassandra - https://phabricator.wikimedia.org/T306627 (10JArguello-WMF) 05Open→03Resolved
[12:13:45] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform Value Stream, 10Goal: Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430 (10JArguello-WMF)
[12:13:53] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: [SPIKE] Decide on technical solution for page state stream backfill process - https://phabricator.wikimedia.org/T314389 (10JArguello-WMF) 05Open→03Resolved
[12:13:57] <wikibugs>	 10Data-Engineering, 10Epic, 10Event-Platform Value Stream (Sprint 00): [Shared Event Platform] [SPIKE] Decide on page state change storing and backfill approach - https://phabricator.wikimedia.org/T311085 (10JArguello-WMF)
[12:14:01] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00), 10Spike: Define how to authenticate with Cassandra and test Flink POC - https://phabricator.wikimedia.org/T313628 (10JArguello-WMF) 05Open→03Resolved
[12:14:05] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): [Shared Event Platform] Rename scala package and setup a multi module maven project. - https://phabricator.wikimedia.org/T310626 (10JArguello-WMF) 05Open→03Resolved
[12:14:09] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10Epic: [Shared Event Platform] Design and Implement POC Flink Service to Combine Existing Streams, Enrich and Output to New Topic - https://phabricator.wikimedia.org/T307959 (10JArguello-WMF)
[12:14:12] <wikibugs>	 10Data-Engineering-Kanban, 10Data-Engineering-Radar, 10Event-Platform Value Stream (Sprint 00): Flink output support for Event Platform events - https://phabricator.wikimedia.org/T310218 (10JArguello-WMF) 05Open→03Resolved
[12:14:14] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10Epic: [Shared Event Platform] Design and Implement POC Flink Service to Combine Existing Streams, Enrich and Output to New Topic - https://phabricator.wikimedia.org/T307959 (10JArguello-WMF)
[12:14:22] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform Value Stream, 10Patch-For-Review: [Shared Event Platform] Ability to use Event Platform streams in Flink without boilerplate - https://phabricator.wikimedia.org/T308356 (10JArguello-WMF)
[12:14:32] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream: [Shared Event Platform] Mediawiki Stream Enrichment should consume the consolidated page-change stream. - https://phabricator.wikimedia.org/T311084 (10JArguello-WMF)
[12:15:19] <wikibugs>	 10Data-Engineering-Kanban, 10Event-Platform Value Stream (Sprint 01), 10Patch-For-Review: [BUG] jsonschema-tools materializes fields in yaml in a different order than in json files - https://phabricator.wikimedia.org/T308450 (10JArguello-WMF)
[12:15:34] <wikibugs>	 10Data-Engineering-Kanban, 10Event-Platform Value Stream (Sprint 01), 10Patch-For-Review: [BUG] jsonschema-tools materializes fields in yaml in a different order than in json files - https://phabricator.wikimedia.org/T308450 (10JArguello-WMF)
[12:19:25] <icinga-wm>	 RECOVERY - Check unit status of refinery-drop-webrequest-raw-partitions on an-launcher1002 is OK: OK: Status of the systemd unit refinery-drop-webrequest-raw-partitions https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[13:20:24] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 01): eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10JArguello-WMF)
[13:21:04] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10JArguello-WMF)
[13:24:00] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream: Improve EventGate's error message when the client's HTTP Content-Type is not the one expected - https://phabricator.wikimedia.org/T313202 (10JArguello-WMF) a:03tchin
[13:24:16] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01): Improve EventGate's error message when the client's HTTP Content-Type is not the one expected - https://phabricator.wikimedia.org/T313202 (10JArguello-WMF)
[13:25:56] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 01): eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10lbowmaker) a:03gmodena
[13:26:30] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning, 10SRE, 10serviceops, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10lbowmaker) a:05Jelto→03gmodena
[14:55:07] <wikibugs>	 (03CR) 10Gehel: Fix Array UDFs (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566 (owner: 10Nmaphophe)
[14:55:08] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 01), 10Spike: [SPIKE] Assess what is required for the enrichment pipeline to run on k8 - https://phabricator.wikimedia.org/T315428 (10dcausse) @gmodena thanks for exploring these k8s deployment options! Something I used to test H/A capabilities (resta...
[15:23:27] <wikibugs>	 (03PS1) 10Isaac Johnson: Bump search engine versions. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/829029 (https://phabricator.wikimedia.org/T315329)
[16:05:41] <wikibugs>	 (03PS5) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[16:10:00] <wikibugs>	 (03PS6) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[16:23:17] <wikibugs>	 (03PS7) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[16:28:20] <andrewbogott>	 oh no, I have to meet a notary at 1:30CDT... joal can we do even later? Sorry, it turns out my calendar was not even close to accurate :(
[16:28:51] <joal>	 andrewbogott: later is not great for me - possibly tomorrow despite silent friday?
[16:34:34] <wikibugs>	 (03PS8) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[16:39:00] <andrewbogott>	 joal: that should work.
[16:39:49] <joal>	 andrewbogott: after the short meeting you have tomorrow is ok for you?
[16:46:07] <wikibugs>	 (03PS9) 10Nmaphophe: Fix Array UDFs [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/828566
[16:58:19] <andrewbogott>	 joal: looks good, thanks for being flexible
[16:58:31] <joal>	 no problem andrewbogott - talk to ou tomorrow :)
[17:33:17] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10JArguello-WMF) @OwenBlacker thanks for the ping. We are currently going through the process of reviewing requests. That is why we re...
[17:40:54] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM! Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/829029 (https://phabricator.wikimedia.org/T315329) (owner: 10Isaac Johnson)
[20:55:29] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10OwenBlacker) @JArguello-WMF Thank you 💖
[22:41:20] <wikibugs>	 10Data-Engineering, 10Product-Analytics: Review why total_edits on Mediawiki_History differs from the total_edits on Editors_Daily - https://phabricator.wikimedia.org/T316896 (10Mayakp.wiki) Added this to data engineering's radar.
[22:49:26] <wikibugs>	 10Data-Engineering, 10Product-Analytics: Review why total_edits on Mediawiki_History differs from the total_edits on Editors_Daily - https://phabricator.wikimedia.org/T316896 (10nettrom_WMF) I ended up being more curious about this after our chat and so I looked up the entries in `cu_changes` on `nnwiki` aroun...
[23:06:04] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10Nuria) This is a bot that was crawling this page and realized that it was detected as a bot and it shifted its pattern a bit: (notic...