[00:03:34] 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu', 'nokafor', and 'xcollazo' - https://phabricator.wikimedia.org/T314592 (10odimitrijevic) @xcollazo Does @JEbe-WMF need to create a similar re... [00:14:04] 10Data-Engineering-Planning: Emit lineage information about Airflow jobs to DataHub - https://phabricator.wikimedia.org/T312566 (10odimitrijevic) Yes, thank you @Antoine_Quhen! This is very exciting. Will the lineage be emitted with the upgrade or will there be another step to configure lineage to be emitted? Do... [00:23:08] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 5 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10Dreamy_Jazz) [00:24:01] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 5 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10Dreamy_Jazz) [05:44:20] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Ladsgroup) I'm losing track, when can we drop the old actor columns in cu_chan... [08:46:33] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) >>! In T233004#8573151, @Ladsgroup wrote: > I'm losing track, when can w... [08:47:28] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) also I should really learn how to do task management, this task is proba... [09:26:26] 10Data-Engineering, 10Equity-Landscape: Population input metrics - https://phabricator.wikimedia.org/T309279 (10KCVelaga_WMF) Noting column name change from `population_thousands` to `population_total` [09:49:17] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset : matmarex and cmyrick - https://phabricator.wikimedia.org/T328152 (10BTullis) p:05High→03Unbreak! Raising the priority of this to unbreak now. Two more users have re... [10:01:53] (03PS2) 10Joal: Add hql/webrequest/actor folder and scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) [10:04:52] (03CR) 10Joal: "I think this is ready to merge :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [10:05:29] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) [10:06:57] (03CR) 10Joal: Add hql/webrequest/actor folder and scripts (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [10:43:13] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) [11:19:20] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10MW-1.40-notes (1.40.0-wmf.1; 2022-09-12): Generate $wgEventLoggingStreamNames from $wgEventStreams - https://phabricator.wikimedia.org/T303602 (10phuedx) [11:44:45] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): Deployment pipeline docker image of flink mediawiki stream enrichment pyhon - https://phabricator.wikimedia.org/T326731 (10gmodena) [12:04:51] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) Here is what is supposed to happen. * User tries to access https://superset.wikimedia.org/ * Apache picks up the r... [12:05:17] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): eventutilities-python source and destination stream must be versioned - https://phabricator.wikimedia.org/T327866 (10gmodena) >>! In T327866#8557795, @Ottomata wrote: > Sinks must specify the version. Producer code (which is usually why the someon... [12:05:25] 10Data-Engineering-Planning, 10MediaWiki-extensions-EventLogging, 10Metrics Platform Icebox, 10Notifications, and 4 others: [EPIC] Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263 (10phuedx) [12:47:08] 10Data-Engineering, 10Data-Catalog, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10BTullis) I have manually added the record for @JEbe-WMF using the following technique. * I used a [[https://wik... [12:50:33] 10Data-Engineering, 10Data-Catalog, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10BTullis) >>! In T327884#8568585, @Stevemunene wrote: > Did some more reading on JAAS user extractions specifical... [12:53:33] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Marostegui) [13:01:18] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) I've asked users to try logging out of CAS and back in. It seemed to work for @Stevemunene but @matmarex has report... [13:43:08] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): Flink docker image should work with pyflink - https://phabricator.wikimedia.org/T327494 (10Ottomata) Got some [[ https://lists.apache.org/thread/2p8fxr42z67lg47pormydr34gpbjlzhv | responses to my Qs on the Flink mailing list ]]. >> [AO]: What is... [13:44:44] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink application and flink-kubernetes-operator production docker images - https://phabricator.wikimedia.org/T316519 (10Ottomata) FYI, in order to make pyflink work with this image as well, we changed our installation... [13:55:59] (03PS7) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [13:56:31] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [14:00:51] (03PS8) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [14:01:19] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [14:02:31] (03PS9) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [14:02:58] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [14:11:57] (03PS3) 10Joal: Add hql/webrequest/actor folder and scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) [14:12:35] (03CR) 10Joal: "@mforns, @milimetric - This actually needs a second check for the Oozie change of the pageview_actor job" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [14:15:09] (03PS10) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [14:15:39] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [14:18:42] 10Data-Engineering-Planning: Requesting Kerberos identity for Hxi-ctr - https://phabricator.wikimedia.org/T325857 (10HXi-WMF) Hi! I can now connect using xihua as my shell username but I am still not able to sign into Juypter notebook when I go to localhost:8880 To clarify, I should be using my shell username x... [14:19:26] (03PS11) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [14:19:53] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [14:53:12] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Patch-For-Review, 10SecTeam-Processed, 10Vuln-VulnComponent: Upgrade Puppet code to make Airflow configuration files compatible with version 2.3.4 - https://phabricator.wikimedia.org/T315580 (10Stevemunene) [14:57:30] (03CR) 10Milimetric: "just one thought regarding deprecating the old table, spread over the files I think it affects." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [15:00:29] (03PS12) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [15:00:57] (03CR) 10CI reject: [V: 04-1] Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) (owner: 10Mazevedo) [15:35:04] 10Data-Engineering, 10Event-Platform Value Stream, 10Shared-Data-Infrastructure: Add dse k8s networks to puppet network constants - https://phabricator.wikimedia.org/T328447 (10Ottomata) [15:55:55] (03CR) 10Milimetric: [C: 04-1] "oh, realized there's one more thing and it's a blocker" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [15:56:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2035 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2035%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:01:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2035 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2035%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:03:01] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) I believe that this incident can now be considered resolved. The main problem seems to be that user permissions wer... [16:03:36] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) p:05Unbreak!→03Medium [16:08:43] 10Data-Engineering, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 08), 10Patch-For-Review: Add dse k8s networks to puppet network constants - https://phabricator.wikimedia.org/T328447 (10Ottomata) Done. [16:09:14] 10Data-Engineering, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 08), 10Patch-For-Review: Add dse k8s networks to puppet network constants - https://phabricator.wikimedia.org/T328447 (10Ottomata) [16:09:23] 10Data-Engineering, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 08), 10Patch-For-Review: Add dse k8s networks to puppet network constants - https://phabricator.wikimedia.org/T328447 (10Ottomata) p:05Triage→03High a:03Ottomata [16:10:00] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Support topics without a schema in Flink Catalog - https://phabricator.wikimedia.org/T328232 (10EChetty) [16:10:02] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Support NULL values in RowData in eventutilities - https://phabricator.wikimedia.org/T328211 (10EChetty) [16:10:04] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Use new PageUndeleteComplete hook to emit mediawiki.page_change undelete event - https://phabricator.wikimedia.org/T328308 (10EChetty) [16:10:06] 10Data-Engineering-Planning, 10Event-Platform Value Stream: [NEEDS GROOMING] Improve mediawiki-event-enrichment test suite - https://phabricator.wikimedia.org/T328013 (10EChetty) [16:10:08] 10Analytics, 10Data-Engineering-Planning: Add cawiki to clickstream dataset - https://phabricator.wikimedia.org/T327982 (10EChetty) [16:10:10] 10Data-Engineering-Planning, 10Product-Analytics, 10Wmfdata-Python: Wmfdata-Python's CSV loading cannot handle standard quoted CSV values - https://phabricator.wikimedia.org/T327983 (10EChetty) [16:10:12] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): eventutilities-python should support nested row type info - https://phabricator.wikimedia.org/T327900 (10EChetty) [16:10:14] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): eventutilities-python source and destination stream must be versioned - https://phabricator.wikimedia.org/T327866 (10EChetty) [16:10:16] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): Flink docker image should work with pyflink - https://phabricator.wikimedia.org/T327494 (10EChetty) [16:10:18] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Q4 eventutilities-python should bundle java deps. - https://phabricator.wikimedia.org/T327251 (10EChetty) [16:10:20] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Growth-Team (Current Sprint), 10MW-1.40-notes (1.40.0-wmf.21; 2023-01-30): Remove CommentFormatter from EventFactory constructor, or otherwise make its usage optional - https://phabricator.wikimedia.org/T327065 (10EChetty) [16:10:22] 10Data-Engineering-Planning, 10IP Masking: Update Data Engineering-owned products that may be affected by IP Masking - https://phabricator.wikimedia.org/T326875 (10EChetty) [16:10:25] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): Streaming services errors should be routed to an error event topic. - https://phabricator.wikimedia.org/T326536 (10EChetty) [16:10:27] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08), 10Patch-For-Review: Tests for mediawiki-stream-enrichment-python flink job via eventutilities-python - https://phabricator.wikimedia.org/T326565 (10EChetty) [16:10:29] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10MediaWiki-Core-Hooks: Add $comment and $performer to ArticleRevisionVisibilitySet params - https://phabricator.wikimedia.org/T321411 (10EChetty) [16:10:33] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): Deployment pipeline docker image of flink mediawiki stream enrichment pyhon - https://phabricator.wikimedia.org/T326731 (10EChetty) [16:10:37] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 08), 10Patch-For-Review: Add dse k8s networks to puppet network constants - https://phabricator.wikimedia.org/T328447 (10EChetty) [16:10:41] 10Data-Engineering-Planning, 10Data-Catalog, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10EChetty) [16:10:49] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 10 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10EChetty) [16:11:03] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install an-worker11[49-56] - https://phabricator.wikimedia.org/T327295 (10EChetty) [16:12:26] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Epic: Create puppet profiles for the new ceph cluster - https://phabricator.wikimedia.org/T328123 (10EChetty) [16:12:55] 10Analytics, 10Data-Engineering-Planning, 10Data Pipelines: Add cawiki to clickstream dataset - https://phabricator.wikimedia.org/T327982 (10EChetty) [16:14:08] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10EChetty) [16:14:41] 10Data-Engineering-Planning, 10Data-Catalog, 10Shared-Data-Infrastructure, 10Patch-For-Review: Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10EChetty) [16:14:43] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q3:rack/setup/install an-worker11[49-56] - https://phabricator.wikimedia.org/T327295 (10EChetty) [16:31:17] (03PS13) 10Mazevedo: Add MobileWikiAppiOSUserHistory to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885004 (https://phabricator.wikimedia.org/T328312) [16:31:52] 10Data-Engineering-Planning: Review Superset permissions and assign roles as appropriate - https://phabricator.wikimedia.org/T328457 (10BTullis) [16:32:52] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) 05Open→03Resolved a:03BTullis Here is the follow-up ticket about reviewing Superset access. {T328457} [16:35:10] (03PS7) 10Peter Fischer: Provide internal schema for CirrusSearch update-pipeline updates. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/856507 (https://phabricator.wikimedia.org/T317202) [16:39:43] Do we have anything to be deployed in the Analytics train today? I see one refinery CR on the etherpad: https://gerrit.wikimedia.org/r/c/analytics/refinery/+/883525 [16:49:26] 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu', 'nokafor', and 'xcollazo' - https://phabricator.wikimedia.org/T314592 (10xcollazo) >>! In T314592#8572667, @odimitrijevic wrote: > @xcollazo... [17:05:17] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10Htriedman) My SQL Lab on superset has also not been working for the past week or so! [17:06:52] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10BTullis) >>! In T328152#8575217, @Htriedman wrote: > My SQL Lab on superset has also not been working for the past week or s... [17:15:46] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10Htriedman) Up and running! thanks for the help [17:23:17] 10Data-Engineering-Planning, 10Data Pipelines, 10Pageviews-Anomaly, 10Wikipedia-iOS-App-Backlog, and 4 others: Analyze possible bot traffic for enwiki article Index (statistics), Index & XXX:_Return_of_Xander_Cage - https://phabricator.wikimedia.org/T328127 (10LGoto) p:05Triage→03High [17:34:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2032 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2032%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:39:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2032 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2032%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:40:38] 10Analytics, 10Scap: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 (10demon) [17:43:39] 10Data-Engineering, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Some users' presto queries are no longer working in Superset - https://phabricator.wikimedia.org/T328152 (10jwang) Hi, My SQL Lab doesn't work. [17:54:18] 10Data-Engineering-Planning: Review Superset permissions and assign roles as appropriate - https://phabricator.wikimedia.org/T328457 (10kzimmerman) @BTullis Can you please review permissions for the Product Analytics team? We should all have sql_lab access if we don't already. Connie Chen Irene Florez (iflorez)... [18:06:44] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Product-Analytics (Kanban): Include EU Registered Country in the canonical country database - https://phabricator.wikimedia.org/T324995 (10mpopov) a:05mforns→03nshahquinn-wmf Neil will do the necessary fixes [18:08:10] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Product-Analytics (Kanban): Include EU Registered Country in the canonical country database - https://phabricator.wikimedia.org/T324995 (10mforns) Oh, I can do that, @mpopov! I was looking into that today. If that's OK to you. [18:14:33] 10Data-Engineering-Planning, 10Product-Analytics, 10Wmfdata-Python: Wmfdata-Python's CSV loading cannot handle standard quoted CSV values - https://phabricator.wikimedia.org/T327983 (10mpopov) p:05Triage→03Low [18:33:35] 10Data-Engineering, 10Event-Platform Value Stream: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment - https://phabricator.wikimedia.org/T328478 (10Ottomata) [18:49:17] 10Data-Engineering-Planning: Review Superset permissions and assign roles as appropriate - https://phabricator.wikimedia.org/T328457 (10JAnstee_WMF) @BTullis Can you also review permissions for the Global Data & Insights team - I know my access has disappeared along with others - The following members should als... [18:55:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2034 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2034%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:00:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2034 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2034%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:01:23] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Product-Analytics (Kanban): Include EU Registered Country in the canonical country database - https://phabricator.wikimedia.org/T324995 (10nshahquinn-wmf) a:05nshahquinn-wmf→03mforns @mforns oh sure! I thought I might as well do it since you se... [19:15:13] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10colewhite) [19:27:01] 10Data-Engineering-Planning, 10Data Pipelines, 10Pageviews-Anomaly, 10Wikipedia-iOS-App-Backlog, and 5 others: Analyze possible bot traffic for enwiki article Index (statistics), Index & XXX:_Return_of_Xander_Cage - https://phabricator.wikimedia.org/T328127 (10LGoto) [19:39:39] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 07), 10Product-Analytics (Kanban): Include EU Registered Country in the canonical country database - https://phabricator.wikimedia.org/T324995 (10mforns) :+1: [20:06:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2037 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2037%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:10:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp5029 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5029%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:11:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2037 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2037%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:15:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp5029 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5029%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:19:45] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:26:33] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): Flink SQL queries should access Kafka topics from a Catalog - https://phabricator.wikimedia.org/T322022 (10tchin) After testing more, it seems better to explicitly define options `event-stream-name` and `event-stream-prefix` instead of try... [20:44:46] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:50:47] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 08): Flink SQL queries should access Kafka topics from a Catalog - https://phabricator.wikimedia.org/T322022 (10Ottomata) > After testing more, it seems better to explicitly define options event-stream-name and event-stream-prefix Sounds good.... [21:25:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:25:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2039 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2039%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:30:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:30:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2039 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2039%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:08:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp5020 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp5020%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:12:48] 10Data-Engineering, 10API Platform (Sprint 03), 10AQS2.0, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews: Implement Unit Tests - https://phabricator.wikimedia.org/T299735 (10BPirkle) @SGupta-WMF, please see below regarding test failures: Regarding the 400 on `/metrics/pageviews/aggreg... [22:13:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp5020 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp5020%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:55:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2040 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2040%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:00:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2040 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2040%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages