[00:31:42] (03PS1) 10Krinkle: build: Add brief "Getting started" guide [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/714874 [00:31:44] (03PS1) 10Krinkle: build: Document simpler alternative contribution flow [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/714875 [00:32:06] (03CR) 10jerkins-bot: [V: 04-1] build: Document simpler alternative contribution flow [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/714875 (owner: 10Krinkle) [04:27:20] PROBLEM - Check unit status of monitor_refine_event_sanitized_analytics_immediate on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_sanitized_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:31:22] PROBLEM - Check unit status of monitor_refine_event_sanitized_analytics_delayed on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_sanitized_analytics_delayed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:58:13] hello folks, Bigtop officially created the 3.0.0 branch for the release, they are starting to be close :) [06:12:49] also an-launcher1002's root partition is getting full :( [06:19:41] one interesting thing is that /var/lib/puppet/clientbucket weights ~16G, I found https://phabricator.wikimedia.org/T165885 but there is no mention of what we do in prod [06:21:00] !log root@an-launcher1002:/var/lib/puppet/clientbucket# find -type f -delete -mtime +60 [06:21:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:22:35] !log root@an-launcher1002:/var/lib/puppet/clientbucket# find -type d -empty -delete [06:22:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:24:19] all good, also tested a puppet run [06:36:18] the other alert that I see is mariadb memory usage for dbstore1007 [06:38:09] task is https://phabricator.wikimedia.org/T270112 [06:45:19] btullis: o/ not sure how much exposure you had with dbstore nodes but I can give you a quick intro when you are online if you want [07:32:54] 10Analytics-Clusters, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10vm-requests: Site: Eqiad - 1 VM request for analytics test cluster - coordinator replica role - https://phabricator.wikimedia.org/T289664 (10jcrespo) Just to be clear, I am not saying "analytics cannot use ganeti"... [07:47:30] 10Analytics-Clusters, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10vm-requests: Site: Eqiad - 1 VM request for analytics test cluster - coordinator replica role - https://phabricator.wikimedia.org/T289664 (10elukey) Another alternative that we may want to explore (that wasn't ava... [07:55:56] (03CR) 10David Caro: [C: 03+1] "Neat, LGTM, nits can be ignored." [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/714632 (owner: 10Michael DiPietro) [10:06:52] (03PS5) 10Andrew-WMDE: Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) [10:11:30] (03PS6) 10Andrew-WMDE: Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) [10:18:46] (03CR) 10Andrew-WMDE: [C: 03+1] Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) (owner: 10Andrew-WMDE) [10:42:17] (03PS7) 10Andrew-WMDE: Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) [10:53:13] (03PS4) 10Andrew-WMDE: Aggregate the number of times template data was available in TemplateWizard [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713647 (https://phabricator.wikimedia.org/T272589) [10:57:10] (03CR) 10Andrew-WMDE: [C: 03+1] Aggregate the number of times template data was available in TemplateWizard [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713647 (https://phabricator.wikimedia.org/T272589) (owner: 10Andrew-WMDE) [10:59:43] (03PS8) 10Andrew-WMDE: Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) [11:00:23] (03CR) 10Andrew-WMDE: [C: 03+1] Aggregate the number of times template data was available in VE's template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/713650 (https://phabricator.wikimedia.org/T272589) (owner: 10Andrew-WMDE) [11:03:34] (03PS3) 10Andrew-WMDE: Track multi-part transclusion stats for VE template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/714065 (https://phabricator.wikimedia.org/T272589) (owner: 10Awight) [11:07:22] (03CR) 10Andrew-WMDE: [C: 03+1] Track multi-part transclusion stats for VE template dialog [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/714065 (https://phabricator.wikimedia.org/T272589) (owner: 10Awight) [11:20:38] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] T283575 [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/714540 (owner: 10GoranSMilovanovic) [11:31:40] (03PS4) 10Hnowlan: Add docker-compose environment with cassandra [analytics/aqs] - 10https://gerrit.wikimedia.org/r/679295 (https://phabricator.wikimedia.org/T257572) [11:34:33] (03CR) 10jerkins-bot: [V: 04-1] Add docker-compose environment with cassandra [analytics/aqs] - 10https://gerrit.wikimedia.org/r/679295 (https://phabricator.wikimedia.org/T257572) (owner: 10Hnowlan) [11:35:49] (03PS5) 10Hnowlan: Add docker-compose environment with cassandra [analytics/aqs] - 10https://gerrit.wikimedia.org/r/679295 (https://phabricator.wikimedia.org/T257572) [11:38:45] (03CR) 10jerkins-bot: [V: 04-1] Add docker-compose environment with cassandra [analytics/aqs] - 10https://gerrit.wikimedia.org/r/679295 (https://phabricator.wikimedia.org/T257572) (owner: 10Hnowlan) [11:40:21] (03PS1) 10GoranSMilovanovic: minor [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/714999 [11:40:23] (03PS1) 10GoranSMilovanovic: rdbs_runSQL() [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/715000 [11:40:34] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] minor [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/714999 (owner: 10GoranSMilovanovic) [11:40:46] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] rdbs_runSQL() [analytics/wmde/WD/WikidataAnalytics] - 10https://gerrit.wikimedia.org/r/715000 (owner: 10GoranSMilovanovic) [12:13:28] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10jcrespo) Hey, AbbanWMDE, according to https://ldap.toolforge.org/user/abban you are in the nda list of users, which, according to https://wikitech.wikimedia.org/wiki/Analytics/Data_access#LDAP_acces... [12:58:31] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10AbbanWMDE) Hi @jcrespo, I'm logging in in the browser through https://superset.wikimedia.org with the same credentials I use for Wikitech. My first thought was maybe I'm missing a MySQL permission.... [13:09:55] (03PS1) 10Jgiannelos: WIP: Map tile expiration event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/715011 (https://phabricator.wikimedia.org/T289771) [13:10:29] (03CR) 10jerkins-bot: [V: 04-1] WIP: Map tile expiration event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/715011 (https://phabricator.wikimedia.org/T289771) (owner: 10Jgiannelos) [13:10:31] (03PS1) 10Lucas Werkmeister (WMDE): Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/715013 (https://phabricator.wikimedia.org/T286903) [13:12:01] (03CR) 10Ladsgroup: [C: 03+2] Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/715013 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [13:13:11] (03Merged) 10jenkins-bot: Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/715013 (https://phabricator.wikimedia.org/T286903) (owner: 10Lucas Werkmeister (WMDE)) [13:13:13] (03CR) 10Jgiannelos: "First attempt to define an event schema for maps tile expiration." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/715011 (https://phabricator.wikimedia.org/T289771) (owner: 10Jgiannelos) [13:13:42] (03PS1) 10Ladsgroup: Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714856 (https://phabricator.wikimedia.org/T286903) [13:13:48] (03CR) 10Ladsgroup: [C: 03+2] Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714856 (https://phabricator.wikimedia.org/T286903) (owner: 10Ladsgroup) [13:15:56] (03Merged) 10jenkins-bot: Make active_items.php executable [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/714856 (https://phabricator.wikimedia.org/T286903) (owner: 10Ladsgroup) [13:20:56] (03PS2) 10Jgiannelos: WIP: Map tile expiration event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/715011 (https://phabricator.wikimedia.org/T289771) [13:35:12] (03PS3) 10Jgiannelos: Map tile expiration event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/715011 (https://phabricator.wikimedia.org/T289771) [13:35:35] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10jcrespo) Ah, this part is interesting: ` analytics-privatedata-users ` Maybe you are missing some server access, as an ldap-only-user, or you are expected to use a differeant way to query that data.... [13:38:52] 10Analytics, 10Wikidata, 10Wikidata-Query-Service: Move wikidata RDF triples files/table to the wmf namespace in hdfs and hive - https://phabricator.wikimedia.org/T272961 (10Gehel) 05Open→03Declined We'll reopen something similar when we have a more concrete data management strategy. [13:39:47] (03CR) 10Michael DiPietro: [C: 03+2] celery update [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/714632 (owner: 10Michael DiPietro) [13:40:31] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10AbbanWMDE) Nice one, thanks for the help, Jamie! [13:43:35] (03Merged) 10jenkins-bot: celery update [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/714632 (owner: 10Michael DiPietro) [13:44:18] (03CR) 10Michael DiPietro: celery update (031 comment) [analytics/quarry/web] (buster) - 10https://gerrit.wikimedia.org/r/714632 (owner: 10Michael DiPietro) [15:08:57] 10Analytics-Radar, 10Data-Services, 10cloud-services-team (Kanban): Mitigate breaking changes from the new Wiki Replicas architecture - https://phabricator.wikimedia.org/T280152 (10Jhernandez) a:05Jhernandez→03None [16:01:30] 10Analytics-Clusters, 10Analytics-Kanban, 10 Data-Engineering, 10Data-Engineering-Kanban, 10vm-requests: Site: Eqiad - 1 VM request for analytics test cluster - coordinator replica role - https://phabricator.wikimedia.org/T289664 (10BTullis) I have formally requested a server for this requirement: {T2897... [16:25:59] tiziano: Hi - I just killed your spark job, it was using 6Tb of ram [16:28:39] (03PS1) 10MNeisler: Add the SearchSatisfaction legacy schema to the allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/715055 (https://phabricator.wikimedia.org/T274607) [16:30:18] 10Analytics, 10Patch-For-Review, 10Product-Analytics (Kanban): Add SearchSatisfaction to the allowlist - https://phabricator.wikimedia.org/T274607 (10MNeisler) [16:32:29] joal: Hi! Sorry, I think there was something weird with the Jupyter kernel. I restarted the Python kernel, but it looks like the spark process continued. [16:33:01] tiziano: no problem, I wanted to let you know though :) [16:34:42] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10odimitrijevic) p:05Triage→03High [16:39:20] lol 6T of ram :D [16:39:44] I still get impressed by these numbers after all these years [16:42:49] 10Analytics: Add Execute Permissions for WMDE Developer Abban Dunne - https://phabricator.wikimedia.org/T289775 (10JAllemandou) Hi @Abban, Indeed to query (most) data with Presto you need to be in the `analytics-privatedata-users` group. This requires a formal demand with manager approval (you'll get access to P... [16:43:36] 10Analytics-Clusters, 10 Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10odimitrijevic) [16:44:45] 10Analytics, 10 Data-Engineering: Data Loss Check always shows false positives - https://phabricator.wikimedia.org/T288496 (10odimitrijevic) p:05Triage→03Medium [16:50:52] 10Analytics, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10odimitrijevic) p:05Triage→03High [17:03:20] 10Analytics, 10Product-Analytics: [REQUEST] - https://phabricator.wikimedia.org/T289799 (10kzimmerman) p:05Triage→03High [17:05:46] 10Analytics, 10Product-Analytics: [REQUEST] Investigate decrease in New Registered Users - https://phabricator.wikimedia.org/T289799 (10kzimmerman) [17:19:07] 10Analytics, 10Product-Analytics: [REQUEST] Investigate decrease in New Registered Users - https://phabricator.wikimedia.org/T289799 (10MMiller_WMF) Thanks for creating this, @kzimmerman. One thing that might be part of this story is a change to the Create:Account page on English Wikipedia, which added a warn... [18:50:46] 10Analytics, 10SRE: Remove fdans from analytics-alerts mailing list - https://phabricator.wikimedia.org/T289807 (10JAllemandou) [18:51:55] 10Analytics, 10Product-Analytics: [REQUEST] Investigate decrease in New Registered Users - https://phabricator.wikimedia.org/T289799 (10kzimmerman) [19:46:37] 10Analytics-Clusters, 10 Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10RobH) [20:25:55] 10Analytics, 10Product-Analytics (Kanban): Add mediawiki_pref_diff to the allowlist - https://phabricator.wikimedia.org/T287255 (10MNeisler) a:05MNeisler→03jwang [20:52:26] (03CR) 10Urbanecm: Add analytics/mediawiki/mentor_dashboard/visit (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/714099 (https://phabricator.wikimedia.org/T289369) (owner: 10Urbanecm) [21:07:45] (03PS2) 10Krinkle: build: Document simpler alternative contribution flow [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/714875 [21:23:44] 10Analytics, 10Pageviews-API, 10cloud-services-team (Kanban): pageviews-analytics stopped working - https://phabricator.wikimedia.org/T286115 (10Krinkle)