[00:24:20] <wikibugs>	 10Analytics-Radar, 10JsonConfig, 10Platform Team Workboards (MW Expedition), 10Wikimedia-production-error: Exception: Serialization of 'Closure' is not allowed - https://phabricator.wikimedia.org/T286610 (10Krinkle)
[00:25:41] <wikibugs>	 10Analytics-Radar, 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Platform Team Workboards (MW Expedition), 10Wikimedia-production-error: Exception: Serialization of 'Closure' is not allowed - https://phabricator.wikimedia.org/T286610 (10Krinkle) Re-tagging EventLogging. The JsonSchemaContent...
[02:06:05] <EpicPupper>	 Hiya! The docs mention "Please ping @madhuvishy on freenode #wikimedia-analytics", is madhuvishy here (so the docs can be changed) or not?
[02:06:36] <EpicPupper>	 Also, it looks like the .travis.yml file still references notifications going to the Freenode channel, is this still applicable, or should they be moved here?
[07:14:15] <icinga-wm>	 PROBLEM - Check unit status of mediawiki-history-drop-snapshot on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki-history-drop-snapshot https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[09:45:20] <btullis>	 I'm investigating the mediawiki-history-drop-snapshot failure. --^
[09:45:31] <btullis>	 The error message is as follows: `Mar 15 06:32:50 an-launcher1002 kerberos-run-command[31666]: 2022-03-15T06:32:50 ERROR  Selected partitions extracted from table specs ({'snapshot=2021-12-27', 'snapshot=2022-01-17', 'snapshot=2021-12-20', 'snapshot=2021-12-13', 'snapshot=2022-01-10', 'snapshot=2022-01-03'}) does not match selected partitions extracted from data paths (set()). HDFS directories to check: []`
[09:52:29] <btullis>	 EpicPupper: I think that those docs are out of date because madhuvishy no longer works here. Have you a link to the relevant docs, so we can have a look? Thanks.
[10:06:53] <wikibugs>	 (03CR) 10Btullis: [C: 03+2] Make these quotes uniform with the other setup tasks [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/767859 (owner: 10Btullis)
[10:23:18] <btullis>	 I've just sent an email about the above error. It is something to do with missing _SUCCESS files for the wikidata/entity table.
[10:42:51] <wikibugs>	 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi)
[10:43:04] <wikibugs>	 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi)
[11:52:23] <wikibugs>	 10Data-Engineering-Radar, 10Growth-Team, 10MediaWiki-extensions-GuidedTour, 10Patch-For-Review: Finish decommissioning the legacy GuidedTour schemas - https://phabricator.wikimedia.org/T303712 (10phuedx)
[11:54:30] <wikibugs>	 10Data-Engineering-Radar, 10Growth-Team, 10MediaWiki-extensions-GuidedTour, 10Patch-For-Review: Finish decommissioning the legacy GuidedTour schemas - https://phabricator.wikimedia.org/T303712 (10phuedx) > Mark all schemas as inactive    - https://meta.wikimedia.org/w/index.php?title=Schema_talk:GuidedTour...
[11:57:13] <wikibugs>	 (03PS1) 10Phuedx: Remove mentions of GuidedTour* legacy EventLogging schemas [analytics/refinery] - 10https://gerrit.wikimedia.org/r/770900 (https://phabricator.wikimedia.org/T303712)
[11:57:28] <wikibugs>	 10Data-Engineering-Radar, 10Growth-Team, 10MediaWiki-extensions-GuidedTour, 10Patch-For-Review: Finish decommissioning the legacy GuidedTour schemas - https://phabricator.wikimedia.org/T303712 (10phuedx)
[11:57:52] <wikibugs>	 10Data-Engineering-Radar, 10Growth-Team, 10MediaWiki-extensions-GuidedTour, 10Patch-For-Review: Finish decommissioning the legacy GuidedTour schemas - https://phabricator.wikimedia.org/T303712 (10phuedx) > Remove entries from the allowlist > Request the deletion of allowlisted data  These were both done as...
[12:18:26] <wikibugs>	 10Analytics, 10API Platform, 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10DAbad) **AQS 2.0 to API Platform**   - as per team meeting AQS 2.0 work will begin to fall under the API Platform workstream   - @Eevans will start to include @nnikkhoui and @B...
[12:20:45] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10DAbad)
[12:29:10] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10DAbad)
[12:37:13] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10DAbad)
[13:20:20] <milimetric>	 btullis: I think this is one of the airflow / oozie mismatches
[13:21:56] <milimetric>	 yep, sorry, just saw your email
[13:22:41] <joal>	 last time we talked about it (last month) IIRC we agreed on modifying the purging script to not check _SUCCESS files
[13:29:59] <btullis>	 joal: milimetric: Thanks for that. What should I do in the short term? Should I a) modify the purging script, b) manually touch _SUCCESS files and re-run, or c) something else?
[13:39:59] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure MariaDB database for DataHub on an-coord1001 - https://phabricator.wikimedia.org/T301459 (10BTullis) I have created the user manually on an-coord1001. {F35007059,width=80%}  The password is in pwstore.
[13:55:17] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure MariaDB database for DataHub on an-coord1001 - https://phabricator.wikimedia.org/T301459 (10Ottomata) @BTullis you could instead put the password in puppet private repo, and then use that to render it in the configs you need.  IIRC You...
[13:55:39] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform, 10Goal: Modern Event Platform: Stream Connectors - https://phabricator.wikimedia.org/T214430 (10lbowmaker)
[13:58:10] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure MariaDB database for DataHub on an-coord1001 - https://phabricator.wikimedia.org/T301459 (10BTullis) > @BTullis you could instead put the password in puppet private repo, and then use that to render it in the configs you need.  Thanks,...
[13:59:22] <ottomata>	 !log roll restarting kafka jumbo brokers to set max.incremental.fetch.session.cache.slots=2000 - T303324 
[13:59:25] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:59:25] <stashbot>	 T303324: Increase max.incremental.fetch.session.cache.slots on Kafka jumbo eqiad - https://phabricator.wikimedia.org/T303324
[14:03:35] <ottomata>	 cc joal ^
[14:06:16] <joal>	 nice ottomata!
[14:06:32] <joal>	 btullis: about the actions to take, I think it's best to talk about it in standup
[14:06:55] <joal>	 btullis: changing the script would be best, but it would also take more time than fixing this one off
[14:10:44] <wikibugs>	 10Data-Engineering, 10Data-Catalog: Streamline CI for our fork of DataHub - https://phabricator.wikimedia.org/T303381 (10BTullis) Oh look, upstream has some GitHub workflow configuration for publishing jars: https://github.com/linkedin/datahub/blob/master/.github/workflows/publish-datahub-jars.yml  I don't thi...
[14:35:00] <ottomata>	 !log change stats uid and gid on all stat boxes to 918 - T291384
[14:35:03] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:35:03] <stashbot>	 T291384: Standardize the stats system user uid - https://phabricator.wikimedia.org/T291384
[15:15:00] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Epic: Define and implement archiving for Airflow - https://phabricator.wikimedia.org/T300039 (10EChetty) @Antoine_Quhen -> This has become an implementation task yeah?
[15:23:37] <btullis>	 ottomata: thanks, agreed. Unfortunately I can't make stand-up today. Will be online a bit later today, or tomorrow. 
[15:24:05] <ottomata>	 joal:  i think ^ was for you :)
[15:25:17] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Standardize the stats system user uid - https://phabricator.wikimedia.org/T291384 (10Ottomata) Done. Changed stats uid and gid on stat boxes using the following scripts:  `lang=bash #!/bin/bash  set -x  change_uid() {     # $1 new uid     # $...
[16:05:19] <btullis>	 Oh yes, sorry both. I pinged the wrong person.
[16:05:45] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Create debian package of karapace - https://phabricator.wikimedia.org/T301565 (10razzi)
[16:46:54] <wikibugs>	 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) Thanks for this task! Now that we have a clear path forward in T298087, it makes sens to focus on this one a...
[17:24:35] <ottomata>	 !log also change stats uid and gid to 918 on an-web1001 - T291384
[17:24:37] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:24:38] <stashbot>	 T291384: Standardize the stats system user uid - https://phabricator.wikimedia.org/T291384
[17:28:37] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Standardize the stats system user uid - https://phabricator.wikimedia.org/T291384 (10Ottomata) I did not realize the stats user was also on an-web1001.  Since puppet already changed the uid,gid there, I had to manually run the following to ch...
[17:46:15] <wikibugs>	 (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM! Thanks :]" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/769937 (https://phabricator.wikimedia.org/T303508) (owner: 10Phuedx)
[18:09:36] <wikibugs>	 (03CR) 10Joal: "Comments mostly about naming - all good from the coding perspective :)" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/767178 (owner: 10Joal)
[18:14:39] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10Mayakp.wiki) Hey Everyone, here is a full report on [[ https://docs.google.com/presentation/d/1EFg6oa5WNbrL...
[18:32:43] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: Increase max.incremental.fetch.session.cache.slots on Kafka jumbo eqiad - https://phabricator.wikimedia.org/T303324 (10Ottomata) Deployed and restarted brokers.  Let's watch fetch sessions and cache evictions over the next few days.
[19:17:53] <wikibugs>	 10Data-Engineering-Radar, 10ExternalGuidance, 10MediaWiki-extensions-WikimediaEvents, 10Patch-For-Review: Decommission the ExternalGuidance instrument - https://phabricator.wikimedia.org/T303508 (10phuedx)
[19:56:45] <wikibugs>	 (03PS1) 10Vivian Rook: Replace space with underscore in link [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/771010 (https://phabricator.wikimedia.org/T72166)
[19:57:49] <wikibugs>	 10Quarry, 10Patch-For-Review: Replace spaces with underscores for wiki usernames in URL - https://phabricator.wikimedia.org/T72166 (10rook) a:03rook
[19:58:23] <razzi>	 !log upload karapace_2.1.3-py3.7-0_amd64.deb to apt.wikimedia.org
[19:58:24] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:15:33] <wikibugs>	 (03PS12) 10Ottomata: Add PrometheusEventReporter [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/767178 (owner: 10Joal)
[20:15:57] <wikibugs>	 (03CR) 10Ottomata: Add PrometheusEventReporter (033 comments) [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/767178 (owner: 10Joal)
[20:17:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add PrometheusEventReporter [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/767178 (owner: 10Joal)
[20:38:44] <wikibugs>	 10Data-Engineering-Radar, 10Privacy Engineering, 10Privacy: Privacy review for dataset publishing (Wikidata topic -> pageview data) - https://phabricator.wikimedia.org/T303304 (10Htriedman) Hi @Addshore! I'm Hal, a privacy engineer at WMF, and I'll be taking a look at this (rerunning the notebook, assessing...
[21:58:49] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Epic: Define and implement archiving for Airflow - https://phabricator.wikimedia.org/T300039 (10Antoine_Quhen) Yes, it is also an implementation now.
[22:16:30] <razzi>	 !log upload karapace_2.1.3-py3.7-1_amd64.deb to apt.wikimedia.org
[22:16:33] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[22:24:54] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Create debian package of karapace - https://phabricator.wikimedia.org/T301565 (10razzi) Deb is uploaded: https://apt.wikimedia.org/wikimedia/pool/main/k/karapace/  Code is also on the standard operations/deb gerrit repositor...