[00:26:21] 06Data-Engineering-Icebox, 06Movement-Insights, 10Movement-Metrics: Consider recalculating revert rate - https://phabricator.wikimedia.org/T267053#9603783 (10nshahquinn-wmf) [00:27:10] 06Data-Engineering-Icebox, 06Movement-Insights, 05Campaign-Registration: Develop a consistent rule for which special pages count as pageviews - https://phabricator.wikimedia.org/T240676#9603784 (10nshahquinn-wmf) [00:30:13] 06Data-Engineering, 06Movement-Insights, 07Epic: Readership Retention: New vs. Returning Unique devices - https://phabricator.wikimedia.org/T269815#9603786 (10nshahquinn-wmf) [00:30:59] 06Data-Engineering, 10Foundational Technology Requests, 06Movement-Insights, 10Movement-Metrics: "Source of truth" dataset for pageviews - https://phabricator.wikimedia.org/T310732#9603790 (10nshahquinn-wmf) [00:46:12] 14Analytics-Radar, 06Data-Engineering-Icebox, 06Product-Analytics, 06Research-Freezer, 10Research-consulting: Propose metrics along with qualifiers for the press kit - https://phabricator.wikimedia.org/T144639#9603840 (10nshahquinn-wmf) 05Open→03Resolved a:03nshahquinn-wmf As with T117221, this was... [00:46:53] 06Data-Engineering, 10Data-Engineering-Wikistats, 10Movement-Metrics: Support including edits to deleted pages in editing metrics - https://phabricator.wikimedia.org/T295212#9603846 (10nshahquinn-wmf) [00:48:40] 06Data-Engineering, 06Movement-Insights: Keep canonical_data.wikis updated - https://phabricator.wikimedia.org/T241741#9603847 (10nshahquinn-wmf) [03:27:27] 14Analytics, 14Analytics-Kanban, 06Data-Engineering, 10Data-Engineering-Wikistats: "Total Article Count" (a.k.a "pages to date") Wikistats metric (per project and overall) - https://phabricator.wikimedia.org/T198425#9604028 (10Sj) For en:wp, this seems to differ from [[ https://en.wikipedia.org/wiki/Wiki... [03:47:03] 10Data-Engineering (Sprint 9), 13Patch-For-Review: [Maintenance] Migrate ReportUpdater browser queries to Airflow - https://phabricator.wikimedia.org/T354552#9604167 (10CodeReviewBot) ebysans opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/622 Update browser metric air... [09:47:55] If anyone who understands the `mediawiki_history` pipeline is around and is able to help me to fix an issue with the `check_page_history` task errors, I would be grateful. [10:00:24] Hi btullis - I can help, will be available in about 1/2h if ok for you [10:01:43] joal: Perfect, many thanks. [10:40:39] joal: While I think about it, this page still references oozie jobs. https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Mediawiki_History_Snapshot_Check#Jobs_organization [10:41:10] I can update it to reference Airflow DAGs, but I'm not very familiar with the job organisation, so I might get it wrong. [10:41:15] Ah! interesting :) We should link to Airflow [10:41:22] I'm available btullis if you wish [10:41:33] Great! Batcave? [10:41:38] sure! OMW [11:10:02] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#9605542 (10Fabfur) Update: Benthos is installed on cp4037 and after some minor fixes, is finally ready to ingest, process and... [11:21:14] (03CR) 10Gmodena: "LGTM. Left a couple of nits / questions." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1008934 (https://phabricator.wikimedia.org/T354692) (owner: 10Snwachukwu) [13:30:07] 14Analytics, 06Data-Engineering, 06DBA, 10Event-Platform: Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242#9606387 (10Ottomata) [13:34:41] 06Data-Engineering, 10Foundational Technology Requests, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Enable the Marketing Campaigns Reporting plugin for matomo - https://phabricator.wikimedia.org/T319013#9606427 (10BTullis) [13:37:11] btullis: With your permission I'll update the task you create about the cirrus-search data refine issue - there are some aspects I think could improve in the task title/description [13:37:38] oh and btullis, thank you so much for creating this task originally, it was much needed :) [13:42:12] 06Data-Engineering, 06Data-Platform-SRE, 10superset.wikimedia.org: Superset Timeout Logging - https://phabricator.wikimedia.org/T294772#9606481 (10Gehel) [13:42:15] 06Data-Engineering, 06Data-Platform-SRE, 07Epic: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710#9606482 (10Gehel) [13:42:20] 06Data-Engineering, 06Data-Platform-SRE, 10superset.wikimedia.org: Superset Timeout Logging - https://phabricator.wikimedia.org/T294772#9606479 (10Gehel) This might get fixed by T347710 [13:45:41] btullis: I have proceeded - please proffread https://phabricator.wikimedia.org/T359215 and let me know if it makes sense :) [13:50:14] joal: Excellent, thanks. Looking now. [13:51:19] Makes perfect sense, thanks. [14:04:14] (03CR) 10Joal: Extract RefineSingleApp code from Refine (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1003745 (https://phabricator.wikimedia.org/T356363) (owner: 10Joal) [14:27:02] (03PS20) 10Joal: Extract RefineSingleApp code from Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1003745 (https://phabricator.wikimedia.org/T356363) [14:34:57] (03PS21) 10Joal: Extract RefineSingleApp code from Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1003745 (https://phabricator.wikimedia.org/T356363) [14:43:04] (03PS22) 10Joal: Extract RefineSingleApp code from Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1003745 (https://phabricator.wikimedia.org/T356363) [14:57:58] 06Data-Engineering, 06Structured-Data-Backlog: Make HTML Dumps available in hadoop - https://phabricator.wikimedia.org/T305688#9607332 (10mfossati) [15:00:12] (03PS23) 10Joal: Extract RefineSingleApp code from Refine [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1003745 (https://phabricator.wikimedia.org/T356363) [15:28:58] 10Data-Engineering (Sprint 9): [NEEDS GROOMING] we should provide DQ integration with Python - https://phabricator.wikimedia.org/T353940#9607547 (10gmodena) [15:32:32] 10Data-Engineering (Sprint 9): [NEEDS GROOMING] we should provide DQ integration with Python - https://phabricator.wikimedia.org/T353940#9607561 (10lbowmaker) [16:04:49] 06Data-Engineering, 10Foundational Technology Requests, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Enable the Marketing Campaigns Reporting plugin for matomo - https://phabricator.wikimedia.org/T319013#9607695 (10BTullis) For now, I have manually installed the plugin in `/usr/share/matomo/plugins/Marketi... [16:05:32] 06Data-Engineering, 10Foundational Technology Requests, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Enable the Marketing Campaigns Reporting plugin for matomo - https://phabricator.wikimedia.org/T319013#9607696 (10BTullis) [17:10:33] 06Data-Engineering, 10Foundational Technology Requests, 10Data-Platform-SRE (2024.03.04 - 2024.03.24), 13Patch-For-Review: Enable the Marketing Campaigns Reporting plugin for matomo - https://phabricator.wikimedia.org/T319013#9607991 (10BTullis) I also had to run `console core:update` as shown, in order to... [17:39:30] 06Data-Engineering, 10Foundational Technology Requests, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Enable the Marketing Campaigns Reporting plugin for matomo - https://phabricator.wikimedia.org/T319013#9608188 (10SCampos-WMF) Thank you @BTullis for deploying this so quickly! As mentioned on Slack, I just... [17:45:08] the maps tile invalidation service failed sending data to eventgate earlier - interestingly it was a connection refused rather than a 503 but possibly related to T249745. We had a spike in 503 errors for jobs failing to enqueue again today [17:45:09] T249745: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745 [17:46:54] hnowlan: :-( [17:53:11] hnowlan: Which eventgate service? Main? It looks like there was some kind of blip at about 17:22 https://logstash.wikimedia.org/goto/8c5502226f2dfc7ea9cb9868cea88266 but that was eventgate-analytics-external [17:56:22] btullis: a few times today but the ones I observed were around 11:40-12:00 https://grafana.wikimedia.org/goto/lYdPVpAIk?orgId=1 [17:56:46] resulted in quite a few lost events https://logstash.wikimedia.org/goto/44c01345cc4d79dcf01a52ea91423964 [17:57:14] frustratingly I cannot find a hint of an issue on the eventgate side, there are lots of errors but no spikes that correspond with these spikes [17:58:55] Yes, I see. Very frustrating. [18:06:28] if claim.e's change to the heap usage was a help it doesn't seem like the service itself is actually taking advantage of the increase https://grafana.wikimedia.org/goto/q46FSp0Sz?orgId=1 [19:01:59] 06Data-Engineering: [Airflow] SparkSqlOperator fails when executing via Skein with master=local - https://phabricator.wikimedia.org/T359435 (10mforns) 03NEW [19:35:40] (03CR) 10Snwachukwu: Mediawiki History Data Quality Metrics (038 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1008934 (https://phabricator.wikimedia.org/T354692) (owner: 10Snwachukwu) [20:14:43] 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, 06serviceops, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9608695 (10ssastry) Given the six 9's reliability that Joe cited above ( T249745#958681 ) and... [20:14:54] 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, 06serviceops, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9608697 (10ssastry) p:05Unbreak!→03High [20:22:43] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products: arywiki view stats too low for agent = user? - https://phabricator.wikimedia.org/T359004#9608741 (10Maurusian) [20:23:03] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Data Products: arywiki view stats too low for agent = user? - https://phabricator.wikimedia.org/T359004#9608745 (10Maurusian) [20:39:52] (03PS2) 10Snwachukwu: Mediawiki History Data Quality Metrics [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1008934 (https://phabricator.wikimedia.org/T354692) [21:39:11] 10Data-Engineering (Sprint 9), 06Data Products: Adapt Sqoop to pagelinks schema change - https://phabricator.wikimedia.org/T345771#9609371 (10lbowmaker) 05Open→03Resolved [21:41:52] 10Data-Engineering (Sprint 9): Turn off ReportUpdater jobs no longer used - https://phabricator.wikimedia.org/T357419#9609375 (10lbowmaker) 05Open→03Resolved [21:41:54] 06Data-Engineering, 10Data Pipelines: [Airflow Migration] Migrate 1+ reportupdater jobs - https://phabricator.wikimedia.org/T307540#9609376 (10lbowmaker) [21:42:34] 10Data-Engineering (Sprint 9), 06Data Products, 06Movement-Insights, 10Movement-Metrics, 13Patch-For-Review: Skip Wikidata when loading XML dumps to the Data Lake - https://phabricator.wikimedia.org/T357859#9609378 (10lbowmaker) 05Open→03Resolved [21:53:26] 06Data-Engineering, 06Movement-Insights: Keep canonical_data.wikis updated - https://phabricator.wikimedia.org/T241741#9609410 (10JAnstee_WMF) p:05High→03Medium