[05:52:33] 10Data-Engineering, 10Data-Catalog, 10Data Engineering and Event Platform Team (Sprint 1), 10Event-Platform: Event Platform and DataHub Integration - https://phabricator.wikimedia.org/T318863 (10tchin) While adding a workaround to T344235, I noticed that `additionalProperties` isn't very well represented i... [07:04:05] 10Data-Platform-SRE, 10Discovery-Search: Unable to use kafka-topic.sh - Topic authorization failed - https://phabricator.wikimedia.org/T344989 (10Ladsgroup) [07:34:49] 10Data-Platform-SRE, 10Discovery-Search: Unable to use kafka-topic.sh - Topic authorization failed - https://phabricator.wikimedia.org/T344989 (10elukey) 05Open→03Resolved a:03elukey @pfischer ` elukey@kafka-jumbo1001:~$ kafka topics --describe --topic eqiad.mediawiki.page_change.v1 kafka-topics --zook... [08:34:08] Cross-posting from a research question since there's no answer: I'd like to publish a data set of page metrics calculated from the Enterprise HTML dump. It's ~3GB however, so I'm not sure how to host it efficiently for the long-term. Full post: https://lists.wikimedia.org/hyperkitty/list/wiki-research-l@lists.wikimedia.org/thread/M3IDFYT44O2NDGKKU7FG5Q25YTY4KGCS/ [08:37:04] awight: o/ one solution could be https://wikitech.wikimedia.org/wiki/Analytics/Web_publication, but keep in mind that that data is provided from a single node (with RAID etc..) but not backed up IIRC [08:37:22] (assuming also that the dataset doesn't contain PII etc..) [08:41:26] 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 1), 10Event-Platform: mw-page-content-change-enrich: filter out events larger than max.request.size - https://phabricator.wikimedia.org/T342399 (10gmodena) [08:41:36] 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 1), 10Event-Platform: mw-page-content-change-enrich: filter out events larger than max.request.size - https://phabricator.wikimedia.org/T342399 (10gmodena) a:03gmodena [08:45:26] elukey: Interesting possibility. Yes, it's all public data. 3-4 GB would fit there? Lack of backups is okay since we plan to repeat the analysis in the future. But it's a bit expensive: 2 months of 16-core processing for this last batch. [08:49:38] awight: there is space yes, but the Data Platform SRE folks should give their final +1 [08:49:49] I don't have any other valid alternative at the moment [09:36:18] elukey: Thanks, I'll look into this idea! [10:43:51] elukey's idea is the one I'd have suggested :) There is enough space for a few Gb dataset [10:43:55] awight: --^ [12:00:07] 10Data-Platform-SRE, 10Discovery-Search (Current work): Unable to use kafka-topic.sh - Topic authorization failed - https://phabricator.wikimedia.org/T344989 (10Gehel) [12:17:40] joal: o/ [12:45:18] 10Data-Engineering, 10Data Pipelines (Sprint 14), 10Data Products (Sprint 00), 10Google-Chrome-User-Agent-Deprecation, 10Product-Analytics (Kanban): [SPIKE] Model impact of User-Agent deprecation on top line metrics - https://phabricator.wikimedia.org/T336084 (10mforns) I need a background task while I w... [12:46:46] Hi elukey :) [13:15:12] 10Data-Engineering, 10Data Pipelines (Sprint 14), 10Data Products (Sprint 00), 10Google-Chrome-User-Agent-Deprecation, 10Product-Analytics (Kanban): [SPIKE] Model impact of User-Agent deprecation on top line metrics - https://phabricator.wikimedia.org/T336084 (10Milimetric) a:05Milimetric→03mforns [13:36:25] 10Data-Platform-SRE, 10Discovery-Search, 10Patch-For-Review: Create and publish new elastic dev image - https://phabricator.wikimedia.org/T344841 (10bking) Releng added permissions for me, so I merged David's patch. Per t[[ https://gitlab.wikimedia.org/repos/releng/dev-images#prerequisites-and-installation |... [13:54:44] 10Data-Platform-SRE, 10DC-Ops, 10decommission-hardware: decommission wdqs1005.eqiad.wmnet - https://phabricator.wikimedia.org/T345081 (10bking) [14:01:43] 10Data-Engineering, 10Product-Analytics, 10Patch-For-Review: Email notifications of new MediaWiki history snapshot availabilty - https://phabricator.wikimedia.org/T344854 (10CodeReviewBot) milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/481 Update mw_histo... [14:02:51] 10Data-Platform-SRE, 10Patch-For-Review: Decommission wdqs10[03-05] - https://phabricator.wikimedia.org/T344198 (10bking) [14:02:53] 10Data-Platform-SRE, 10DC-Ops, 10decommission-hardware: decommission wdqs1005.eqiad.wmnet - https://phabricator.wikimedia.org/T345081 (10bking) [14:25:26] 10Analytics-Radar, 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Epic: Explore an API for logging events sampled by session - https://phabricator.wikimedia.org/T168380 (10mpopov) @phuedx: Some extra details as you follow up on the protocol part. Here are some thoughts on the topic from me and... [15:11:09] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Event-Platform: Add $comment and $performer to ArticleRevisionVisibilitySet params - https://phabricator.wikimedia.org/T321411 (10Krinkle) I'm untagging #MediaWiki-Core-Hooks as this does not appear to be a proposal or problem about the Hooks... [16:57:23] 10Data-Platform-SRE: Export Blazegraph JNL file from wdqs1009 - https://phabricator.wikimedia.org/T344732 (10bking) This is complete! Closing... [17:03:14] 10Data-Engineering, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): Set data permission on new snapshot generation (discovery.wikibase_rdf) - https://phabricator.wikimedia.org/T342416 (10EBernhardson) New dataset for 20230821 has updated permissions as expected. [19:01:06] 10Data-Platform-SRE: Migrate WDQS and WCQS servers to Debian Bullseye - https://phabricator.wikimedia.org/T343124 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host wdqs1004.eqiad.wmnet with OS bullseye [19:09:46] 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Patch-For-Review: Provision Zookeeper Cluster for storing Flink HA data - https://phabricator.wikimedia.org/T341792 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bo... [19:36:09] 10Data-Platform-SRE: Migrate WDQS and WCQS servers to Debian Bullseye - https://phabricator.wikimedia.org/T343124 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host wdqs1004.eqiad.wmnet with OS bullseye completed: - wdqs1004 (**WARN**) - Downtimed on Icinga/Alertman... [19:38:22] 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Patch-For-Review: Provision Zookeeper Cluster for storing Flink HA data - https://phabricator.wikimedia.org/T341792 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host flink-zk2001.codfw.wmnet with OS bookwo... [20:47:39] (03CR) 10Mforns: Add analytics/metrics_platform/{app,web}_click schemas (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/952252 (https://phabricator.wikimedia.org/T344833) (owner: 10Phuedx) [20:49:34] 10Data-Platform-SRE: Investigate wdqs1005 (hangs/crashes) - https://phabricator.wikimedia.org/T344960 (10bking) 05Open→03Resolved a:03bking We've decided to decommission the host (see T345081 ). Closing ticket... [20:54:48] 10Data-Engineering, 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Event-Platform: Test common operations in the flink operator/k8s/Flink ZK environment - https://phabricator.wikimedia.org/T342149 (10bking) [21:17:26] 10Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work): Allow federated queries with the NLG endpoint (data.nlg.gr) - https://phabricator.wikimedia.org/T337296 (10bking) a:03bking [21:30:52] 10Data-Platform-SRE, 10Patch-For-Review: Decommission wdqs100[3-5] - https://phabricator.wikimedia.org/T344198 (10RKemper) [22:12:02] 10Data-Platform-SRE, 10DC-Ops, 10decommission-hardware, 10ops-eqiad: hw troubleshooting: ipmi down for wdqs1005.eqiad.wmnet - https://phabricator.wikimedia.org/T345081 (10RKemper) a:03Papaul [22:39:12] 10Data-Platform-SRE, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-eqiad: hw troubleshooting: ipmi down for wdqs1005.eqiad.wmnet - https://phabricator.wikimedia.org/T345081 (10Papaul) @Jclark-ctr @VRiley-WMF can someone please check the mgmt cable for this servers, I can not ping the mgmt IP if the cab... [22:47:26] 10Data-Engineering, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Movement-Insights, 10WMDE-FUN-Sprint-2023-08-21: Unique Devices seasonal trends on small projects - https://phabricator.wikimedia.org/T344381 (10Mayakp.wiki) cc: @odimitrijevic , @Milimetric tagging Data-platform-engineer... [23:11:27] 10Data-Engineering, 10Movement-Insights, 10Product-Analytics, 10Research-Freezer: Investigate relation of UA deprecation to increase in automated traffic and reduction in unique devices - https://phabricator.wikimedia.org/T336715 (10Mayakp.wiki) We are meeting this week to make a decision to turn off pre-f...