[01:28:29] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.192% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:28:29] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.064% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:27:23] * brouberol waves good morning [08:30:52] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE: [Data Platform] Deploy Spark History Service - https://phabricator.wikimedia.org/T330176 (10brouberol) [08:33:29] btullis: I was thinking about the upcoming freeze and the spark-history server. Maybe a conservative deployment objective would be to have the spark-history-analytics-hadoop server deployed and integrated behind the YARN UI *for jobs that we manually start with the spark.eventLog.dir and spark.yarn.historyServer.address configs*, and release these [08:33:29] default config values after the thaw? [08:43:15] 10Data-Engineering (Sprint 6), 10Patch-For-Review: [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables - https://phabricator.wikimedia.org/T347879 (10CodeReviewBot) aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/564 Unique devices druid... [08:55:48] (03PS1) 10Aqu: Unique devices druid ingestion job - Iceberg migration [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983673 (https://phabricator.wikimedia.org/T347879) [09:00:20] (03PS1) 10Aqu: Remove Oozie folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983674 [09:04:54] brouberol: sounds good to me. [09:05:38] nice! So, in that case we'd merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/983193 and then use your jupyter notebook to trigger spark jobs with these 2 configs? [09:06:19] We can check that it works in principle, end to end. Maybe we can work on the docs a bit too. [09:08:03] yep, I was planning to leverage the code freeze to work on monitoring, docs and such [09:08:18] alright, I'll be merging that Apache PR [09:11:11] Yep, looks good. [09:22:57] !log deploying refinery version 0.02.27 to production refinery jobs with https://gerrit.wikimedia.org/r/c/operations/puppet/+/980923 for T349121 [09:23:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:23:01] T349121: [metrics platform] Event sanitization fails when processing monoschema events - https://phabricator.wikimedia.org/T349121 [09:28:30] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.048% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:43:11] !log cleared some space on an-test-worker1001 by deleting old refinery jars from /tmp `btullis@an-test-worker1001:/tmp$ sudo find . -type f -mtime +60 -name *.jar -delete` [09:43:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:46:55] (DiskSpace) resolved: Disk space an-test-worker1001:9100:/ 1.156% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:26:14] 10Data-Engineering (Sprint 6), 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Configure the YARN resource manager with the spark history service URL - https://phabricator.wikimedia.org/T352863 (10brouberol) We have setup a proxy_pass rule from `https://yarn.wikimedia.org/history` to `https... [10:37:29] 10Data-Engineering (Sprint 6), 10Data Pipelines, 10Discovery-Search, 10Java-Scala-Standardization, 10Patch-For-Review: [Maintenance] We should have a top level maven parent pom based on wikimedia-discovery-discovery-parent-pom, - https://phabricator.wikimedia.org/T309097 (10Gehel) Decision: we'll publish... [10:50:21] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 (10BTullis) @nshahquinn-wmf - welcome back! >If I'm understanding correctly, after gerrit 709713 is deployed, Wmfdata < 2.2 will no lon... [10:54:04] !log deploy conda-analytics v 0.0.27 to the hadoop-test-analytics cluster for T345482 [10:54:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:54:07] T345482: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 [11:01:13] (DiskSpace) firing: Disk space an-test-ui1001:9100:/ 2.484% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-ui1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:32:27] 10Data-Engineering (Sprint 6), 10Section-Level-Image-Suggestions, 10Patch-For-Review, 10Structured-Data-Backlog (Current Work): [S] Coalesce section alignment image suggestions output - https://phabricator.wikimedia.org/T347558 (10CodeReviewBot) mfossati merged https://gitlab.wikimedia.org/repos/structured... [11:52:27] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 (10BTullis) I have verified that conda-analytics version 0.0.27 seems fine with jupyter. It can create a new conda environment and wmfd... [11:59:26] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 9 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10DAlangi_WMF) >>! In T350806#9410485, @matmarex wrote: > As discovered above, Wikibase (and CommonsMet... [11:59:39] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 9 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10DAlangi_WMF) [12:00:59] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 9 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10DAlangi_WMF) @Krinkle, what would be a good next logical step here? Maybe double check callers again... [12:30:42] (SystemdUnitFailed) firing: apache2.service Failed on an-tool1008:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:35:42] (SystemdUnitFailed) resolved: apache2.service Failed on an-tool1008:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:09:28] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 10 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10Krinkle) @DAlangi_WMF The problematic hook that initiated this task (via the parent task) is `onPage... [13:20:23] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Bring an-coord100[3-4] into service - https://phabricator.wikimedia.org/T336045 (10BTullis) I have discovered something interesting! Until now, we have only been running a single presto coordinator, which has been on an-coord1001. Our standby se... [13:26:17] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 10 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10DAlangi_WMF) [13:28:01] 10Data-Engineering, 10CommonsMetadata, 10DiscussionTools, 10Growth-Team, and 10 others: Phase out Title::getPageViewLanguage in favour of ParserOutput metadata - https://phabricator.wikimedia.org/T350806 (10DAlangi_WMF) >>! In T350806#9412606, @Krinkle wrote: > @DAlangi_WMF The problematic hook that initia... [13:34:34] 10Data-Platform-SRE (2023/24 Q2 Milestone 1): Create a helm chart for Superset - https://phabricator.wikimedia.org/T352166 (10BTullis) [13:41:59] 10Data-Engineering, 10Data-Platform-SRE, 10Epic: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10BTullis) [14:00:20] (03CR) 10Ottomata: [C: 03+1] "Wow" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983674 (owner: 10Aqu) [14:42:17] 10Data-Engineering, 10EventStreams, 10Event-Platform: EventStreams should deserialize Kafka key - https://phabricator.wikimedia.org/T353650 (10Ottomata) [14:44:47] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10BTullis) [14:45:24] 10Data-Engineering (Sprint 6), 10Section-Level-Image-Suggestions, 10Patch-For-Review, 10Structured-Data-Backlog (Current Work): [S] Coalesce section alignment image suggestions output - https://phabricator.wikimedia.org/T347558 (10mfossati) Deployed! [14:46:16] 10Data-Engineering (Sprint 6), 10Patch-For-Review: [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables - https://phabricator.wikimedia.org/T347879 (10Antoine_Quhen) I have the first version of the code in review. There is one drawback of the current version of the Iceberg sensor: t... [14:53:52] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10BTullis) > Evaluate the [[https://github.com/apache/superset/tree/master/helm/superset|upstream chart]] and if appropri... [15:01:14] (DiskSpace) firing: Disk space an-test-ui1001:9100:/ 2.47% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-ui1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:04:06] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10BTullis) [15:06:14] (DiskSpace) resolved: Disk space an-test-ui1001:9100:/ 2.47% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-ui1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:13:04] (03PS2) 10Aqu: Remove Oozie folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983674 (https://phabricator.wikimedia.org/T336739) [15:15:32] (03PS19) 10Gmodena: refinery-job: add WebrequestMetrics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/979359 (https://phabricator.wikimedia.org/T349763) [15:19:31] (03PS20) 10Ottomata: refinery-job: add WebrequestMetrics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/979359 (https://phabricator.wikimedia.org/T349763) (owner: 10Gmodena) [15:20:16] (03PS21) 10Ottomata: refinery-job: add WebrequestMetrics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/979359 (https://phabricator.wikimedia.org/T349763) (owner: 10Gmodena) [15:22:25] (03CR) 10Ottomata: refinery-job: add WebrequestMetrics. (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/979359 (https://phabricator.wikimedia.org/T349763) (owner: 10Gmodena) [15:23:26] (03PS4) 10TChin: Add iceberg version of aqs_hourly table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) [15:24:13] (03CR) 10Joal: [C: 03+1] "Thanks for this!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983674 (https://phabricator.wikimedia.org/T336739) (owner: 10Aqu) [15:25:50] 10Data-Platform-SRE, 10GitLab (Project Migration): Migrate https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/ to gitlab - https://phabricator.wikimedia.org/T353275 (10Gehel) [15:32:16] (03PS22) 10Gmodena: refinery-job: add WebrequestMetrics. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/979359 (https://phabricator.wikimedia.org/T349763) [15:42:47] Hello SREs ! My puppet patch was merge, and it'c cool. Thanks. But, according to what I see with a browser on an-test-client1002 from the statsd-exporter (on ...:9112/metrics), I think we may need to restart the puppet agent as the file /etc/prometheus/statsd_exporter.conf has not been updated from here: https://github.com/wikimedia/operations-puppet/blob/production/hieradata/common/profile/airflow.yaml Could you help me [15:42:47] ? cc butllis stevemunene [15:43:24] https://gerrit.wikimedia.org/r/c/operations/puppet/+/979118 [15:48:00] aqu: hi! can you test it now? [15:48:31] I restarted the prometheus statsd exporter on the node [15:54:13] (03PS1) 10Xcollazo: Revert "Revert "Add the wikifunctions_ui metrics platform schema to the allowlist"" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983740 (https://phabricator.wikimedia.org/T349121) [16:10:48] (03CR) 10Joal: "One comment about naming - the code looks good :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [16:12:28] Thanks elukey ! I still have a problem, but I think it's because of the mapping itself in production compare to my tests... [16:13:04] (03CR) 10Joal: [C: 04-1] "Nit" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/972348 (owner: 10Phuedx) [16:14:26] 10Data-Platform-SRE, 10Discovery-Search (Current work): Test backfilling for cirrus-streaming-updater - https://phabricator.wikimedia.org/T350826 (10Gehel) We probably want to run the backfill test once we have enabled all wikis, so that the load generated is maximized. [16:14:51] 10Data-Platform-SRE, 10Discovery-Search (Current work): Migrate search-loader into Kubernetes - https://phabricator.wikimedia.org/T346189 (10bking) [16:15:20] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10serviceops, 10Discovery-Search (Current work): Enable mediawiki.cirrussearch.page_rerender.v1 on all public wikis - https://phabricator.wikimedia.org/T351503 (10pfischer) [16:15:27] (03CR) 10Joal: "I have not checked the code details, but overall structure looks ok. I wonder if this change implies growing the data by a lot and if it s" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/958518 (https://phabricator.wikimedia.org/T345446) (owner: 10Fabian Kaelin) [16:18:40] 10Data-Platform-SRE: ProbeDown - https://phabricator.wikimedia.org/T353652 (10bking) a:03bking [16:22:17] (03CR) 10Xcollazo: "Some comments below." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [16:31:35] (03CR) 10Joal: Add iceberg version of aqs_hourly table (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [16:46:46] (03CR) 10Xcollazo: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [17:03:20] (03CR) 10Joal: "Open question: Should we fix the issue earlier in the pipeline process, possibly at webrequest level, so that all downstream datasets are " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982899 (https://phabricator.wikimedia.org/T353296) (owner: 10Milimetric) [17:06:02] (03CR) 10Joal: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [17:25:18] 10Data-Engineering, 10Structured-Data-Backlog (Current Work): NEW BUG REPORT fiwiki’s section-level image suggestions aren’t generated in production - https://phabricator.wikimedia.org/T343844 (10MarkTraceur) [17:25:30] 10Data-Engineering, 10Data Pipelines: Refine: Use Spark SQL instead of Hive JDBC - https://phabricator.wikimedia.org/T209453 (10Ottomata) A few more investigations: @JAllemandou suggested that we might be able to use the v2 alter table syntax on our existent hive parquet tables. This does not work (at least... [17:35:32] 10Data-Engineering, 10Data Pipelines: [Migrate] Oozie jobs migration for Webrequest - https://phabricator.wikimedia.org/T324484 (10Ottomata) 05Open→03Resolved a:03Ottomata Being bold and closing this. CC @Antoine_Quhen [17:35:44] 10Data-Engineering, 10Observability-Logging, 10Traffic: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Ottomata) [17:35:48] 10Data-Engineering, 10Event-Platform: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10Ottomata) [17:41:18] 10Data-Engineering, 10Observability-Logging, 10Traffic: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Ottomata) Hey all, whatever the chosen solution for producing logs, we need a nice migration plan. I just talked with @Ahoelzl, and we realized that it would prob... [17:42:19] 10Data-Engineering, 10Observability-Logging, 10Traffic: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Ottomata) I'm going to work on T314956 now, before I go on sabbatical + parent leave in January. (I'll be out Jan 8 - April 12 2024.) I hope to get the new stream... [17:42:25] (03CR) 10Xcollazo: Add iceberg version of aqs_hourly table (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/982869 (https://phabricator.wikimedia.org/T352669) (owner: 10TChin) [17:43:44] 10Data-Engineering, 10Event-Platform: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10Ottomata) We are going to do this, but as a new stream. This new stream will be used for HAProxy logging as described in https://phabricator.wikimedia.org/T351117... [17:55:56] 10Data-Engineering: Remove nickifeajika from analytics-privatedata-users - https://phabricator.wikimedia.org/T353665 (10fkaelin) [18:18:46] 10Data-Platform-SRE: ProbeDown - https://phabricator.wikimedia.org/T353652 (10bking) This error is a result of merging [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/983438 | this change ]] without enabling IPv6 for envoy proxy. I've prepared [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/98389... [18:20:29] (03PS1) 10Ottomata: WIP - Add webrequest schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/983898 (https://phabricator.wikimedia.org/T314956) [18:21:00] (03CR) 10CI reject: [V: 04-1] WIP - Add webrequest schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/983898 (https://phabricator.wikimedia.org/T314956) (owner: 10Ottomata) [18:23:55] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983740 (https://phabricator.wikimedia.org/T349121) (owner: 10Xcollazo) [18:30:53] 10Data-Engineering, 10Data Pipelines: Refine: Use Spark SQL instead of Hive JDBC - https://phabricator.wikimedia.org/T209453 (10Ottomata) Was able to try this in Spark 3.5 via @xcollazo's instructions `USING hive`: ` scala> spark.sql("ALTER TABLE otto.test_table04 ADD COLUMN s1.added_field0 STRING") ` ` org.a... [18:34:18] (03PS2) 10Ottomata: WIP - Add webrequest schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/983898 (https://phabricator.wikimedia.org/T314956) [18:34:43] (03CR) 10CI reject: [V: 04-1] WIP - Add webrequest schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/983898 (https://phabricator.wikimedia.org/T314956) (owner: 10Ottomata) [19:01:54] (03PS1) 10Ottomata: WIP - Add gobblin job webrequest_frontend to pull new webrequest stream [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983926 (https://phabricator.wikimedia.org/T314956) [19:02:21] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10Ottomata) [19:28:32] 10Data-Platform-SRE, 10Wikidata, 10collaboration-services: Review alerts for miscweb-hosted domains commons-query.wikimedia.org and query.wikidata.org - https://phabricator.wikimedia.org/T352921 (10Dzahn) @bking @Gehel Meanwhile we have duplicate both monitoring checks. So for query.wikidata.org and commons-... [19:36:15] 10Data-Platform-SRE, 10Wikidata, 10collaboration-services: Review alerts for miscweb-hosted domains commons-query.wikimedia.org and query.wikidata.org - https://phabricator.wikimedia.org/T352921 (10bking) No objections from the Search Platform side. Side note, our new team (Data Platform SRE) is [[ https://p... [19:36:33] 10Data-Platform-SRE, 10Epic: [Epic] define a strategy around alerting for Data Platform SRE and implement it - https://phabricator.wikimedia.org/T345698 (10bking) [19:36:35] 10Data-Platform-SRE, 10Wikidata, 10collaboration-services: Review alerts for miscweb-hosted domains commons-query.wikimedia.org and query.wikidata.org - https://phabricator.wikimedia.org/T352921 (10bking) 05Open→03Resolved [19:46:16] (03CR) 10Xcollazo: [C: 03+2] "Merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983740 (https://phabricator.wikimedia.org/T349121) (owner: 10Xcollazo) [19:46:48] (03CR) 10Xcollazo: [V: 03+2 C: 03+2] Revert "Revert "Add the wikifunctions_ui metrics platform schema to the allowlist"" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983740 (https://phabricator.wikimedia.org/T349121) (owner: 10Xcollazo) [19:52:24] 10Data-Platform-SRE: Expose Prometheus Blackbox's ability to add http headers in puppet module - https://phabricator.wikimedia.org/T353672 (10bking) [19:52:42] 10Data-Platform-SRE: Expose Prometheus Blackbox Exporter's ability to add http headers in puppet module - https://phabricator.wikimedia.org/T353672 (10bking) [19:54:40] 10Data-Platform-SRE, 10Wikidata, 10collaboration-services: Review alerts for miscweb-hosted domains commons-query.wikimedia.org and query.wikidata.org - https://phabricator.wikimedia.org/T352921 (10Dzahn) Thank you @bking! There is one more little thing, we can enable TLS now and get this merged: https://ger... [20:12:01] 10Data-Engineering (Sprint 6), 10Event-Platform: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys - https://phabricator.wikimedia.org/T338231 (10Ottomata) [20:24:12] 10Data-Platform-SRE (2023/24 Q2 Milestone 1), 10Patch-For-Review: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 (10nshahquinn-wmf) >>! In T345482#9412275, @BTullis wrote: > That is absolutely correct. > > This roll-out is going to have to happen... [21:08:13] 10Data-Engineering (Sprint 6), 10Event-Platform: [Event Platform] mediawiki.page_content_change.v1 topic should be partitioned. - https://phabricator.wikimedia.org/T345806 (10Ottomata) To apply these changes: - Suspended flink app: `helmfile -e apply --set app.job.state='suspended'` - Alter kafka topic... [21:08:26] 10Data-Engineering (Sprint 6), 10Event-Platform, 10Patch-For-Review: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys - https://phabricator.wikimedia.org/T338231 (10Ottomata) [21:10:56] 10Data-Engineering (Sprint 6), 10Event-Platform, 10Patch-For-Review: [Event Platform] mw-page-content-change-enrich should (re)produce kafka keys - https://phabricator.wikimedia.org/T338231 (10Ottomata) @gmodena and I deployed this and {T345806} today. We did not add any Kafka partitions to mediawiki.page_c...