[02:32:29] (03CR) 10Gergő Tisza: [C: 03+2] homepagemodule: Document total_pageviews_count in action_data [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886355 (https://phabricator.wikimedia.org/T328391) (owner: 10Kosta Harlan) [02:33:02] (03Merged) 10jenkins-bot: homepagemodule: Document total_pageviews_count in action_data [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886355 (https://phabricator.wikimedia.org/T328391) (owner: 10Kosta Harlan) [02:49:51] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:00:31] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:45:53] 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10Joe) >>! In T300977#7899855, @jbond wrote: >>>! In T300977#7836272, @Volans wrote: >> If I may add my use case too, I... [08:28:49] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:29:05] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) We'll depool eqiad I would assume? cc @Joe @akosiaris We'd still need to switchover m1 master (we do have m1 databases but I guess we are... [08:41:27] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:45:54] (03CR) 10DCausse: Remove Guava from dependency (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) (owner: 10Aqu) [09:26:11] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): Remove hardcoded kafka parameters - https://phabricator.wikimedia.org/T329061 (10gmodena) [09:32:31] 10Data-Engineering-Planning, 10Observability-Alerting, 10SRE, 10Shared-Data-Infrastructure, 10Traffic: Reduce/eliminate false positives for VarnishKafkaNoMessages alert - https://phabricator.wikimedia.org/T324522 (10nfraison) a:05BTullis→03nfraison [09:41:57] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10akosiaris) eqiad will still be depooled for this one. The current timeline for repooling eqiad in on March 8th, 1 day after the proposed timeline on... [09:45:40] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [10:01:58] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [10:08:29] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10elukey) @Ottomata we decided, as a team, to offer the support for simple Streams via Change-prop leaving the choice of the source event... [10:16:37] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) [10:26:54] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [10:57:02] 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10jbond) @Joe thanks for the input >>! In T300977#8596499, @Joe wrote: > > This would break a lot of workflows, I t... [11:00:13] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) [11:14:21] 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, 10Research, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) > I would maintain that it's more urgent to provide an artifact repository for having local npm/pypi/go pack... [11:54:40] elukey: o/ [11:56:05] FYI I'm about to shut down dse-k8s-worker1001,an-worker1096 and an-worker1097 for T318696 [11:56:06] T318696: Attempt to move some GPUs from Hadoop to the DSE-K8S cluster - https://phabricator.wikimedia.org/T318696 [12:00:40] btullis: sure ack! [12:02:34] Hopefully it will go well. Will let you know. [12:04:43] !log shut down an-worker109[67] and dse-k8s-worker1001 ready for GPU swap. [12:04:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:04:26] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Marostegui) [13:04:33] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Marostegui) [13:38:20] 10Data-Engineering, 10Data-Catalog, 10Infrastructure-Foundations, 10CAS-SSO, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10EChetty) [13:39:10] 10Data-Engineering, 10Data-Catalog, 10Infrastructure-Foundations, 10CAS-SSO, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10EChetty) a:05BTullis→03Stevemunene [13:43:10] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) > live with any reliability promises: TBD Wait! Actually, what I said here is not true! I believe we will have `mediawiki.p... [13:43:26] 10Data-Engineering-Planning, 10Observability-Alerting, 10SRE, 10Traffic, and 2 others: Reduce/eliminate false positives for VarnishKafkaNoMessages alert - https://phabricator.wikimedia.org/T324522 (10EChetty) [13:45:24] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): Remove hardcoded kafka parameters - https://phabricator.wikimedia.org/T329061 (10gmodena) I extracted hardcoded strings into properties. Currently these are populated by env variables, till we have more clarity in OP task. I took a look at this py... [14:05:19] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08): Remove hardcoded kafka parameters - https://phabricator.wikimedia.org/T329061 (10JArguello-WMF) [14:14:49] Hi mforns - copying from ops chan here: IIRC you planned yesterday on doing a deploy today - is that correct? [14:44:07] (03CR) 10Bearloga: "Correct me if I'm wrong but shouldn't hql/webrequest/create_webrequest_table.hql also be updated?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887371 (https://phabricator.wikimedia.org/T327074) (owner: 10Snwachukwu) [14:45:06] (03CR) 10Joal: Update Webrequest table to include referer_data column. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887371 (https://phabricator.wikimedia.org/T327074) (owner: 10Snwachukwu) [14:58:46] 10Data-Engineering, 10Equity-Landscape: Population input metrics - https://phabricator.wikimedia.org/T309279 (10JAnstee_WMF) @KCVelaga_WMF - With the caveat of the column name change, signing off on the QA for the data [14:59:05] 10Data-Engineering, 10Equity-Landscape: Population output rank metrics - https://phabricator.wikimedia.org/T306624 (10JAnstee_WMF) @KCVelaga_WMF - With the caveat of the column name changes, signing off on the QA for the data [15:05:53] mforns, milimetric: I'll be back around standup tiem - is it ok for ou if we deploy at that time? [15:36:54] 10Data-Engineering, 10Equity-Landscape: Population output rank metrics - https://phabricator.wikimedia.org/T306624 (10KCVelaga_WMF) a:05KCVelaga_WMF→03ntsako [15:38:19] 10Data-Engineering, 10Equity-Landscape: Access output metrics - https://phabricator.wikimedia.org/T329185 (10KCVelaga_WMF) [15:39:04] 10Data-Engineering, 10Equity-Landscape: Access output metrics - https://phabricator.wikimedia.org/T329185 (10KCVelaga_WMF) a:05ntsako→03KCVelaga_WMF [15:39:26] joal: sure [15:41:08] 10Data-Engineering-Planning, 10Equity-Landscape: Load language data - https://phabricator.wikimedia.org/T315886 (10ntsako) ` SELECT * FROM ntsako.brief_projects_edited_metrics WHERE year=2021 SELECT * FROM ntsako.unesco_endangered_lang_metrics WHERE year=2021 ` Take the above to production. [15:43:15] 10Data-Engineering-Planning, 10Equity-Landscape: Load language data - https://phabricator.wikimedia.org/T315886 (10JAnstee_WMF) Note that the active projects are those which are edited by at least 1% and 5 or more editors in the region each month on average during [15:44:14] 10Data-Engineering-Planning, 10Equity-Landscape: Load language data - https://phabricator.wikimedia.org/T315886 (10KCVelaga_WMF) query ` WITH monthly_avg AS ( SELECT country_code, ROUND(AVG(distinct_editors)) AS average_monthly_acitve_editors, wiki_db FR... [15:44:42] 10Data-Engineering, 10Event-Platform Value Stream: [Flink Operations] How to handle restarting a Flink application - https://phabricator.wikimedia.org/T328563 (10lbowmaker) [15:45:35] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 09): [Flink Operations] How to handle restarting a Flink application - https://phabricator.wikimedia.org/T328563 (10JArguello-WMF) [15:48:52] 10Data-Engineering, 10Event-Platform Value Stream: [Flink Operation] How to handle app upgrades - https://phabricator.wikimedia.org/T328569 (10lbowmaker) [15:49:19] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 09): [Flink Operation] How to handle app upgrades - https://phabricator.wikimedia.org/T328569 (10JArguello-WMF) [15:58:05] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Productionize PyFlink Enrichment Service - https://phabricator.wikimedia.org/T325303 (10lbowmaker) [15:58:14] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Productionize PyFlink Enrichment Service - https://phabricator.wikimedia.org/T325303 (10lbowmaker) a:03Ottomata [15:58:16] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 08), 10MW-1.40-notes (1.40.0-wmf.23; 2023-02-13): mediawiki.page-undelete stream is empty - https://phabricator.wikimedia.org/T329064 (10JArguello-WMF) [15:58:38] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Productionize PyFlink Enrichment Service - https://phabricator.wikimedia.org/T325303 (10lbowmaker) [15:59:45] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10Ottomata) [16:00:28] 10Data-Engineering-Planning, 10Event-Platform Value Stream: Deploy to production k8s - https://phabricator.wikimedia.org/T325307 (10Ottomata) [16:00:32] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10Ottomata) [16:10:08] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10JAnstee_WMF) @ntsako Reassigning following review sign-off for update to production [16:10:24] 10Data-Engineering, 10Equity-Landscape: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [16:44:56] Starting build #17 for job wikimedia-event-utilities-maven-release-docker [16:47:20] 10Data-Engineering-Planning, 10Data-Catalog, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10BTullis) [16:47:22] 10Data-Engineering, 10Data-Catalog, 10Infrastructure-Foundations, 10CAS-SSO, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) [16:47:44] Project wikimedia-event-utilities-maven-release-docker build #17: 09SUCCESS in 2 min 47 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/17/ [16:57:51] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 08), 10Patch-For-Review, 10SecTeam-Processed, 10Vuln-VulnComponent: Upgrade Puppet code to make Airflow configuration files compatible with version 2.3.4 - https://phabricator.wikimedia.org/T315580 (10Stevemunene) >>! In T315580#8543559, @Antoine_Quh... [17:06:01] (03PS1) 10Joal: Add hql edit hourly computation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887810 [17:08:58] PROBLEM - Checks that the airflow database for airflow search is working properly on an-airflow1005 is CRITICAL: CRITICAL: /usr/bin/env AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow db check did not succeed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow [17:10:36] PROBLEM - Checks that the local airflow scheduler for airflow @search is working properly on an-airflow1005 is CRITICAL: CRITICAL: /usr/bin/env AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow jobs check --job-type SchedulerJob --hostname an-airflow1005.eqiad.wmnet did not succeed https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow [17:33:21] (03PS4) 10Mforns: Support snapshot partitioning in HiveToDruid and DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/886114 (https://phabricator.wikimedia.org/T324485) [17:34:34] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add hql edit hourly computation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887810 (owner: 10Joal) [17:35:06] (03CR) 10Mforns: Support snapshot partitioning in HiveToDruid and DataFrameToDruid (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/886114 (https://phabricator.wikimedia.org/T324485) (owner: 10Mforns) [17:46:12] (03CR) 10Ottomata: [C: 03+1] Support snapshot partitioning in HiveToDruid and DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/886114 (https://phabricator.wikimedia.org/T324485) (owner: 10Mforns) [17:47:41] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [17:49:20] (03CR) 10Milimetric: [C: 03+2] "love how much duplicated code this gets rid of. Thank you!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/886114 (https://phabricator.wikimedia.org/T324485) (owner: 10Mforns) [17:56:55] (03Merged) 10jenkins-bot: Support snapshot partitioning in HiveToDruid and DataFrameToDruid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/886114 (https://phabricator.wikimedia.org/T324485) (owner: 10Mforns) [18:00:24] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Discovery-Search, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [18:09:23] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/884359 (https://phabricator.wikimedia.org/T324483) (owner: 10Joal) [18:15:04] (03PS9) 10Milimetric: [WIP] Stream revision topics into iceberg table [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/858344 (https://phabricator.wikimedia.org/T322326) [18:18:43] (03CR) 10CI reject: [V: 04-1] [WIP] Stream revision topics into iceberg table [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/858344 (https://phabricator.wikimedia.org/T322326) (owner: 10Milimetric) [18:21:10] (03PS1) 10Milimetric: Update changelog.md with v0.2.11 changes [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/887813 [18:21:38] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/887813 (owner: 10Milimetric) [18:22:16] Starting build #116 for job analytics-refinery-maven-release-docker [18:34:31] Project analytics-refinery-maven-release-docker build #116: 09SUCCESS in 12 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/116/ [18:43:58] Starting build #75 for job analytics-refinery-update-jars-docker [18:44:11] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.11 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/886858 [18:44:12] Project analytics-refinery-update-jars-docker build #75: 09SUCCESS in 13 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/75/ [18:54:43] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add refinery-source jars for v0.2.11 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/886858 (owner: 10Maven-release-user) [19:26:57] !log finished deploying refinery-source 0.2.11, refinery, and synced to hdfs [19:26:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:31:40] hm, don't wanna bother people, but I said I'd ping... [19:31:41] hi! [19:31:44] I finished deploying! [19:34:07] hi milimetric :) [19:34:16] :) [19:34:31] I'm resolving merge conflicts from my patch now... no rush on deploying tonight, up to you tho [19:35:10] milimetric: I'll feel more comfy tomorrow, but I also wish not to exclude you (it'll be too early tomorrow morning) [19:35:46] no worries, I'm around if you need support, but I know what's involved so I don't feel like I'm missing out. Maybe one of the new folks would want to pair? Jennifer? [19:36:03] it could be a very educational deploy [19:36:04] milimetric: good call - I'll ping tomorrow before starting :) [21:51:18] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Create airflow v2 instance and supporting repos for search platform - https://phabricator.wikimedia.org/T327970 (10bking) @BTullis no worries, I run that job regularly on my laptop. It was actually Puppet failing due to a missi... [22:02:28] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): migrate mjolnir application and dag to airflow v2 and spark3 - https://phabricator.wikimedia.org/T329239 (10EBernhardson) [22:03:08] (03CR) 10Phedenskog: [C: 03+2] Remove elementtiming,firstinputtiming,layoutshift,resourcetiming,rumspeedindex [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/887425 (https://phabricator.wikimedia.org/T281103) (owner: 10Krinkle) [22:03:43] (03Merged) 10jenkins-bot: Remove elementtiming,firstinputtiming,layoutshift,resourcetiming,rumspeedindex [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/887425 (https://phabricator.wikimedia.org/T281103) (owner: 10Krinkle)