[02:22:36] (03CR) 10Ottomata: [C:03+1] "Nits, but LGTM." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [04:48:23] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10005622 (10Fabfur) >>! In T370668#10003489, @Ottomata wrote: > I might be out of my league here, but have yall considered the [[ https://www.haproxy.com/blog/exte... [04:55:12] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741 (10Fabfur) 03NEW [07:37:25] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 2 others: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#10005759 (10Gehel) >>! In T367403#10003586... [07:43:26] (03CR) 10DCausse: [C:03+1] "lgtm!" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [09:16:01] (03PS23) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:19:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [09:19:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:43:59] 14Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#10006018 (10diego) Hi! Apparently the data has missing again: ` SELECT revision_id, revision_timestamp FROM wmf.mediawiki_wikitext_history... [09:44:21] 14Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#10006020 (10diego) 05Resolved→03Open [09:44:56] (03PS24) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:45:53] (03PS25) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [09:55:38] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review, 10Release-Engineering-Team (Seen): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10006086 (10Stevemunene) We are now upgrading to the latest version v2.9.3. There are still some... [10:00:27] (03PS8) 10Peter Fischer: Introducing cirrussearch/weighted_tags [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) [10:01:44] (03CR) 10Peter Fischer: "Thank you for your comments! I adapted the schema accordingly." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [10:05:20] 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review, 10Release-Engineering-Team (Seen): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10006135 (10Stevemunene) [10:17:21] 06Data-Engineering, 10Data-Engineering-Wikistats, 07dark-mode: Dark mode support for stats.wikimedia.org - https://phabricator.wikimedia.org/T370758 (10Diskdance) 03NEW [10:19:03] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [10:19:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:38:29] (03PS26) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [12:41:22] (03CR) 10DCausse: [C:03+1] "nice!" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [13:00:00] (03CR) 10Ottomata: [C:03+2] Introducing cirrussearch/weighted_tags [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1055226 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [13:59:40] 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#10006906 (10Jclark-ctr) @BTullis can this drive be changed at anytime? [14:05:56] (03PS1) 10Peter Fischer: Require wiki_id for development/cirrussearch/page_weighted_tags_change [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1056169 (https://phabricator.wikimedia.org/T366253) [14:17:53] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10006973 (10Ottomata) @Antoine_Quhen @tchin and I brainbounced this problem today. The summary was: if we want to handle late events well, we need to re-write the Gobblin+Refine... [14:21:32] (03CR) 10DCausse: [C:03+1] Require wiki_id for development/cirrussearch/page_weighted_tags_change [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1056169 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [14:22:45] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10006982 (10Ottomata) There are questions we should answer before we spend too much time on this: 1. How often are late events encountered? I'd expect not that often, since we p... [15:24:58] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10007249 (10mforns) Since we are moving to using ExternalTaskSensor to track DAG dependencies (instead of data sensors), we could take advantage of the ExternalTaskMarker and allo... [16:37:37] 06Data-Engineering, 06collaboration-services, 10Data Pipelines, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10007638 (10LSobanski) [17:05:52] 06Data-Engineering, 06collaboration-services, 10Data Pipelines, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10007775 (10thcipriani) This is an interesting problem. It seems that the trusted runners should have picked up t... [18:08:36] 06Data-Engineering, 06Structured-Data-Backlog: DagProperties don't automatically update Airflow variables - https://phabricator.wikimedia.org/T348963#10008042 (10mforns) We could also restrict the creation of Airflow Variables and all the overriding to the dev environment. And let the DagProperties module be a... [18:09:40] deploying refinery-source [18:10:04] Starting build #11 for job analytics-refinery-maven-release [18:27:30] Project analytics-refinery-maven-release build #11: 09SUCCESS in 17 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/11/ [18:36:04] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10008166 (10Ottomata) > I would need a serious help w/ C. Ya, me too! Perhaps the SPOE go lib @Vgutierrez mentioned might be easier? > ATM we decided to go down... [18:38:51] Starting build #11 for job analytics-refinery-update-jars [18:39:02] (03CR) 10Ottomata: [C:03+2] "I wonder if we should consider making a fragment schema just for this field... 😊" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/1056169 (https://phabricator.wikimedia.org/T366253) (owner: 10Peter Fischer) [18:40:27] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.45 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1056225 [18:40:28] Project analytics-refinery-update-jars build #11: 09SUCCESS in 1 min 36 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/11/ [18:41:56] 06Data-Engineering: Handle Late-Arrived Events from Gobblin into Airflow triggered Refine - https://phabricator.wikimedia.org/T370665#10008180 (10Ottomata) Oh ho! What a great idea! > Note that, the dag_run can be promptly re-run many times, but only when there are updates. This is how the Refine late event... [18:46:47] (03CR) 10Milimetric: [C:03+2] Add refinery-source jars for v0.2.45 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1056225 (owner: 10Maven-release-user) [18:46:53] (03CR) 10Milimetric: [V:03+2 C:03+2] Add refinery-source jars for v0.2.45 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1056225 (owner: 10Maven-release-user) [18:58:27] !log done deploying refinery-source, deploying airflow dags now [18:58:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:05:57] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Add host level instrumentation on webrequest - https://phabricator.wikimedia.org/T362785#10008280 (10Milimetric) I deployed this and started the job, checking in now to make sure it runs. [19:06:48] 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: Add instrumentation for actor signatures - https://phabricator.wikimedia.org/T362783#10008284 (10Milimetric) Deployed, started job, waiting to see if it works. [19:18:36] 14Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#10008300 (10Milimetric) The airflow sensor timed out. But I never saw an alert for it (maybe it was before this week). I cleared it and will report ba... [21:12:12] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 06Release-Engineering-Team: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#10008694 (10brennen) > Do you have an opinion on where to host artifacts coming from projects hoste... [21:27:06] 14Data-Engineering (Q4 2024 April 1st - June 30th): [Bug?] Can't find wikidatawiki on wmf.mediawiki_wikitext_history - https://phabricator.wikimedia.org/T364045#10008730 (10Milimetric) Ok, dug into this a bit more. Looks like the job set up to import the dumps XML is running fine but the status file says wikida... [21:32:38] 06Data-Engineering, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.14; 2024-07-16), 07Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814#10008755 (10pfische... [21:53:16] 06Data-Engineering, 10Data-Engineering-Wikistats, 06Web-Team-Backlog, 07dark-mode: Dark mode support for stats.wikimedia.org - https://phabricator.wikimedia.org/T370758#10008863 (10KSarabia-WMF) [23:06:12] 06Data-Engineering, 10Data-Engineering-Wikistats, 07dark-mode: Dark mode support for stats.wikimedia.org - https://phabricator.wikimedia.org/T370758#10009007 (10Jdlrobson)