[00:08:02] <wikibugs>	 (03CR) 10Aleksandar Mastilovic: "LGTM in general! I agree with pulling onYarn and mkResultKey into a DataQualityHelper trait." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[03:19:03] <jinxer-wm>	 FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ...
[03:19:03] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[04:19:03] <jinxer-wm>	 RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ...
[04:19:04] <jinxer-wm>	 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=codfw.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected
[05:32:55] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9983949 (10Marostegui)
[05:44:07] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9983962 (10Marostegui)
[05:58:08] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122 (10Marostegui) 03NEW
[06:09:28] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122#9983994 (10Marostegui) p:05Triage→03Medium
[07:15:07] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9984018 (10ABran-WMF)
[07:24:07] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9984021 (10ABran-WMF)
[07:28:44] <wikibugs>	 06Data-Engineering, 10Pageviews-API: Missed pageview data over API - https://phabricator.wikimedia.org/T370108#9984026 (10Bugreporter)
[08:05:08] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9984099 (10Marostegui)
[08:06:57] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9984108 (10Marostegui)
[08:12:51] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 13Patch-For-Review, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9984120 (10ABran-WMF)
[08:26:57] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122#9984175 (10BTullis) @Marostegui - please feel free to go ahead and re-clone it. Now 'ish a relatively good time of the month to do it.
[08:30:23] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122#9984191 (10Marostegui) {P66590}
[08:36:18] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122#9984228 (10Marostegui) I've upgraded dbstore1008 to 10.6.18
[09:40:14] <wikibugs>	 06Data-Engineering, 10CirrusSearch, 06Data Products, 10MediaWiki-extensions-EventLogging, and 2 others: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543#9984458 (10phuedx) The problem appears to have been //highlighted// by [[ https://meta.wikimedia...
[10:02:07] <wikibugs>	 06Data-Engineering, 10CirrusSearch, 06Data Products, 10MediaWiki-extensions-EventLogging, and 2 others: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543#9984545 (10phuedx) >>! In T368543#9926759, @Lucas_Werkmeister_WMDE wrote: > I’m not really sure...
[10:16:57] <wikibugs>	 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: LDAP access to the analytics-privatedata-users group for Quiddity - https://phabricator.wikimedia.org/T370091#9984595 (10Clement_Goubert) a:03KStineRowe_WMF Hi,  Can you please read and sign the L3 document, as well as read the Dat...
[10:18:13] <btullis>	 !log restarted `cirrussearch-dump-s5.service` on snapshot1017 after fixing issue with aewikimedia search indices for T362529 
[10:18:16] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[10:18:17] <stashbot>	 T362529: Create a Wikimedians of United Arab Emirates User Group Wiki - https://phabricator.wikimedia.org/T362529
[11:02:11] <wikibugs>	 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9984719 (10Marostegui)
[11:02:52] <wikibugs>	 06Data-Engineering, 06DBA: dbstore1008:3317 (s7) crashed - https://phabricator.wikimedia.org/T370122#9984715 (10Marostegui) 05Open→03Resolved a:03Marostegui This is done - host recloned from db1174
[13:22:08] <claime>	 btullis: I just saw that your change to get the stream config from mw-api-int didn't work, did it hit meta.wikimedia.org directly before?
[13:25:48] <btullis>	 claime: I think I might have been barking up the wrong tree a little. The timing doesn't fit. But we think we have found it. This timing fits perfectly: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/analytics/dags/canary_events/canary_events_dag.py?ref_type=heads#L48
[13:29:10] <btullis>	 claime: This is the patch that we think should fix it: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/1054560 - Now looking to deploy. When the the turndown scheduled to happen?
[13:29:50] <claime>	 btullis: I think tonight for the "soft" turndown (depool everything)
[13:30:47] <claime>	 and yeah, I was actually trying to find what instantiated WikimediaDefaults.WIKIMEDIA_HTTP_CLIENT
[13:31:33] <btullis>	 claime: Cool, thanks. So we should have a chance to deploy this and see these bumps disappear before it gets switched off: https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?var-cluster=api_appserver&orgId=1&refresh=1m&from=now-6h&to=now
[13:31:47] <claime>	 I wonder why that repo isn't surfacing in codesearch
[13:32:33] <btullis>	 Yeah. Good point. 
[13:38:46] <wmf-insecte>	 Starting build #2 for job wikimedia-event-utilities-maven-release
[13:42:00] <wmf-insecte>	 Project wikimedia-event-utilities-maven-release build #2: 09SUCCESS in 3 min 13 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release/2/
[13:42:20] <btullis>	 claime: We'll fix the codesearch indexing for this. Probably carry out a quick review of what else of ours isn't there.
[13:42:38] <claime>	 btullis: <3
[13:51:10] <wikibugs>	 (03PS1) 10Gmodena: pom.xml: version bump eventutilities. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054564 (https://phabricator.wikimedia.org/T367949)
[13:53:07] <wikibugs>	 (03CR) 10Btullis: [C:03+1] "Looks good." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054564 (https://phabricator.wikimedia.org/T367949) (owner: 10Gmodena)
[14:06:08] <wikibugs>	 (03CR) 10DCausse: [C:03+1] pom.xml: version bump eventutilities. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054564 (https://phabricator.wikimedia.org/T367949) (owner: 10Gmodena)
[14:07:57] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9985587 (10Ottomata)...
[14:09:15] <wikibugs>	 (03CR) 10Btullis: [C:03+2] pom.xml: version bump eventutilities. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054564 (https://phabricator.wikimedia.org/T367949) (owner: 10Gmodena)
[14:11:24] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 10MW-1.43-notes (1.43.0-wmf.11; 2024-06-25): Event validation errors for mediawiki.page_change.v1 due to missing performer field on revision suppressions - https://phabricator.wikimedia.org/T367923#9985614 (10Ottomata) I sent this anno...
[14:14:52] <wikibugs>	 (03PS1) 10Btullis: Update changelog for 0.2.44 release [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054577 (https://phabricator.wikimedia.org/T367949)
[14:20:21] <wikibugs>	 (03CR) 10Btullis: [V:03+2 C:03+2] Add ae.wikimedia project to pageview allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054282 (https://phabricator.wikimedia.org/T362529) (owner: 10Btullis)
[14:25:46] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update changelog for 0.2.44 release [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054577 (https://phabricator.wikimedia.org/T367949) (owner: 10Btullis)
[14:26:49] <wikibugs>	 (03CR) 10Btullis: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054577 (https://phabricator.wikimedia.org/T367949) (owner: 10Btullis)
[14:45:11] <wikibugs>	 (03CR) 10Btullis: [C:03+2] Update changelog for 0.2.44 release [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1054577 (https://phabricator.wikimedia.org/T367949) (owner: 10Btullis)
[14:45:30] <wmf-insecte>	 Starting build #10 for job analytics-refinery-maven-release
[15:03:25] <wmf-insecte>	 Project analytics-refinery-maven-release build #10: 09SUCCESS in 17 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/10/
[15:07:23] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th): [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow - https://phabricator.wikimedia.org/T356362#9986055 (10Ottomata) @Antoine_Quhen, Can we close this task?  Link to the work done in to get canary events in a...
[15:11:11] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 13Patch-For-Review: [Refine Refactoring] Implement airflow dag to schedule RefineDataset job using EventStreamConfig settings - https://phabricator.wikimedia.org/T367134#9986090 (10Ottomata)
[15:11:18] <wikibugs>	 06Data-Engineering, 06Data Products, 10Metrics Platform Backlog: [MPIC] Analyse risk of potential performance issues with static approach to stream configuration - https://phabricator.wikimedia.org/T366627#9986092 (10Ottomata) @phuedx ^ in case you missed: > For prioritization: do you have a sense on timelin...
[15:16:10] <wmf-insecte>	 Starting build #10 for job analytics-refinery-update-jars
[15:17:45] <wikibugs>	 (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.44 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054587
[15:17:46] <wmf-insecte>	 Project analytics-refinery-update-jars build #10: 09SUCCESS in 1 min 35 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/10/
[15:18:27] <wikibugs>	 (03CR) 10Gmodena: refinery-job: add webrequest instrumentation. (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[15:20:19] <wikibugs>	 (03PS10) 10Gmodena: refinery-job: add webrequest instrumentation. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117)
[15:20:33] <wikibugs>	 (03CR) 10Gmodena: refinery-job: add webrequest instrumentation. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[15:20:56] <wikibugs>	 (03CR) 10Btullis: [C:03+2] Add refinery-source jars for v0.2.44 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054587 (owner: 10Maven-release-user)
[15:20:58] <wikibugs>	 (03CR) 10Btullis: [V:03+2 C:03+2] Add refinery-source jars for v0.2.44 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1054587 (owner: 10Maven-release-user)
[15:35:36] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Release-Engineering-Team, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#9986240 (10BTullis)
[15:39:13] <wikibugs>	 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#9986267 (10Ottomata)
[15:39:14] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th): Implement automatic sync of refinery HQL files to HDFS - https://phabricator.wikimedia.org/T365659#9986268 (10Ottomata)
[16:32:04] <wikibugs>	 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: LDAP access to the analytics-privatedata-users group for Quiddity - https://phabricator.wikimedia.org/T370091#9986638 (10Quiddity) I've read and signed the L3, and read the Responsibilities document. Thanks.
[16:32:10] <wikibugs>	 14Data-Engineering (Q4 2024 April 1st - June 30th): [Refine Refactoring] Refactor refinery code for compatibility with Airflow integration - https://phabricator.wikimedia.org/T356363#9986641 (10Ottomata) I'm not sure if this task is resolved?  I think the relevant CR is [Refactor Refine to be triggerd by Air...
[16:33:44] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products: Add user Central ID to mediawiki_history table in Hive - https://phabricator.wikimedia.org/T365648#9986646 (10VirginiaPoundstone)
[16:35:20] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 10Dumps-Generation, and 2 others: Provide attribute to indicate that user is temporary account in exported content - https://phabricator.wikimedia.org/T365693#9986649 (10VirginiaPoundstone)
[16:36:32] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products: Investigate why we consume empty partitions from webrequests - https://phabricator.wikimedia.org/T343238#9986662 (10VirginiaPoundstone)
[16:37:12] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 10Pageviews-API, and 3 others: There are anomalies in some of the mostread data on zhwiki for March 2024 - https://phabricator.wikimedia.org/T360499#9986685 (10VirginiaPoundstone)
[16:37:15] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Pipelines, 06Data Products: Windows 11 missing in analytics ? - https://phabricator.wikimedia.org/T346890#9986666 (10VirginiaPoundstone)
[16:37:31] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products: Mediarequests top articles: should use a disallow filter just like top articles - https://phabricator.wikimedia.org/T343793#9986688 (10VirginiaPoundstone)
[16:37:49] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 07Documentation: Clickstream dataset documentation should be extracted from Research page - https://phabricator.wikimedia.org/T356528#9986691 (10VirginiaPoundstone)
[16:37:59] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 06Data-Platform, 06Movement-Insights: Wikistats "Active Editors by Country" does not follow definition for active editors - https://phabricator.wikimedia.org/T360073#9986679 (10VirginiaPoundstone)
[16:38:28] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Data Pipelines, 10LPL Technical Support, and 2 others: Merge ks-Arab and ks-Deva to ks - https://phabricator.wikimedia.org/T314476#9986708 (10VirginiaPoundstone)
[16:38:30] <wikibugs>	 14Analytics, 10Data Pipelines, 06Data Products: Add cawiki to clickstream dataset - https://phabricator.wikimedia.org/T327982#9986701 (10VirginiaPoundstone)
[16:38:30] <wikibugs>	 14Analytics, 06Data-Engineering, 10Data-Engineering-Wikistats: No page views by country data for Turkey - https://phabricator.wikimedia.org/T355404#9986705 (10VirginiaPoundstone)
[16:39:14] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Structured-Data-Backlog: [Maintenance] Set up deletion jobs for Structured Data's data pipelines - https://phabricator.wikimedia.org/T347561#9986711 (10VirginiaPoundstone)
[16:40:10] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Data Pipelines, 06Data Products: A proposed modification to pageviews data protection for Wikistats - https://phabricator.wikimedia.org/T340926#9986717 (10VirginiaPoundstone)
[16:40:52] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Pipelines, 06Data Products: Provide visitor stats that can be imported into caniuse.com - https://phabricator.wikimedia.org/T359750#9986720 (10VirginiaPoundstone)
[16:41:30] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Movement-Insights, 10Movement-Metrics: Data Quality Issue: Wikitext History Job fail / rerun in Airflow - https://phabricator.wikimedia.org/T342911#9986726 (10VirginiaPoundstone)
[16:43:04] <wikibugs>	 06Data-Engineering, 06Data Products: NEW BUG REPORT - Pageviews Missing Hourly Partition - https://phabricator.wikimedia.org/T358142#9986727 (10VirginiaPoundstone) 05Open→03Resolved a:03VirginiaPoundstone
[16:43:07] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 06Data Products, 06Data-Platform, 06Movement-Insights: New Data Pipeline for New and Returning Active Editor metrics by Geo (Country & Region) and wiki - https://phabricator.wikimedia.org/T359646#9986730 (10VirginiaPoundstone)
[16:44:51] <wikibugs>	 06Data-Engineering, 10AQS2.0, 10Cassandra, 06Data Products: DELETE mechanism for Cassanda Analytics datasets - https://phabricator.wikimedia.org/T366631#9986743 (10VirginiaPoundstone)
[16:45:21] <wikibugs>	 06Data-Engineering, 10Data-Engineering-Wikistats, 10Data Pipelines, 06Data Products: Wikistats Link with Language Option - https://phabricator.wikimedia.org/T366820#9986745 (10VirginiaPoundstone)
[16:47:03] <wikibugs>	 06Data-Engineering, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9986756 (10VirginiaPoundstone)
[17:00:10] <wikibugs>	 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: LDAP access to the analytics-privatedata-users group for Quiddity - https://phabricator.wikimedia.org/T370091#9986828 (10Ottomata) Approved!
[17:28:13] <wikibugs>	 (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[17:28:25] <wikibugs>	 (03CR) 10Aleksandar Mastilovic: [V:03+2 C:03+2] refinery-job: add webrequest instrumentation. (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[17:38:56] <wikibugs>	 (03Merged) 10jenkins-bot: refinery-job: add webrequest instrumentation. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1019867 (https://phabricator.wikimedia.org/T351117) (owner: 10Gmodena)
[18:10:51] <wikibugs>	 14Data-Engineering (Sprint 9), 10Event-Platform: Decomission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186 (10Ottomata) 03NEW
[18:12:16] <wikibugs>	 14Data-Engineering (Sprint 9), 10Event-Platform, 13Patch-For-Review: Decomission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987093 (10Ottomata)
[18:13:10] <wikibugs>	 14Data-Engineering (Sprint 9), 10Event-Platform, 13Patch-For-Review: Decomission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987095 (10Ottomata) Disabling the job now to unblock SRE in {T367949}.  See CR for context.
[18:14:02] <wikibugs>	 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Decomission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987099 (10Ottomata)
[18:19:34] <wikibugs>	 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Decomission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987126 (10Ottomata)
[18:20:19] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Decommission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987128 (10Ottomata)
[18:20:25] <wikibugs>	 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Event-Platform, 13Patch-For-Review: Decommission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186#9987134 (10Ottomata)
[18:31:14] <wikibugs>	 06Data-Engineering, 06cloud-services-team, 10Data-Services, 06Privacy Engineering: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944#9987174 (10sguebo_WMF) Hi @joanna_borun, tagging you here as per @acooper’s guidance. The current task is a co...
[18:53:55] <wikibugs>	 06Data-Engineering, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: LDAP access to the analytics-privatedata-users group for Quiddity - https://phabricator.wikimedia.org/T370091#9987384 (10KStineRowe_WMF) approved
[19:11:56] <wikibugs>	 (03PS1) 10Ottomata: Bump wikimedia-event-utilities version to 1.3.6 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1054652 (https://phabricator.wikimedia.org/T367949)
[19:14:02] <wikibugs>	 (03PS2) 10Ottomata: Bump wikimedia-event-utilities version to 1.3.6 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1054652 (https://phabricator.wikimedia.org/T370199)
[19:19:24] <wikibugs>	 06Data-Engineering, 10MW-on-K8s, 13Patch-For-Review: gobblin-wmf: bump event-utilities dependency to unblock MW on K8s migration. - https://phabricator.wikimedia.org/T370199#9987526 (10Ottomata)
[19:27:55] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Bump wikimedia-event-utilities version to 1.3.6 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/1054652 (https://phabricator.wikimedia.org/T370199) (owner: 10Ottomata)
[20:06:02] <wikibugs>	 06Data-Engineering, 06DC-Ops, 10ops-eqiad, 06SRE: Degraded RAID on dumpsdata1007 - https://phabricator.wikimedia.org/T369829#9987783 (10Jclark-ctr) a:03Jclark-ctr
[20:33:32] <wikibugs>	 06Data-Engineering, 10CirrusSearch, 03Discovery-Search (Current work), 13Patch-For-Review: [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#9987882 (10EBernhardson) a:03EBernhardson
[20:34:29] <ottomata>	 !log disabled produce_canary_events systemd timer to unblock mw on k8s.  airflow should suffice now.  T370186
[20:34:32] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:34:33] <stashbot>	 T370186: Decommission produce_canary_events systemd timer - https://phabricator.wikimedia.org/T370186
[21:55:27] <wikibugs>	 06Data-Engineering, 10Data Pipelines, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review, 10Release-Engineering-Team (Seen): Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#9988287 (10brennen)
[22:14:53] <wikibugs>	 06Data-Engineering, 06Data Products, 10Pageviews-API: Missed pageview data over API - https://phabricator.wikimedia.org/T370108#9988358 (10Milimetric) weird... quick steps as I look into this.  1. check [[ https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/Barack...