[03:08:33] 10Quarry: Quarry not working today - https://phabricator.wikimedia.org/T375988#10186396 (10Liz) [07:04:30] 10Quarry: Quarry not working today - https://phabricator.wikimedia.org/T375988#10186544 (10rook) Looks like k8s is having trouble with garbage collection ` Warning FreeDiskSpaceFailed 118s (x159 over 13h) kubelet Failed to garbage collect required amount of images. Attempted to free 4281512755 bytes, but o... [07:29:07] 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997 (10rook) 03NEW [07:30:52] 10Quarry: Quarry not working today - https://phabricator.wikimedia.org/T375988#10186590 (10rook) Quarry is working again. Though I didn't have time to investigate what is happening so this may happen again. Opening T375997 to investigate the underlying issue. [07:32:16] 10Quarry: Quarry not working today - https://phabricator.wikimedia.org/T375988#10186592 (10rook) 05Open→03Resolved a:03rook [09:00:05] 10Quarry: Quarry shows error: This web service cannot be reached - https://phabricator.wikimedia.org/T375988#10186916 (10Aklapper) [09:02:38] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Prod-Kubernetes, 06serviceops, 10Data-Platform-SRE (2024.09.28 - 2024.10.18), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10186930 (10gmodena) [09:47:43] 06Data-Engineering: [SPIKE] Learn and document how to use Flink-CDC from MediaWiki MariaDB locally - https://phabricator.wikimedia.org/T373144#10187068 (10NoZeroDay) Thanks for this - I'll make a note to downgrade to Flink 1.18 [10:02:50] 06Data-Engineering, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10187122 (10brouberol) What @Stevemunene @Btullis and myself talked about today was the following. - The airflow-dags CI triggers the gitlab synchronizer as th... [10:10:08] 06Data-Engineering, 10Data-Platform-SRE (2024.09.28 - 2024.10.18): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10187166 (10brouberol) From @BTullis slack message, it turns out we can't mount a PV in multiple locations: > Following on from our conversation in the sync ab... [11:19:55] !log rolling out conda-analytics v0.0.36 to production for T372417 [11:19:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:19:59] T372417: Investigate whether we should switch from miniconda to miniforge - https://phabricator.wikimedia.org/T372417 [11:57:31] 06Data-Engineering, 10Event-Platform: Update event-producing tools to overwrite `meta.dt` - https://phabricator.wikimedia.org/T376026 (10JAllemandou) 03NEW [14:22:42] 06Data-Engineering, 03Discovery-Search (Current work): Datahub - ingest Hive discovery database - https://phabricator.wikimedia.org/T374118#10187843 (10EBernhardson) While the ingestion now claims to work succesfully, I'm unable to find our tables in the datahub UI. It's possible I simply don't know how to use... [15:09:07] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are ... - https://phabricator.wikimedia.org/T367405#10187972 [15:09:45] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work), 10Release-Engineering-Team (Radar): Validate CI integration so that Ci can release Maven artifacts on user's dem... - https://phabricator.wikimedia.org/T367403#10187981 [15:09:52] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work), 10Release-Engineering-Team (Radar): Validate CI integration so that Ci can release Maven artifacts on user's dem... - https://phabricator.wikimedia.org/T367403#10187985 [15:10:02] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Java-Scala-Standardization, 03Discovery-Search (Current work): Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all de... - https://phabricator.wikimedia.org/T367405#10187974 [16:42:41] 06Data-Engineering, 10Cassandra, 10Data Pipelines: Encrypt Spark-Cassandra connection - https://phabricator.wikimedia.org/T310820#10188526 (10Eevans) >>! In T310820#10185030, @BTullis wrote: > I think that this is complete now, isn't it? Encryption to cassandra is enabled by default. If that happened, it ha... [17:11:34] 06Data-Engineering, 06Data Products, 10GlobalBlocking, 06Stewards-and-global-tools, and 4 others: Allow autoblocks on IP addresses in the globalblocks table for IP addresses which are already globally blocked - https://phabricator.wikimedia.org/T376052 (10Dreamy_Jazz) 03NEW [17:12:09] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, and 5 others: Allow autoblocks on IP addresses in the globalblocks table for IP addresses which are already globally blocked - https://phabricator.wikimedia.org/T376052#10188641 (10Dreamy_Jazz) [17:12:55] 06Data-Engineering, 06Data Products, 06DBA, 10GlobalBlocking, and 5 others: Allow autoblocks on IP addresses in the globalblocks table for IP addresses which are already globally blocked - https://phabricator.wikimedia.org/T376052#10188640 (10Dreamy_Jazz) [17:18:34] 06Data-Engineering, 10Cassandra, 10Data Pipelines: Encrypt Spark-Cassandra connection - https://phabricator.wikimedia.org/T310820#10188657 (10BTullis) Ah, right. Thanks @eevans for the clarification. [17:21:44] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Deprecate use of desktop- and mobilewebuiactions in Event Platform - https://phabricator.wikimedia.org/T368678#10188681 (10KSarabia-WMF) [18:12:23] 06Data-Engineering, 06DBA, 10GlobalBlocking, 07Schema-change-in-production, 10Trust and Safety Product Sprint (Sprint Beatboxing (Sept 16-27)): Apply a default to gbw_address in the global_block_whitelist table - https://phabricator.wikimedia.org/T375184#10189033 (10Ladsgroup) a:03Ladsgroup Running on... [18:25:53] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065 (10KSarabia-WMF) 03NEW [18:25:55] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10189108 (10KSarabia-WMF) [18:25:58] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10189109 (10KSarabia-WMF) FYI - We broke th... [18:31:43] 06Data-Engineering, 10Event-Platform, 10Web Team Essential Work 2024 (Migrate to new Event Platform), 10Web-Team-Backlog (FY2024-25 Q2 Sprint 1): Delete redundant mobile- and desktopwebuiactions event in WikimediaEvents - https://phabricator.wikimedia.org/T376065#10189135 (10KSarabia-WMF) p:05Triage→03M... [18:34:49] 06Data-Engineering, 06DBA, 10GlobalBlocking, 07Schema-change-in-production, 10Trust and Safety Product Sprint (Sprint Beatboxing (Sept 16-27)): Apply a default to gbw_address in the global_block_whitelist table - https://phabricator.wikimedia.org/T375184#10189138 (10Ladsgroup) 05Open→03Resolved D... [19:47:25] 06Data-Engineering, 10Data Pipelines, 06Data Products, 10Dumps-Generation, and 2 others: MediaWiki Dumps XML - Provide attribute to indicate that user is temporary account in exported content - https://phabricator.wikimedia.org/T365693#10189430 (10kostajh) If nothing is done here, then it means that the ex...