[00:04:10] <wikibugs>	 10Quarry: Search or filter queries by title or summary - https://phabricator.wikimedia.org/T90509 (10samuelguebo) Hey there. Is this still needed? Also, could someone clarify whether the pull requests should be made to this [[ https://github.com/toolforge/quarry | Github repository]] or to Gerrit? Thanks
[01:25:18] <wikibugs>	 10Quarry: Search or filter queries by title or summary - https://phabricator.wikimedia.org/T90509 (10rook) Hi @samuelguebo yeah let's see if we can get this in! And yes, github is the right place to put in patches. I'll test it and hopefully get it in tomorrow. Thanks for the patch!
[01:25:27] <wikibugs>	 10Quarry: Search or filter queries by title or summary - https://phabricator.wikimedia.org/T90509 (10rook) a:03samuelguebo
[11:08:15] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Rebuild spark2 for Debian Buster - https://phabricator.wikimedia.org/T229347 (10Aklapper)
[11:23:33] <wikibugs>	 10Data-Engineering, 10Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10JAllemandou) Indeed we're waiting for Spark3 -  We'll be able to make this happen soon !
[11:23:49] <wikibugs>	 10Data-Engineering, 10Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10JAllemandou) a:05JAllemandou→03None
[11:24:04] <wikibugs>	 10Data-Engineering, 10Patch-Needs-Improvement: HiveExtensions.convertToSchema does not properly convert arrays of structs - https://phabricator.wikimedia.org/T259924 (10JAllemandou) a:03JAllemandou
[12:00:11] <jbond>	 hi all who would be the right person to speak with about an-airflow
[12:01:32] <jbond>	 nfraison: stashbot: btullis: perhaps? ^^^
[12:01:43] * jbond not stashbot lol
[12:02:02] <btullis>	 I'm around. You can talk to me about an-airflow
[12:02:33] <jbond>	 hi ben so i was looking at the systemd status as its showing on nagios
[12:02:48] <jbond>	 wmf_auto_restart_airflow-kerberos@search.service is having an error which seems unrelated
[12:03:21] <jbond>	 so i ran th command manually which caused airflow-kerberos@search.service to restart
[12:03:25] <jbond>	 however it failed to start
[12:03:39] <jbond>	 althugh it looks started now :/
[12:03:46] <btullis>	 Ah right, an-airflow1005 ? This one is currently being setup by the search_platform team and bking.
[12:04:19] <jbond>	 ahh ok ill ignore it for now then
[12:04:22] <jbond>	 thanks
[12:04:24] <btullis>	 Under this ticket: T327970
[12:04:24] <stashbot>	 T327970: Create airflow v2 instance and supporting repos for search platform - https://phabricator.wikimedia.org/T327970
[12:04:34] <jbond>	 thanks ill leave a ot there 
[12:05:33] <btullis>	 Cool, yeah last I heard they tried a bullseye update, but that didn't work for other reasons, then it was reimaged back to buster and it's still not ready, but I think it was supposed to be indowntime until we can get the airflow 2.5 upgrade ready for them.
[12:07:51] <jbond>	 ack its possible that the downtime has expired
[12:11:05] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): [Tracking] Migrate Search Airflow jobs to Airflow 2 and use shared supporting code from the data engineering Airflow - https://phabricator.wikimedia.org/T318414 (10jbond)
[12:11:13] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Create airflow v2 instance and supporting repos for search platform - https://phabricator.wikimedia.org/T327970 (10jbond) 05Open→03In progress Hi all as this machine is still been set up i have added 24 hours down time to k...
[12:22:42] <wikibugs>	 10Data-Engineering, 10conftool: an-launcher1002: failed services - https://phabricator.wikimedia.org/T330652 (10jbond)
[12:23:16] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_eventlogging_analytics.service,monitor_refine_eventlogging_legacy.service John Bond T330652 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[12:38:54] <wikibugs>	 10Data-Engineering, 10Machine-Learning-Team, 10Observability-Logging: centrallog1002: failed to start kafkatee - https://phabricator.wikimedia.org/T330654 (10jbond)
[12:41:32] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[12:48:56] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[12:54:32] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[13:02:00] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[13:03:35] <wikibugs>	 10Quarry: Quarry cannot store results with identical column names - https://phabricator.wikimedia.org/T170464 (10github-toolforge-bot) supertassu opened https://github.com/toolforge/quarry/pull/16
[13:05:53] <wikibugs>	 10Quarry: Query in quarry seems not to finish but same query runs fine in PAWS - https://phabricator.wikimedia.org/T299292 (10taavi)
[13:05:55] <wikibugs>	 10Quarry, 10Patch-For-Review: Quarry cannot store results with identical column names - https://phabricator.wikimedia.org/T170464 (10taavi) a:03taavi
[13:06:24] <wikibugs>	 10Quarry, 10Patch-For-Review: Quarry cannot store results with identical column names - https://phabricator.wikimedia.org/T170464 (10taavi)
[13:14:58] <icinga-wm>	 RECOVERY - Check systemd state on an-worker1132 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[14:21:03] <wikibugs>	 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Epic: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10Ottomata)
[14:23:50] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata)
[14:41:09] <wikibugs>	 10Quarry: Search or filter queries by title or summary - https://phabricator.wikimedia.org/T90509 (10rook) 05Open→03Resolved
[14:51:27] <wikibugs>	 10Data-Engineering, 10Patch-For-Review: Investigate trend of gradual hive server heap exhaustion - https://phabricator.wikimedia.org/T303168 (10nfraison) coord1002 hive metastore and hiveserver2 restarted. waiting for https://gerrit.wikimedia.org/r/c/operations/dns/+/892460 to be merged and enabled before to d...
[14:52:58] <nfraison>	 !log restarted hive-metastore and hiveserver2 on an-coord1002 (standby hive server)
[14:52:59] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:09:59] <wikibugs>	 10Data-Engineering, 10Machine-Learning-Team, 10Observability-Logging: centrallog1002: failed to start kafkatee - https://phabricator.wikimedia.org/T330654 (10fgiunchedi) Thank you for the heads up @jbond, cc @andrea.denisse
[15:47:20] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service,systemd-timedated.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[15:53:09] <wikibugs>	 (03PS2) 10Neil P. Quinn-WMF: Update Wikipedia Preview ETL to extract instrumentation version [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/891866 (https://phabricator.wikimedia.org/T328703)
[15:55:11] <wikibugs>	 (03CR) 10Neil P. Quinn-WMF: [V: 03+2 C: 03+2] "Thanks, Stephane!" [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/891866 (https://phabricator.wikimedia.org/T328703) (owner: 10Neil P. Quinn-WMF)
[15:58:10] <icinga-wm>	 PROBLEM - Check systemd state on an-worker1132 is CRITICAL: CRITICAL - degraded: The following units failed: export_smart_data_dump.service,systemd-timedated.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[16:09:19] <wikibugs>	 10Analytics-Radar: stat1005: failing systemd job - https://phabricator.wikimedia.org/T330671 (10jbond) p:05Triage→03Medium
[16:21:55] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate mediawiki_revision_recommendation_create.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T330447 (10TJones)
[16:22:17] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate transfer_to_es.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329881 (10TJones)
[16:22:21] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate search_satisfaction.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329880 (10TJones)
[16:22:29] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329879 (10TJones)
[16:22:36] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate query_clicks.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329878 (10TJones)
[16:22:42] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate popularity_score.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329877 (10TJones)
[16:22:48] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate ores_predictions.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329876 (10TJones)
[16:22:58] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate incoming_links.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329875 (10TJones)
[16:23:57] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate glent_weekly.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329872 (10TJones)
[16:24:00] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate export_queries_to_relforge.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329871 (10TJones)
[16:26:21] <wikibugs>	 10Data-Engineering, 10Data-Persistence, 10Infrastructure-Foundations, 10Machine-Learning-Team, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10MPhamWMF)
[16:40:34] <icinga-wm>	 RECOVERY - Check systemd state on an-worker1132 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[17:26:17] <mforns>	 joal: this is the mr: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/243
[17:27:14] <joal>	 ack mforns - will read
[17:59:54] <wikibugs>	 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 74 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Jdlrobson)
[19:23:26] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10gmodena)
[19:25:43] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10Ottomata)
[19:31:11] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10gmodena)
[19:55:10] <wikibugs>	 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 74 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Krinkle)
[21:00:13] <wikibugs>	 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09): [Flink Operation] How to handle app upgrades - https://phabricator.wikimedia.org/T328569 (10gmodena) Give how the flink k8s operator handles restarts, this task might end up being a subtask of https://phabricator.wikimedia.org/T328563.   T...
[21:07:48] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10Ottomata)
[21:14:53] <wikibugs>	 10Quarry, 10cloud-services-team (FY2022/2023-Q3): Consider moving Quarry to be an installation of Redash - https://phabricator.wikimedia.org/T169452 (10rook) Superset seems to have caching https://superset.apache.org/docs/installation/cache/  Though I'm having trouble finding documentation on it in official lo...
[21:15:56] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10gmodena)
[21:17:21] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream: Flink EventStreamCatalog should not prevent creation of VIEWs - https://phabricator.wikimedia.org/T330703 (10Ottomata)
[21:24:52] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10SRE-swift-storage: Storage request: swift s3 bucket for mediawiki-page-content-change-enrichment checkpointing - https://phabricator.wikimedia.org/T330693 (10Ottomata)
[21:26:59] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate import_ttl.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329874 (10EBernhardson) a:03pfischer
[23:21:24] <wikibugs>	 10Data-Engineering, 10Machine-Learning-Team, 10Observability-Logging: centrallog1002: failed to start kafkatee - https://phabricator.wikimedia.org/T330654 (10andrea.denisse) 05Open→03In progress a:03andrea.denisse
[23:41:37] <wikibugs>	 10Quarry, 10Patch-For-Review: Make available more options for number of shown rows of resultset (Quarry) - https://phabricator.wikimedia.org/T126540 (10samuelguebo) >>! In T126540#4140551, @gerritbot wrote: > Change 328602 had a related patch set uploaded (by Zhuyifei1999; owner: XXN): > [analytics/quarry/web@...