[01:11:54] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 3.569% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:20:55] (SystemdUnitFailed) firing: (10) user-runtime-dir@21734.service Failed on stat1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:36:15] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [01:39:09] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_event.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:39:29] (SystemdUnitFailed) firing: (11) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:11:30] (03PS2) 10Snwachukwu: [WIP] Add Reportupdater Browser All Sites Queries. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/995740 (https://phabricator.wikimedia.org/T354552) [05:11:54] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 3.489% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:36:31] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [05:40:56] (SystemdUnitFailed) firing: (11) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:22:59] * brouberol waves good morning! [09:11:54] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 3.452% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:36:31] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [09:40:56] (SystemdUnitFailed) firing: (11) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:41:05] Morning all. [09:59:17] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) [10:00:13] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) [10:02:13] (03PS3) 10Aqu: Add session_lenght Iceberg table [analytics/refinery] - 10https://gerrit.wikimedia.org/r/994697 (https://phabricator.wikimedia.org/T352672) [10:03:42] (03CR) 10Aqu: "Thanks for the review @Joal." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/994697 (https://phabricator.wikimedia.org/T352672) (owner: 10Aqu) [10:37:52] (03CR) 10Gmodena: [C: 03+2] IcebergWriter: don't create missing tables if absent [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/995101 (https://phabricator.wikimedia.org/T356401) (owner: 10Gmodena) [10:46:59] (03CR) 10Joal: [C: 03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/994697 (https://phabricator.wikimedia.org/T352672) (owner: 10Aqu) [10:48:40] (03Merged) 10jenkins-bot: IcebergWriter: don't create missing tables if absent [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/995101 (https://phabricator.wikimedia.org/T356401) (owner: 10Gmodena) [10:53:39] 10Data-Engineering, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Ensure necessary firewall rules are open between the DSE worker nodes and external services - https://phabricator.wikimedia.org/T356623 (10brouberol) [10:58:13] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10brouberol) [11:04:52] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10Gehel) p:05Triage→03High [11:05:32] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) a:03BTullis [11:14:23] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) The key part of the error message above is the following: ` "Unknown SEQUENCE: 'ab_user_id_seq'" ` I have seen this message before when performing the upgrade, since... [11:18:52] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) I have manually created the sequence for the `ab_user_id` column on both the staging and production superset databases. ` MariaDB [(none)]> use superset_staging; Rea... [11:30:31] 10Data-Engineering, 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Ensure necessary firewall rules are open between the DSE worker nodes and external services - https://phabricator.wikimedia.org/T356623 (10CodeReviewBot) brouberol opened https://gitlab.wikimedia.org/repos/data-engineering... [11:34:25] 10Data-Engineering (Sprint 8): [NEEDS GROOMING][Data quality] Create database and tables for DQ backend - https://phabricator.wikimedia.org/T356628 (10gmodena) [11:34:36] 10Data-Engineering (Sprint 8): [NEEDS GROOMING][Data quality] Create database and tables for DQ backend - https://phabricator.wikimedia.org/T356628 (10gmodena) [11:49:04] 10Analytics-Clusters, 10Product-Analytics: Configure superset cache - https://phabricator.wikimedia.org/T268784 (10awight) We're curious to know whether caching can be turned on after the superset 3 upgrade? Having trouble finding the newest task about this... [12:31:22] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) I believe that it comes about because we are using SQLAlchemy (version 1.4 https://docs.sqlalchemy.org/en/14/dialects/mysql.html) and the MySQL dialect, but our Supe... [12:32:08] 10Data-Engineering (Sprint 8), 10Discovery-Search, 10Image-Suggestions: Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15 - https://phabricator.wikimedia.org/T356030 (10Gehel) p:05Triage→03High [12:32:58] 10Data-Engineering (Sprint 8), 10Image-Suggestions, 10Discovery-Search (Current work): Search dag image_suggestions_weekly failed waiting for analytics_platform_eng.image_suggestions_search_index_delta/snapshot=2024-01-15 - https://phabricator.wikimedia.org/T356030 (10Gehel) [12:35:30] !log deploying conda-analytics version 0.0.28 to hadoop-test [12:35:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:37:51] !log roll-restarting druid analtyics workers for T356382 [12:37:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:39:36] 10Data-Engineering, 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Ensure necessary firewall rules are open between the DSE worker nodes and external services - https://phabricator.wikimedia.org/T356623 (10CodeReviewBot) brouberol merged https://gitlab.wikimedia.org/repos/data-engineering... [13:11:55] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.665% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:36:31] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [13:40:56] (SystemdUnitFailed) firing: (11) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:50:36] !log increasing pod & container limits in the dse-k8s-eqiad superset/superset-next namespaces - T352166 [13:50:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:50:39] T352166: Create a helm chart for Superset - https://phabricator.wikimedia.org/T352166 [13:57:43] 10Data-Engineering (Sprint 8): [Data quality] Create database and tables for DQ backend - https://phabricator.wikimedia.org/T356628 (10gmodena) [14:07:29] !log deploying conda-analytics version 0.0.28 to hadoop-all for T345482 [14:07:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:07:32] T345482: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 [14:13:43] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Bring stat1010 into service with GPU from stat1005 - https://phabricator.wikimedia.org/T336040 (10BTullis) Added the kerberos principals and keytabs for stat1010. ` analytics-privatedata/stat1010.eqiad.wmnet@WIKIMEDIA analytics-product/stat1010.eqiad.wmnet@WIKIMEDIA... [14:14:27] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Bring stat1011 into service - https://phabricator.wikimedia.org/T354526 (10BTullis) Added the kerberos principals and keytabs for stat1011. ` analytics-privatedata/stat1011.eqiad.wmnet@WIKIMEDIA analytics-product/stat1011.eqiad.wmnet@WIKIMEDIA analytics-search/stat1... [14:18:57] 10Data-Platform-SRE: Check home/HDFS leftovers of mhoutti - https://phabricator.wikimedia.org/T356641 (10MoritzMuehlenhoff) [14:19:29] (SystemdUnitFailed) firing: (14) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:38:19] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10Ottomata) [14:56:12] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Wmfdata should connect to Presto using the analytics-presto CNAME - https://phabricator.wikimedia.org/T345482 (10BTullis) I have pushed out the new version of conda-analytics containing the new wmfdata-python, plus I have annouced the update to... [14:57:28] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Migrate cloudelastic from public to private IPs - https://phabricator.wikimedia.org/T355617 (10bking) >>! In T355617#9493745, @cmooney wrote: > @bking I see that cloudelastic1010 seems to be happy on it's new IP/hostname? Glad that it seems to... [15:06:37] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Create a helm chart for Superset - https://phabricator.wikimedia.org/T352166 (10brouberol) We're at a point where the superset container runs in Kubernetes and responds to health checks: ` brouberol@deploy2002:~/charts/superset$ kubectl get pod... [15:14:32] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Ganeti, 10Infrastructure-Foundations, 10Patch-For-Review: Suggest install-console tool in sre.makevm cookbook failure message - https://phabricator.wikimedia.org/T345778 (10bking) [15:34:41] 10Data-Engineering, 10Data-Platform-SRE (2024.01.22 - 2024.02.11): Make configuration secrets available to helmfile - https://phabricator.wikimedia.org/T356480 (10brouberol) 05Open→03Resolved Fixed by `6badd4c1` in `/srv/private` on `puppetmaster` [15:34:47] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10brouberol) [15:36:09] 10Data-Engineering, 10Data-Platform-SRE, 10Epic, 10Patch-For-Review: Migrate the Analytics Superset instances to our DSE Kubernetes cluster - https://phabricator.wikimedia.org/T347710 (10brouberol) [15:44:11] 10Data-Engineering, 10Tech-Docs-Team, 10Goal: Redesign Data Platform docs on Wikitech - https://phabricator.wikimedia.org/T350911 (10TBurmeister) Good point! I added a [[ https://wikitech.wikimedia.org/wiki/Category:Contribution_data | parent category for Contribution data ]] and nested Edits data under that. [15:48:39] 10Data-Platform-SRE, 10Discovery-Search (Current work): Rebuild and deploy textify plugin - https://phabricator.wikimedia.org/T356651 (10TJones) [15:49:16] 10Data-Platform-SRE, 10Discovery-Search (Current work): Rebuild and deploy textify plugin - https://phabricator.wikimedia.org/T356651 (10TJones) [15:51:38] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Patch-For-Review: Migrate Search Platform-owned hosts to Puppet 7 - https://phabricator.wikimedia.org/T354959 (10MoritzMuehlenhoff) [15:52:05] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [15:57:22] 10Data-Platform-SRE, 10Discovery-Search, 10Patch-For-Review: Track and clean up object storage used by rdf-streaming-updater - https://phabricator.wikimedia.org/T348685 (10bking) I created a design doc for Flink object storage cleanup [[ https://docs.google.com/document/d/15zSqwu5h5Z3OHn162MYGTNtjX744hqXa0tc... [16:15:14] (03PS1) 10Gmodena: data-quality: rename source table column [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/997485 (https://phabricator.wikimedia.org/T356628) [16:15:42] 10Data-Platform-SRE (2024.01.22 - 2024.02.11), 10Discovery-Search (Current work): Investigate connection timeouts between Search Update Pipeline and MediaWiki APIs - https://phabricator.wikimedia.org/T354289 (10bking) 05Open→03Resolved a:03bking Per today's stand-up, we believe this issue has been fixed... [16:25:39] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T356313 (10Gehel) p:05Triage→03High [16:25:55] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T356313 (10Gehel) Related to T348685 [16:59:29] (SystemdUnitFailed) firing: (15) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:11:55] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.5% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:36:31] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [17:58:46] 10Data-Engineering (Sprint 8): [Data Quality] Implement basic data quality metrics for MW history - https://phabricator.wikimedia.org/T354692 (10Ahoelzl) @JAllemandou mentioned existing MW checks that should be migrated. [18:04:24] 10Data-Engineering (Sprint 8): [Data Quality] Implement basic data quality metrics for MW history - https://phabricator.wikimedia.org/T354692 (10JAllemandou) Indeed! here is the code: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/j... [19:09:04] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) firing: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [19:09:05] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [19:24:14] 10Analytics, 10AQS2.0, 10Tech-Docs-Team, 10Data Products (Epics Timeline), and 3 others: AQS 2.0 documentation - https://phabricator.wikimedia.org/T288664 (10WDoranWMF) [19:53:48] (03PS1) 10Gmodena: hql: add data quality DDL. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/997496 (https://phabricator.wikimedia.org/T356628) [20:01:57] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10gmodena) a:03gmodena [20:04:59] 10Data-Engineering (Sprint 8): [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition - https://phabricator.wikimedia.org/T354694 (10gmodena) [20:05:05] 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: [Event Platform] Declare webrequest as an Event Platform stream - https://phabricator.wikimedia.org/T314956 (10gmodena) [20:07:23] 10Data-Platform-SRE (2024.01.22 - 2024.02.11): A new user is unable to register to superset - https://phabricator.wikimedia.org/T356619 (10BTullis) 05Open→03Resolved I've received confirmation that the fix has worked. [20:09:03] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) resolved: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [20:09:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [20:25:27] 10Data-Engineering, 10Anti-Harassment, 10Data-Persistence, 10Temporary accounts, and 2 others: Adding user_is_temp to the user table - https://phabricator.wikimedia.org/T333223 (10Mayakp.wiki) @Ladsgroup Is this added to the `user` tables in All the wikis? Do you know exactly when this was completed? Is Ja... [20:40:37] random idea: Some table that stores the meta column from all event streams, to give a "global" view on event metadata. Use cases might be find all streams that had an event for request-id 12345 in a given time range [20:40:52] or for a particular meta.uri [20:49:02] 10Data-Platform-SRE: Check home/HDFS leftovers of mhoutti - https://phabricator.wikimedia.org/T356641 (10Isaac) a:03Isaac I'll aim to do this next week and give a heads up then when can be fully removed. [20:59:30] (SystemdUnitFailed) firing: (15) monitor_refine_event.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:11:55] (DiskSpace) firing: Disk space an-test-worker1001:9100:/ 1.431% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:27:18] 10Data-Engineering, 10Wmfdata-Python: wmfdata.__version__ doesn't exist in wmfdata-python - https://phabricator.wikimedia.org/T356708 (10diego) [21:36:31] (HdfsCapacityRemainingPercent) firing: Alarmingly low free space on the analytics-hadoop HDFS cluster. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Capacity_Remaining - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=106&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsCapacityRemainingPercent [21:59:15] 10Data-Engineering, 10Wmfdata-Python: wmfdata.__version__ doesn't exist in wmfdata-python - https://phabricator.wikimedia.org/T356708 (10nshahquinn-wmf) Ah, good point! Thank you for the report 😁 [23:51:37] 10Data-Engineering (Sprint 8): [Refine Refactoring] Refactor refinery code for compatibility with Airflow integration - https://phabricator.wikimedia.org/T356363 (10Ahoelzl) [23:51:39] 10Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505 (10Ahoelzl) [23:52:08] 10Data-Engineering (Sprint 8): [Dataset Config Store] - Define config API for navigationtiming and implement local development instance - https://phabricator.wikimedia.org/T355542 (10Ahoelzl) [23:52:10] 10Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505 (10Ahoelzl) [23:52:55] 10Data-Engineering (Sprint 8): [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow - https://phabricator.wikimedia.org/T356362 (10Ahoelzl) [23:52:57] 10Data-Engineering, 10Data Pipelines: Refine jobs should be scheduled by Airflow - https://phabricator.wikimedia.org/T307505 (10Ahoelzl)