[09:24:19] 06Data-Engineering, 06Product-Analytics: Allow curl commands from Airflow BashOperator - https://phabricator.wikimedia.org/T392288 (10KCVelaga_WMF) 03NEW [09:38:31] 10Quarry, 06cloud-services-team: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10754623 (10taavi) 05Open→03Resolved [09:39:05] 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997#10754627 (10taavi) [09:39:10] 10Quarry, 06cloud-services-team: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10754628 (10taavi) [09:40:38] 10Quarry: Scrape prometheus metrics from Quarry - https://phabricator.wikimedia.org/T360220#10754634 (10taavi) 05Open→03Resolved a:05brouberol→03taavi [09:42:11] 10Quarry, 10observability: Develop the monitoring of Quarry - https://phabricator.wikimedia.org/T205150#10754639 (10taavi) →14Duplicate dup:03T392138 [09:42:13] 10Quarry, 06cloud-services-team: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10754641 (10taavi) [10:25:39] (03CR) 10Filippo Giunchedi: Add Prometheus stats push (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/1136417 (https://phabricator.wikimedia.org/T389344) (owner: 10Hasan Akgün (WMDE)) [15:10:03] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [15:10:08] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [15:40:03] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [15:40:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [15:45:03] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [15:45:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:00:03] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:00:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:10:03] RESOLVED: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:10:03] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:10:33] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:10:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:15:18] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:15:18] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:30:18] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:30:18] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [16:35:18] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [16:35:18] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:25:33] RESOLVED: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:25:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:27:33] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:27:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:35:18] RESOLVED: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:35:18] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:37:33] FIRING: [3x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:37:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [17:42:33] FIRING: [4x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [17:42:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [18:35:18] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [18:35:19] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [19:08:05] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10755444 (10tchin) [19:42:33] FIRING: [2x] GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [19:42:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [20:36:43] 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, 07Epic: [Trino] Connect to Google Sheets - https://phabricator.wikimedia.org/T385748#10755625 (10IAckerman-WMF) p:05Medium→03High [21:23:13] 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Experimentation Lab: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10755739 (10Milimetric) [23:42:33] FIRING: GobblinKafkaRecordsExtractedNotEqualRecordsExpected: Gobblin job webrequest ingested an unexpected number of records for a Kafka topic partition. ... [23:42:33] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=webrequest&var-kafka_topic=webrequest_text&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected