[00:17:02] (03CR) 10Gergő Tisza: "Sorry for being slow to respond, I was away for a while." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/807565 (https://phabricator.wikimedia.org/T308017) (owner: 10Ottomata) [01:42:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:47:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:50:30] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:00:59] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:22:48] 10Data-Engineering-Planning, 10API Platform (API Platform Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0: Create repository for shared functions - https://phabricator.wikimedia.org/T311541 (10BPirkle) 05Open→03Resolved a:03BPirkle Done via https://gitlab.wikimedia.org/frankie/aqsass... [02:22:53] 10Analytics, 10API Platform (API Platform Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10BPirkle) [02:23:50] 10Data-Engineering-Planning, 10API Platform: Establish testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T311190 (10BPirkle) [06:01:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:06:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [09:44:06] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10gmodena) Thanks for this write up @Ottomata! +1 for leveraging on the decorator pattern to hide implementation details. IMHO Option A is interestin... [11:30:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:35:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:41:34] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10tchin) I definitely feel like the biggest issue here is how we'd map from python types to pyflink `DataTypes`. Would a python `int` turn into a `DataType... [11:51:32] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10tchin) To be fair, the actual idea is easy enough to implement for simple mappings `lang=python def python_to_flink_datatype(val: type) -> DataType:... [12:13:24] FYI, I'm switching dse-k8s-etcd1003 is DRBD for ~ an hour to drain the Ganeti server for a reimage, latencies will go up a bit [12:17:12] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10gmodena) Mapping python to SQL will be tricky, since as you point out there is no 1:1 relationship (floating-point and decimal will be funky too). The db... [12:44:20] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10gmodena) >>! In T320968#8345359, @tchin wrote: > To be fair, the actual idea is easy enough to implement for simple mappings > `lang=python > def python_... [12:45:04] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10Ottomata) > how we'd map from python types to pyflink DataTypes. Would a python int turn into a DataTypes.INT or perhaps a DataTypes.BIGINT This is just... [12:47:52] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Easy Flink Python UDF + SQL enrichment - https://phabricator.wikimedia.org/T320968 (10Ottomata) > when considering my example of get_image, given my lack of python knowledge, I have no idea how this translates into a return annotation nor... [12:52:31] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for flink - https://phabricator.wikimedia.org/T321682 (10BTullis) [12:52:48] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for flink - https://phabricator.wikimedia.org/T321682 (10BTullis) p:05Triage→03High [12:56:24] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 03): [SPIKE] Deploy event driven stateless Flink service to DSE cluster - https://phabricator.wikimedia.org/T320812 (10gmodena) Depends on https://phabricator.wikimedia.org/T321682. [13:00:32] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for flink - https://phabricator.wikimedia.org/T321682 (10Ottomata) Flink is an implementation detail so maybe namespace: `stream_enrichment_poc` or something?... [13:06:52] 10Data-Engineering-Planning, 10Data Pipelines, 10Privacy Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10EChetty) [13:27:37] Hi mforns - I'd like to spend some time with you on the questions/issues I have with refine-sanitize-iceberg - Will you have some time today? [13:28:38] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for flink - https://phabricator.wikimedia.org/T321682 (10BTullis) >>! In T321682#8345590, @Ottomata wrote: > Flink is an implementation detail so maybe namesp... [13:55:19] heya joal, of course, let me know when! [13:56:07] When you wish mforns :) [13:56:14] now! [13:56:16] :] [13:59:08] joal? [14:00:03] YES! [14:00:08] batcave! [14:00:11] sorry mforns [14:00:19] :-) [14:03:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [14:08:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [14:10:53] 10Data-Engineering-Planning, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: Obtain security review of uniqueDevices - https://phabricator.wikimedia.org/T320976 (10VirginiaPoundstone) [14:17:46] joal, you still got 5 more mins? [14:18:15] I do mforns! Back in da cave [14:18:21] ok! [14:52:38] 10Data-Engineering, 10API Platform (API Platform Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Unique Devices service - https://phabricator.wikimedia.org/T288298 (10VirginiaPoundstone) [15:16:54] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for flink - https://phabricator.wikimedia.org/T321682 (10BTullis) For now, what about a simple approach for now of? * user: `stream_enrichment` * namespace: `stream_e... [15:32:29] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 03): Prototype Flink job for content Dumps - https://phabricator.wikimedia.org/T320966 (10Milimetric) Got the basics set up in the Flink SQL client. An example of upserting a row: (on stat1004.eqiad.wmnet) ### Proxies ` export PATH=$PATH:/ho... [15:32:46] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for the stream_enrichment PoC project - https://phabricator.wikimedia.org/T321682 (10BTullis) [15:32:49] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Shared-Data-Infrastructure (Sprint 03): Create kubernetes namespace and user for the stream_enrichment PoC project - https://phabricator.wikimedia.org/T321682 (10BTullis) [15:46:37] 10Data-Engineering: Bot Detection - https://phabricator.wikimedia.org/T321707 (10Milimetric) [15:48:59] 10Data-Engineering: Bot Detection - https://phabricator.wikimedia.org/T321707 (10Milimetric) [15:49:02] 10Data-Engineering, 10Research-Backlog: [Open question] Improve bot identification at scale - https://phabricator.wikimedia.org/T138207 (10Milimetric) [15:51:31] 10Analytics-Jupyter, 10Data-Engineering, 10Product-Analytics, 10Data Pipelines (Sprint 03), 10Patch-For-Review: Add support for jupyterhub on conda-analytics - https://phabricator.wikimedia.org/T321088 (10xcollazo) `conda-analytics` MR has been merged: https://gitlab.wikimedia.org/repos/data-engineering/... [15:52:42] 10Data-Engineering: Bot Detection - https://phabricator.wikimedia.org/T321707 (10Aklapper) @Milimetric: See #Pageviews-anomaly and especially T263908. What is the "stats team" exactly? [16:36:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4039 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4039%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:38:47] aqu_: ping? [16:39:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:41:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4039 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4039%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:44:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:47:43] 10Data-Engineering: Bot Detection - https://phabricator.wikimedia.org/T321707 (10Milimetric) >>! In T321707#8346430, @Aklapper wrote: > @Milimetric: See #Pageviews-anomaly and especially T263908. What is the "stats team" exactly? Thx, I didn't know about that tag. Yeah, these are all related, so we're keeping... [18:46:32] 10Data-Engineering-Planning, 10API Platform: Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10VirginiaPoundstone) [18:46:51] 10Data-Engineering-Planning, 10API Platform (Sprint 00): Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10VirginiaPoundstone) [18:49:36] 10Data-Engineering-Planning, 10API Platform (Sprint 00): Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10EChukwukere-WMF) @VirginiaPoundstone I created this https://phabricator.wikimedia.org/T321726 :) I guess we can close it for redundancy [18:51:14] 10Data-Engineering-Planning, 10API Platform (Sprint 00): Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10VirginiaPoundstone) @Emeka-okechukwu I will close the one I made :) [18:51:16] 10Data-Engineering-Planning, 10API Platform (Sprint 00): Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10VirginiaPoundstone) 05Open→03Resolved [18:51:18] 10Data-Engineering-Planning, 10API Platform: Establish testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T311190 (10VirginiaPoundstone) [18:53:13] 10Data-Engineering-Planning, 10API Platform (Sprint 00): Review testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T321727 (10EChukwukere-WMF) @VirginiaPoundstone you tagged the wrong "EMEKA" ... LOL [19:00:24] 10Data-Engineering, 10API Platform (API Platform Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Unique Devices service - https://phabricator.wikimedia.org/T288298 (10VirginiaPoundstone) [19:02:27] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: Editors code refactoring - https://phabricator.wikimedia.org/T321730 (10VirginiaPoundstone) [19:15:03] 10Data-Engineering-Planning, 10Wikidata, 10Wikidata Analytics, 10Data Pipelines (Sprint 03): Some reliability metrics missing since June 20th '22 - https://phabricator.wikimedia.org/T314131 (10mforns) Hi @Michael! Yes, we will back-fill as much as we can. I have to talk to the team tomorrow to see how we w... [19:21:28] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews: Implement Unit Tests - https://phabricator.wikimedia.org/T299735 (10VirginiaPoundstone) [19:36:50] 10Data-Engineering, 10API Platform (API Platform Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: Obtain a security review of AQS 2.0 - https://phabricator.wikimedia.org/T288663 (10VirginiaPoundstone) [19:56:45] 10Analytics, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0 documentation - https://phabricator.wikimedia.org/T288664 (10VirginiaPoundstone) [20:15:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:20:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:36:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:41:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:46:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:51:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:10:04] 10Data-Engineering-Planning, 10Data Pipelines (Sprint 03): [airflow] Normalize the use of timeouts in Airflow DAGs - https://phabricator.wikimedia.org/T317549 (10mforns) @xcollazo This would be for the analytics and the analytics_test instances. Although, if other teams are following our developer guide docs,... [21:43:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:48:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:22:25] 10Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Patch-For-Review, 10Readers-Web-Backlog (Needs Prioritization (Tech)): Deprecate/delete the mw.eventLog.Schema class - https://phabricator.wikimedia.org/T305491 (10Jdlrobson) @phuedx do you need help moving this one along presumably those usages wo... [22:25:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:28:28] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews: Implement Unit Tests - https://phabricator.wikimedia.org/T299735 (10BPirkle) This no longer blocked by {T318765}, as test data has been added. It is now blocked by [[ https://github.com/julie... [22:30:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:36:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:41:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:42:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:47:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4046 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4046%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:50:24] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state