[05:31:12] (VarnishkafkaNoMessages) firing: (2) varnishkafka on cp1081 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:36:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka on cp1081 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [07:07:29] 10Data-Engineering: Check home/HDFS leftovers of bmansurov - https://phabricator.wikimedia.org/T320367 (10MoritzMuehlenhoff) [07:22:40] 10Data-Engineering, 10Machine-Learning-Team, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) >>! In T319214#8280625, @gmodena wrote: > This looks really interesting, especially for ease of deployment. @elukey do you know if `http_client` calls are... [08:24:59] (03CR) 10Michael Große: "This change is ready for review." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) (owner: 10Michael Große) [08:44:53] 10Data-Engineering, 10Machine-Learning-Team, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10BTullis) I'm in favour of further experiments with benthos, given that it appears to be so simple run and and so flexible. We might think of Benthos as the //Swis... [08:56:34] 10Data-Engineering, 10Machine-Learning-Team, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10elukey) I had a chat with Filippo last week and it shouldn't be too difficult to package/deploy Benthos somewhere. We could create a Debian package and deploy it t... [10:08:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2027 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2027%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:13:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2027 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2027%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:16:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2033 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2033%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:19:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:21:12] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp2027 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:23:42] (VarnishkafkaNoMessages) firing: (2) varnishkafka on cp2035 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:24:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:28:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp2033 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:46:13] (VarnishkafkaNoMessages) firing: varnishkafka on cp5005 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5005%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [10:51:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp5005 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5005%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:35:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp6009 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=drmrs%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp6009%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:40:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp6009 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=drmrs%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp6009%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:41:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp6011 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=drmrs%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp6011%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:43:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp6004 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=drmrs%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp6004%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:46:12] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp6009 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:48:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp6006 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=drmrs%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp6006%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:48:42] (03PS3) 10Michael Große: Track views of EntitySchema namespaces on Wikidata [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) [11:48:55] btullis: Hi! there are many alerts on varnishkafka not sending enough rows - any hint on what is happening? [11:49:42] (VarnishkafkaNoMessages) firing: (3) varnishkafka on cp6009 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:49:42] (VarnishkafkaNoMessages) firing: (3) varnishkafka on cp6009 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:49:50] git review [11:49:52] woops [11:50:00] (03PS20) 10Joal: Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [11:50:06] (03CR) 10Michael Große: "With the additional requirement of T319380 to also track spiders separately, I wonder if it would make sense to rewrite this file to one t" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) (owner: 10Michael Große) [11:51:30] joal: No, I'm afraid I haven't been able to ascertain what the cause is yet. They're all caused by spikes so recover fairly quickly, but I haven't been able to analyse yet whether we simply to refine the thresholds to smooth out these bumps, or what. Sorry. [11:52:02] np - if at least we're confident in that they're not real problems, I'm fine with the status :) [11:52:09] thanks btullis [11:53:12] (VarnishkafkaNoMessages) firing: (2) varnishkafka on cp6006 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:54:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp6011 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:54:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp6011 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:57:10] joal: It's definitely a concern, but I need to find some proper time to look into it in more depth. [11:58:12] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp6004 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:58:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka on cp6006 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [11:59:16] 10Data-Engineering, 10Equity-Landscape: Editorship Output Rank Metrics - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) @JAnstee_WMF after fixing the code, I re-ran the outputs and calculated the interim ranks and final outputs on sheets as well. here is my copy of QC workbook: https://docs.goog... [12:00:59] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10Data Pipelines (Sprint 02): Upgrade WMFData Python Package to use Spark3 - https://phabricator.wikimedia.org/T318587 (10Antoine_Quhen) Some picks from anaconda-wmf conda-create-stacked & conda-activate-stacked: * Modify conda-analytics to ship... [12:39:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp3053 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp3053%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:41:13] (03CR) 10Lucas Werkmeister (WMDE): Track views of EntitySchema namespaces on Wikidata (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) (owner: 10Michael Große) [12:42:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp3054 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp3054%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:43:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp3055 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp3055%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:44:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka on cp3053 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:47:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp3054 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp3054%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:48:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp3055 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp3055%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:49:42] (VarnishkafkaNoMessages) firing: (3) varnishkafka on cp3053 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:49:42] (VarnishkafkaNoMessages) firing: (2) varnishkafka on cp3055 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:54:42] (VarnishkafkaNoMessages) resolved: (4) varnishkafka on cp3053 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:54:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp3055 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:58:42] (VarnishkafkaNoMessages) firing: (3) varnishkafka on cp3059 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:03:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp3059 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:06:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1076 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1076%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:08:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1077 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1077%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:11:12] (VarnishkafkaNoMessages) firing: (3) varnishkafka on cp1076 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:13:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1077 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1077%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:14:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1081 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1081%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:16:12] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp1076 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:17:42] (VarnishkafkaNoMessages) firing: (4) varnishkafka on cp1075 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:19:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka on cp1077 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:21:42] (VarnishkafkaNoMessages) firing: (4) varnishkafka on cp1076 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:22:42] (VarnishkafkaNoMessages) resolved: (4) varnishkafka on cp1075 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:23:42] (VarnishkafkaNoMessages) firing: (5) varnishkafka on cp1080 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:26:42] (VarnishkafkaNoMessages) resolved: (3) varnishkafka on cp1080 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:28:42] (VarnishkafkaNoMessages) resolved: (5) varnishkafka on cp1080 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [13:59:14] wow [13:59:22] that's a sensitive alert :) [13:59:31] vgutierrez: Yeah. I was just going to ask if you had any idea about this. [13:59:52] so I've upgraded HAProxy today [14:00:07] that required pooling and depooling HAProxy hence draining traffic from varnish as well [14:00:28] Ah yes, I saw an entry on the SAL about HAProxy. Was wondering. [14:12:58] I'm also not sure why some of these messages include the link to the Grafana dashboard and others do not. [14:15:52] btullis: o/ [14:16:06] Even if I set the moving average over 10 minutes instead of 3 minutes it seems the alarm would still have fired. [14:16:06] https://thanos.wikimedia.org/graph?g0.expr=sum%20by%20(hostname%2Ccluster)%20(label_replace%20(irate(rdkafka_producer_topic_partition_msgs%5B10m%5D)%2C%20%22hostname%22%2C%20%22%241%22%2C%20%22instance%22%2C%20%22(.*)%3A.*%22))%20%2F%20sum%20by%20(cluster%2Chostname)%20(label_replace%20(irate(varnish_requests%7B%20method!~%22PURGE%22%7D%5B10m%5D)%2C%20%22hostname%22%2C%20%22%241%22%2C%20%22instance%22%2C%20%22(.*)%3A.*%22))%20%3 [14:16:06] C%200.2&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D [14:16:11] I just got a -1 from CI for the datahub chart, weird https://integration.wikimedia.org/ci/job/helm-lint/8008/console [14:16:34] (nothing urgent please keep going with the rest) [14:17:21] elukey: That is weird though. [14:17:50] afaics nothing changed recently right? Seems a weird CI result [14:18:07] it says [14:18:09] "Error: execution error at (datahub/charts/datahub-mae-consumer/templates/deployment.yaml:33:10): Elasticsearch host must be specified" [14:19:40] ah okok all wip in service ops, helm has been upgraded and it is more sensitive [14:26:51] (03CR) 10Lucas Werkmeister (WMDE): Track views of EntitySchema namespaces on Wikidata (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) (owner: 10Michael Große) [14:32:48] 10Data-Engineering, 10Machine-Learning-Team, 10observability: Evaluate Benthos as stream processor - https://phabricator.wikimedia.org/T319214 (10Ottomata) > We might think of Benthos as the Swiss army knife of stream processing, compared with the CNC milling machine of stream processing provided by Flink.... [14:36:26] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10Data Pipelines (Sprint 02): Upgrade WMFData Python Package to use Spark3 - https://phabricator.wikimedia.org/T318587 (10Ottomata) > add SPARK_CONF_DIR=/etc/spark3/conf in the target environment Hm, I'm not sure this is quite the right thing to d... [14:56:38] joal: aqu: how about now? https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/172 [14:57:56] Approved mforns :) Thanks for the patch :) [14:58:05] Hi mforns, looks even better. [14:58:06] mforns: I let ou merge when ou wish [14:58:10] joal: thank you!! [14:58:22] aqu: thanks!!! [15:22:47] !log deployed airflow to launch unique devices cassandra backfilling [15:27:33] joal, should I merge and deploy this as well?? https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/170 [15:27:52] Yes please mforns! [15:27:59] 👍 [15:28:55] joal: is it a good time now to start cassandra backfilling? [15:29:04] anything against? [15:29:10] nope, nothing against [15:29:52] ok! [15:31:30] !log started unique devices daily back-filling in cassandra from 1st of July to end of Sept [15:31:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:31:40] For the archives, the helm error above was addressed by this CR: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/841135 [15:34:37] !log deployed airflow to fix geoeditors_public_monthly DAG [15:34:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:36:52] !log reran geoeditors_public_monthly airflow DAG for Sept 2022, after fix [15:36:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:37:46] 10Data-Engineering: Check home/HDFS leftovers of bmansurov - https://phabricator.wikimedia.org/T320367 (10leila) Thanks, @MoritzMuehlenhoff. I'll remain the point of contact until 2022-10-21. For all questions related to this ticket after that point that you need Research's input, please ping @Miriam. [16:38:28] joal: wanna quickly meet and discuss/decide on deletion script? we can call other people to the meeting [16:38:55] mforns: if ok for you let's wait for tomorrow, to have Andrew on this one [16:39:03] ok, sure! [16:39:10] Thank you :) [16:39:13] no problemo, we have time! [16:42:57] RECOVERY - Check unit status of refinery-drop-eventlogging-legacy-raw-partitions on an-launcher1002 is OK: OK: Status of the systemd unit refinery-drop-eventlogging-legacy-raw-partitions https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [18:42:54] aqu, joal, I created an MR to remove the temporary cassandra loading DAGs, since they finished their job. Do you want to have a look? https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/173 [18:59:16] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017 (10Ottomata) @gmodena @dcausse I talked to @daniel today, and ended up with an interesting question about... [22:10:20] 10Analytics-Wikistats, 10Data Engineering Planning, 10Data Pipelines: [Wikistats] Add newly translated languages - https://phabricator.wikimedia.org/T311315 (10Aftabuzzaman) Hi, Can someone help with this please? Another version was relesed in the meantime but it didn't include Bengali translation. [23:05:48] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:17:04] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers