[04:17:52] RECOVERY - Check unit status of monitor_refine_event_sanitized_main_immediate on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_event_sanitized_main_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:10:19] 10Data-Engineering, 10Cassandra: Ensure AQS Cassandra client connections are multi-datacenter - https://phabricator.wikimedia.org/T307799 (10JAllemandou) The spark-cassandra-connector indeed supports setting the consistency. It defaults to `LOCAL_QUORUM` for writes and `LOCAL_ONE` for reads. [07:13:52] 10Data-Engineering, 10Data-Engineering-Kanban: Throttle big monthly jobs - network saturation - https://phabricator.wikimedia.org/T307779 (10JAllemandou) 05Open→03Resolved [09:11:02] (03Abandoned) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [09:42:43] Oooh: https://developer.nvidia.com/blog/nvidia-releases-open-source-gpu-kernel-modules/ unblocks some things for us [09:43:11] Oh nice. --^ [09:43:31] WOW ! [09:43:53] This indeed could be a game changer in term of GPU usage [09:48:31] We have stat1010 being installed at the moment (T307399) - the plan is/was to move the GPU from stat1005 to stat1010 as the old host host is decommissioned, but I wonder whether we should look at trying out an Nvidia GPU. [09:48:31] T307399: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 [09:49:04] btullis: Interesting question :) [09:49:43] I don't know how much the existing GPUs are currently used, and if waiting for new ones to be operational would be a problem for the oens using them [09:54:17] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [POC] Use airflow-installed Spark3 for an Airflow job - https://phabricator.wikimedia.org/T308168 (10JAllemandou) [09:54:19] 10Analytics, 10Data-Engineering, 10Epic: Upgrade analytics-hadoop to Spark 3 + scala 2.12 - https://phabricator.wikimedia.org/T291464 (10JAllemandou) [09:59:51] joal: Me neither, but I guess we could ask to keep stat1005 in service a while if we decided to try this option. [10:02:22] (03CR) 10Aklapper: "Hi, who in Data Engineering is supposed to watch and review incoming patches?" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/768167 (owner: 10Aklapper) [10:12:27] (03CR) 10Btullis: [C: 03+1] Use dedicated Phabricator bug report / feature request forms (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/768167 (owner: 10Aklapper) [11:05:14] indeed btullis that would be an option [11:25:23] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) I have a patch for this so that instead of using the `uid` value from the LDAP directory, which corresponds to the user's SSH login name, it uses the `cn` (or... [11:26:55] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) For reference I checked how many of our users would be affected by this issue. Out of 437 valid users, 126 have spaces in their Wikitech usernames. ` btullis@... [11:43:40] (03PS4) 10NOkafor: Updated the get Cassandra password function to; [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/790651 (https://phabricator.wikimedia.org/T306895) [13:38:01] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Patch-For-Review: Set up backups and monitoring of airflow instances - https://phabricator.wikimedia.org/T307102 (10BTullis) We have several checks defined in [[https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/a... [13:40:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Patch-For-Review: Set up backups and monitoring of airflow instances - https://phabricator.wikimedia.org/T307102 (10BTullis) My mistake, all of those systemd checks were created successfully. {F35135046,width=70%} [13:48:47] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) Here is a link to my question in the DataHub Slack: https://datahubspace.slack.com/archives/C033H7EMDQ8/p1652353659591429 {F35135079,width=70%} [14:03:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10BTullis) @Mayakp.wiki in answer to your question: > would you recommend that we change our existing Line Charts to Time Series Line... [14:19:55] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) CC @herron and @elukey on this ticket too, since they also do Kafka SRE work. [14:22:15] btullis: could you just continue to use the posix LDAP names for datahub logins? or do not all folks have those in their ldap account? [14:22:19] (i thought they did...?) [14:35:31] !log razzi@cumin1001:~$ sudo confctl select service=wikireplicas-a,name=dbproxy1019.eqiad.wmnet set/pooled=yes # for T298940 [14:35:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:35:34] T298940: Reimage WMCS db proxies to Bullseye - https://phabricator.wikimedia.org/T298940 [14:49:12] !log undo the 2 previous confctl changes to repool dbproxy1019 to wikireplicas-b only [14:49:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:08:37] ottomata: Yes, everyone has one. There's no technical issue with using them but I had complaints that it wasn't the same username as Wikitech. [15:08:43] mforns: I'm trying to figure out how to launch a python script, with dependencies, from airflow. Can I show you where I got so far and see if I'm on the right track [15:09:05] (I didn't find any examples, if there are some, I'll just follow those) [15:09:55] milimetric: yes! I'm in https://meet.google.com/jvn-hedr-ghq doing some airflow docs, please come in! [15:13:21] Starting the superset 1.4.2 upgrade, superset.wikimedia.org will be offline momentarily [15:29:30] Superset 1.4.2 release went smoothly. Going to upgrade debian on the superset staging instance to prepare for the superset and turnilo upgrades, following https://wikitech.wikimedia.org/wiki/Ganeti#Reinstall_/_Reimage_a_VM [15:39:38] (03CR) 10Razzi: [C: 03+2] "Tested this manually on superset-next.wikimedia.org for a week. No errors found." [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/774924 (https://phabricator.wikimedia.org/T304972) (owner: 10Razzi) [15:39:39] (03CR) 10Razzi: [V: 03+2 C: 03+2] Upgrade superset to 1.4.2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/774924 (https://phabricator.wikimedia.org/T304972) (owner: 10Razzi) [15:41:35] 10Analytics-Wikistats, 10Data-Engineering: Is it possible to setup wikistats for a new wiki? - https://phabricator.wikimedia.org/T308253 (10Dajuno1) [15:42:05] (03PS1) 10Snwachukwu: Create HQL scripts to generate Wikidata's ArticlePlaceholder and Reliability metrics. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/791394 (https://phabricator.wikimedia.org/T300021) [15:45:05] (03CR) 10DLynch: "This seems to have added a lot of unnecessary field documentation to the mobile schema's current.yaml. It removed the legacy eventcapsule " [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [15:46:07] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10BTullis) > I added a request for a tiebreaker controller Kafka node in ulsfo (or eqord)... [15:47:03] (03CR) 10Jsn.sherman: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [15:47:29] (03PS1) 10DLynch: Remove duplication from mobilewebuiactionstracking 1.2.0 [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791399 (https://phabricator.wikimedia.org/T306648) [15:50:38] (03CR) 10DLynch: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [15:53:47] (03CR) 10DLynch: "This doesn't need any change to the version number, since it's just a change to how the schema is written without any change at all to the" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791399 (https://phabricator.wikimedia.org/T306648) (owner: 10DLynch) [15:58:48] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) In simple terms it would be a lot easier if everyone were able to use their posix user name as their primary identifier in DataHub. This is the same as the //... [15:59:59] (03CR) 10Jsn.sherman: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:01:19] (03CR) 10DLynch: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:04:13] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Upgrade Datahub - https://phabricator.wikimedia.org/T308052 (10BTullis) [16:09:32] (03PS1) 10Jsn.sherman: Testing understanding of npm run build-modified [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791405 (https://phabricator.wikimedia.org/T306648) [16:09:40] (03CR) 10jerkins-bot: [V: 04-1] Testing understanding of npm run build-modified [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791405 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:10:43] 10Data-Engineering-Kanban, 10Data-Catalog: Spike: Evaluate datahub schema versioning support - https://phabricator.wikimedia.org/T307716 (10BTullis) [16:10:45] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Upgrade Datahub - https://phabricator.wikimedia.org/T308052 (10BTullis) [16:11:20] 10Data-Engineering-Kanban, 10Data-Catalog: Spike: Evaluate datahub schema versioning support - https://phabricator.wikimedia.org/T307716 (10BTullis) We need to upgrade to DataHub 0.8.34 before this will work. I've added that ticket as a parent of this one. [16:11:43] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Increase Java heap for HDFS namenodes - https://phabricator.wikimedia.org/T307549 (10BTullis) 05Open→03Resolved [16:12:15] (03CR) 10Jsn.sherman: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:16:14] (03CR) 10DLynch: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:18:55] (03CR) 10Jsn.sherman: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:19:23] (03CR) 10DLynch: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:20:03] (03CR) 10DLynch: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:20:56] (03Abandoned) 10Jsn.sherman: Testing understanding of npm run build-modified [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791405 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:28:05] (03CR) 10Jsn.sherman: Add namespace to Mobile & Desktop WebUIActionsTracking (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791053 (https://phabricator.wikimedia.org/T306648) (owner: 10Jsn.sherman) [16:30:07] (03CR) 10Jsn.sherman: [C: 03+2] "Thanks for the instruction and the cleanup; this is much better!" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791399 (https://phabricator.wikimedia.org/T306648) (owner: 10DLynch) [16:30:45] (03Merged) 10jenkins-bot: Remove duplication from mobilewebuiactionstracking 1.2.0 [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791399 (https://phabricator.wikimedia.org/T306648) (owner: 10DLynch) [16:34:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10Mayakp.wiki) Thanks @BTullis for the screen recording, it is super helpful! We will discuss about it and come back with questions if a... [16:44:26] razzi, we got a victorops alert for Superset. It doesn't matter, I acknowledged it anyway. :-) [16:45:24] Oh yep thanks for that! [16:45:49] ottomata: would you have minute? [16:45:55] ottomata: spark3 in batcaev [16:49:50] (03PS3) 10DLynch: DesktopUIActions/MobileUIActions: add pageToken field [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/789324 (https://phabricator.wikimedia.org/T307640) [16:53:54] 10Quarry, 10good first task: Make quarry history descending - https://phabricator.wikimedia.org/T306340 (10rook) [16:54:43] 10Quarry, 10good first task: History shows blank for blank queries - https://phabricator.wikimedia.org/T306342 (10rook) [16:54:51] 10Quarry, 10good first task: Truncate text on quarry history links - https://phabricator.wikimedia.org/T306343 (10rook) [17:03:15] 10Data-Engineering, 10Cassandra: Ensure AQS Cassandra client connections are multi-datacenter - https://phabricator.wikimedia.org/T307799 (10Eevans) >>! In T307799#7923338, @JAllemandou wrote: > The spark-cassandra-connector indeed supports setting the consistency. It defaults to `LOCAL_QUORUM` for writes and... [17:11:41] 10Data-Engineering, 10Data-Engineering-Kanban: RAID battery malfunction in an-worker1081 - https://phabricator.wikimedia.org/T308267 (10BTullis) [17:15:05] 10Data-Engineering, 10Data-Engineering-Kanban: RAID battery malfunction in an-worker1081 - https://phabricator.wikimedia.org/T308267 (10BTullis) p:05Triage→03Medium [17:15:47] (03PS2) 10Razzi: Upgrade to upstream version 1.35.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/777881 (https://phabricator.wikimedia.org/T301990) [17:19:30] 10Quarry: Quarry code cleanup - https://phabricator.wikimedia.org/T308269 (10rook) [17:22:10] 10Quarry, 10good first task: add .swp files to .gitignore - https://phabricator.wikimedia.org/T308270 (10rook) [17:23:00] (03PS1) 10Vivian Rook: drop unneeded done function [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791415 (https://phabricator.wikimedia.org/T308269) [17:23:48] (03CR) 10Vivian Rook: "Looking at it, I'm not sure we need the done function there at all. See any reason not to drop it?" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791415 (https://phabricator.wikimedia.org/T308269) (owner: 10Vivian Rook) [17:27:42] 10Data-Engineering, 10Data-Engineering-Kanban: RAID battery malfunction in an-worker1081 - https://phabricator.wikimedia.org/T308267 (10BTullis) Learn cycle requested, but it has not been started. ` btullis@an-worker1081:~$ sudo megacli -AdpBbuCmd -BbuLearn -a0 Adapter 0: BBU Learn Succeeded. Exit Code: 0x00... [17:28:13] 10Data-Engineering, 10Cassandra: Ensure AQS Cassandra client connections are multi-datacenter - https://phabricator.wikimedia.org/T307799 (10JAllemandou) From what I have seen we can't specify neither a load-balancing policy nor the local datacenter. BUT, from the docs: Connections are never made to data cente... [17:48:50] (03PS3) 10Razzi: Upgrade to upstream version 1.35.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/777881 (https://phabricator.wikimedia.org/T301990) [17:57:53] joal: in meetings all day soOoorryryyyy [17:59:09] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Evaluate Kafka Stretch cluster potential, and if possible, request hardware ASAP - https://phabricator.wikimedia.org/T307944 (10Ottomata) Oh that is a great point. It certainly could. I'll note that in the request. [18:25:48] (03PS1) 10Clare Ming: Add is-anon property to schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791422 (https://phabricator.wikimedia.org/T307381) [18:49:31] np ottomata :) [18:51:04] otto: whenever you get out of your meeting and take a break, I need to ask your opinion on running python/CLI stuff through airflow (via skein). I talked to Marcel and asked the team too, we got some decisions to make. [20:08:15] ottomata: got time left today for ^? [20:26:39] an-worker1081 has been alerting all day about mega raid, can someone take a look? [20:39:26] (03PS1) 10Jiyu: .gitignore: Add *.swp to .gitignore [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791453 (https://phabricator.wikimedia.org/T308270) [20:39:32] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791453 (https://phabricator.wikimedia.org/T308270) (owner: 10Jiyu) [20:41:22] 10Quarry, 10Patch-For-Review, 10good first task: add .swp files to .gitignore - https://phabricator.wikimedia.org/T308270 (10Abstract09) a:03Abstract09 [20:56:36] (03CR) 10Vivian Rook: [C: 03+2] "Looks great! Thanks!" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791453 (https://phabricator.wikimedia.org/T308270) (owner: 10Jiyu) [20:56:52] (03CR) 10Vivian Rook: [V: 03+2 C: 03+2] .gitignore: Add *.swp to .gitignore [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/791453 (https://phabricator.wikimedia.org/T308270) (owner: 10Jiyu) [21:00:42] 10Quarry, 10Patch-For-Review, 10good first task: add .swp files to .gitignore - https://phabricator.wikimedia.org/T308270 (10rook) 05Open→03Resolved [21:02:52] milimetric: ahhh hjusut saw this [21:02:53] yes [21:03:00] let me merge this spark thing for joal so he can keep workign tomorrow [21:03:03] then i'm still here for a bit [21:03:31] ok, I just need to chat a bit to know what direction to go in [21:04:21] k 5 mins [21:13:22] ok this is more than 5 minutes, but i am very close [21:14:09] milimetric: l i'm in cave, still finishing up but come on in! [21:38:13] (VarnishkafkaNoMessages) firing: ... [21:38:13] varnishkafka for instance cp3062:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3062:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:43:13] (VarnishkafkaNoMessages) resolved: ... [21:43:13] varnishkafka for instance cp3062:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3062:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:51:37] (03PS1) 10Razzi: Upgrade to superset 1.35.0 [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/791461 (https://phabricator.wikimedia.org/T301990) [21:55:06] joal: aqu, merged spark3 conf! [21:55:07] 10Data-Engineering, 10Airflow, 10Patch-For-Review: Install spark3 in analytics clusters - https://phabricator.wikimedia.org/T295072 (10Ottomata) @Antoine_Quhen @JAllemandou /etc/spark3/conf is now on an-launcher1002 and an-test-client1001 :) thank you! [21:56:13] (VarnishkafkaNoMessages) firing: ... [21:56:13] varnishkafka for instance cp3064:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:01:13] (VarnishkafkaNoMessages) resolved: ... [22:01:13] varnishkafka for instance cp3064:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:12:13] (VarnishkafkaNoMessages) firing: ... [22:12:13] varnishkafka for instance cp3064:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:27:13] (VarnishkafkaNoMessages) resolved: ... [22:27:13] varnishkafka for instance cp3064:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3064:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages