[01:02:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[01:12:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[01:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[01:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[01:35:00] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+2] analytics/mediawiki/editgrowthconfig: Ref analytics/common [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/820227 (https://phabricator.wikimedia.org/T314173) (owner: 10Urbanecm)
[01:35:32] <wikibugs>	 (03Merged) 10jenkins-bot: analytics/mediawiki/editgrowthconfig: Ref analytics/common [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/820227 (https://phabricator.wikimedia.org/T314173) (owner: 10Urbanecm)
[01:36:44] <wikibugs>	 10Data-Engineering, 10Event Metrics, 10EventStreams, 10GrowthExperiments-CommunityConfiguration, and 4 others: editgrowthconfig schema: '' should NOT have additional properties, - https://phabricator.wikimedia.org/T314173 (10Tgr) Thanks for the quick fix!
[01:58:58] <icinga-wm>	 PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:10:16] <icinga-wm>	 RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:37:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:42:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:47:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:52:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:57:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:32:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:37:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[03:47:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[04:37:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[04:47:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[04:57:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[05:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[05:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[06:02:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[06:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:07:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:12:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:32:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[07:37:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[08:48:53] <elukey>	 btullis: sorry for the missed review yesterday, code looks good :)
[08:49:35] <btullis>	 Ah, thanks elukey.
[09:07:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[09:17:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[09:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[09:27:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[10:12:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[10:22:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka  - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[10:53:12] <wikibugs>	 10Quarry: "Unsaved changes" notification - https://phabricator.wikimedia.org/T314566 (10Novem_Linguae)
[11:13:53] <elukey>	 btullis: o/
[11:14:11] <btullis>	 elukey: here, how can I help?
[11:14:11] <elukey>	 one thing that I forgot to check for etcd are the SANs of the current clusters
[11:14:23] <elukey>	 for ml-etcd etcd we have, for example
[11:14:24] <elukey>	 DNS:ml-etcd1001.eqiad.wmnet, DNS:ml-etcd1002.eqiad.wmnet, DNS:ml-etcd1003.eqiad.wmnet, DNS:_etcd-server-ssl._tcp.ml_etcd.eqiad.wmnet
[11:14:56] <elukey>	 I didn't get from your change to the etcd code if the last one is added or not
[11:15:04] <btullis>	 I think it should create it with the same as the DNS, using the PKI.
[11:16:03] <btullis>	 With the `hosts => $certname` parameter here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/820090/4/modules/profile/manifests/etcd/v3.pp#90
[11:16:14] <elukey>	 ah I see, using profile::etcd::v3::discovery: "dns:dse_k8s_etcd.%{::site}.wmnet"
[11:16:26] <elukey>	 mmmm
[11:16:27] <elukey>	 no
[11:16:41] <btullis>	 That was my thinking. Will it not work?
[11:17:44] <btullis>	 It gets `$certname` already from line 61 here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/820090/4/modules/profile/manifests/etcd/v3.pp#61
[11:17:54] <elukey>	 so the SAN record that I refer to is taken care by line 61 in theory, so all good
[11:18:37] <elukey>	 we'll see how the SANs look like after bootstrapping the cluster :)
[11:18:51] <elukey>	 I'd expect the hostname + the srv one 
[11:19:24] <btullis>	 Cool. Yes, that's what I'm hoping for too. The filenames will be based on the SAN, not the hostname.
[11:19:48] <elukey>	 ack super :)
[11:20:00] <elukey>	 (going afk, ttl!)
[11:20:24] <btullis>	 Yep, FYI I've added you as a reviewer here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/820416
[11:54:35] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 02), 10Data Pipelines (Sprint 00): Create cassandra loading HQL files from their oozie definition - https://phabricator.wikimedia.org/T311507 (10EChetty)
[12:32:07] <wikibugs>	 10Data-Engineering, 10SRE: Also intake Network Error Logging events into the Analytics Data Lake - https://phabricator.wikimedia.org/T304373 (10CDanis) @BTullis are you still interested in this?
[12:46:02] <wmf-insecte>	 Starting build #109 for job analytics-refinery-maven-release-docker
[13:01:28] <wmf-insecte>	 Project analytics-refinery-maven-release-docker build #109: 09SUCCESS in 15 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/109/
[13:17:42] <wmf-insecte>	 Starting build #68 for job analytics-refinery-update-jars-docker
[13:18:14] <wikibugs>	 (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.4 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820444
[13:18:15] <wmf-insecte>	 Project analytics-refinery-update-jars-docker build #68: 09SUCCESS in 33 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/68/
[13:38:01] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add refinery-source jars for v0.2.4 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820444 (owner: 10Maven-release-user)
[13:43:18] <wikibugs>	 10Data-Engineering-Kanban, 10Generated Data Platform, 10Data Engineering Planning (Sprint 01), 10Patch-For-Review: [Shared Event Platform] - Research Flink Changelog semantics to inform POC MW schema design - https://phabricator.wikimedia.org/T310082 (10Ottomata)
[14:25:06] <wikibugs>	 (03PS3) 10Milimetric: Add world map echart [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/809683
[14:31:34] <wikibugs>	 (03PS4) 10Milimetric: Add world map echart [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/809683
[14:41:03] <milimetric>	 btullis: I'm totally confused about that druid datasource.  I tested it by querying it directly and it looks perfect to me.
[14:41:07] <milimetric>	 https://www.irccloud.com/pastebin/nW471d9u/
[14:41:13] <milimetric>	 that's an example direct druid query
[14:41:28] <milimetric>	 @druid1004:~$ curl -X POST 'localhost:8082/druid/v2/?pretty' -H 'Content-Type:application/json' -H 'Accept:application/json' -d @query
[14:42:09] <milimetric>	 my best guess is that somehow maybe puppet hadn't applied the change to the config by the time we drained and restarted the canary
[14:49:19] <wikibugs>	 10Data-Engineering: Add the requestctl element of the x-analytics map to turnlio's webrequest_sampled_128 - https://phabricator.wikimedia.org/T314578 (10CDanis)
[14:49:25] <wikibugs>	 10Data-Engineering: Add the requestctl element of the x-analytics map to turnlio's webrequest_sampled_128 - https://phabricator.wikimedia.org/T314578 (10CDanis)
[14:50:43] <btullis>	 milimetric: OK, I'm happy to go again with the upgrade now if you like.
[14:51:07] <milimetric>	 btullis: yeah, wanna try it?  Sandra and I are hanging out, we can try the canary
[14:51:42] <btullis>	 ++1
[14:53:44] <btullis>	 Where are you hanging out?
[14:54:30] <milimetric>	 btullis: https://meet.google.com/ndz-gkqa-vap
[15:06:30] <milimetric>	 btullis: all good! "results":[{"timestamp":"2022-07-01T00:00:00.000Z","edits":49868814}]
[15:06:52] <btullis>	 OK, I'll carry on with the rest of the cookbook, ok?
[15:07:52] <btullis>	 Proceeding
[15:10:19] <milimetric>	 yes!
[15:40:24] <milimetric>	 ottomata: who has rights to add Sandra and Njideka to our gerrit group?  They don't have +2!!
[15:40:24] <milimetric>	 https://gerrit.wikimedia.org/r/admin/groups/d34747bee94be39cff54b5fda1ae36b575107792,members
[15:41:38] <btullis>	 milimetric: I have suggested via Slack to Sandra that she applies with a similar request to the one that I put in: https://wikimedia.slack.com/archives/C02291Z9YQY/p1659623302710539?thread_ts=1659620216.608689&cid=C02291Z9YQY
[15:42:09] <btullis>	 I put this ticket in: https://phabricator.wikimedia.org/T300631
[15:42:15] <btullis>	 (for myself)
[15:54:03] <wikibugs>	 (03PS1) 10Snwachukwu: Bump refinery jar version to 1.2.4 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820476
[15:55:38] <wikibugs>	 (03PS2) 10Milimetric: Bump refinery jar version to 0.2.4 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820476 (owner: 10Snwachukwu)
[15:55:46] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Bump refinery jar version to 0.2.4 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820476 (owner: 10Snwachukwu)
[15:58:58] <milimetric>	 btullis: thanks!  I added https://phabricator.wikimedia.org/T314592
[15:59:01] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team: Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10Milimetric)
[15:59:14] <SandraEbele>	 !log Deploying analytics refinery using scap.
[15:59:16] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:09:07] <wikibugs>	 (03PS1) 10Xcollazo: Add table to compute unique editors per country on a monthly basis. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820478 (https://phabricator.wikimedia.org/T314147)
[16:26:16] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 02), 10Data Pipelines (Sprint 00): Create cassandra loading HQL files from their oozie definition - https://phabricator.wikimedia.org/T311507 (10NOkafor-WMF) **Test result overview** https://docs.google.com/spreadsheets/d/1hwZyuRTCRYi-obsbZvScB9r...