[01:08:26] 10Data-Engineering, 10Anti-Harassment, 10Product-Analytics: Mediawiki history has no data on IP blocks - https://phabricator.wikimedia.org/T211627 (10nettrom_WMF) Moving this to the Data Engineering board because I think having block data for both registered and IPs in `mediawiki_user_history` (or maybe in a... [01:09:22] 10Data-Engineering, 10Anti-Harassment, 10Product-Analytics: Distinguish between types of block events in the Mediawiki user history table - https://phabricator.wikimedia.org/T213583 (10nettrom_WMF) Moving this to the Data Engineering board because I think having this type of data on blocks in the Mediawiki U... [08:01:36] 10Data-Engineering, 10SRE, 10observability, 10serviceops: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10elukey) [08:32:37] * addshore waits for his event table to appear in hive [08:48:00] (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM! Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757035 (https://phabricator.wikimedia.org/T293406) (owner: 10MNeisler) [09:37:29] elukey: Would now be a good time to deploy and test https://gerrit.wikimedia.org/r/c/operations/puppet/+/742747 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/755435 ? [09:37:36] Are you around if I get stuck? [09:43:06] Plan will along the lines of: [09:43:06] * Notify traffic that I'm about to stop puppet and test these changes on cp3050 [09:43:06] * Stop puppet on all cp* hosts [09:43:06] * Merge 742747 and 755435 [09:43:06] * Run puppet on cp3050 and check for correct configuration of /etc/varnishkafka/webrequest.conf [09:43:07] * Check the status of varnishkafka-webrequest.service which should have restarted automatically. [09:43:29] * Restart puppet on the fleet of cp* nodes once all is well. [09:43:42] Have I missed anything? [09:46:26] btullis: o/ [09:46:30] sorry just seen the message [09:46:36] I am available :) [09:46:44] I only just typed it :-) [09:47:15] the plan is ok, remember that we run multiple vk instances [09:47:34] on text nodes, we have vk-webrequest, vk-statsd, vk-eventlogging [09:48:02] all three will be refreshed [09:48:17] you can see them in https://grafana.wikimedia.org/d/000000253/varnishkafka [09:48:36] the bundle-ca change will affect all three of them [09:48:48] OK, so all three will be updated to use the new PKI. Only webrequesat gets the cipher change, right? [09:48:48] meanwhile the format one only the webrequest [09:48:52] exactly [09:49:46] to test the change in the message sent to kafka it is sufficient to use kafkacat on a stat100x node and filter for the host updated (like 3050) [09:50:01] to test the pki change it should be enough to observe metrics and vk logs [09:50:12] and also using kafkacat [09:50:24] if there is a problem no messages will be displayed [09:50:40] OK, thanks. I hadn't thought of that. [10:00:25] I'm confused why this isn't returning anything. [10:00:29] https://www.irccloud.com/pastebin/NQU5PQfJ/ [10:02:10] Ah, `webrequest_text` [10:04:21] to select a single node (for later on) [10:04:22] kafkacat -C -t webrequest_text -b kafka-jumbo1001.eqiad.wmnet | jq '. | select(.hostname == "cp3050.esams.wmnet")' [10:04:27] very handy [10:05:08] Nice, I had a grep, but I see that this was picking up peer cache hits. [10:05:10] at some point in the future we'll have to enforce TLS for all kafka clients, it would be great [10:05:20] Yep yep. [10:05:27] (too many things sigh) [10:07:27] !log btullis@cumin1001:~$ sudo cumin 'O:cache::upload or O:cache::text' 'disable-puppet btullis-T296064-T299401' [10:07:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:07:31] T299401: VarnishKafka to propagate user agent client hints headers to webrequest - https://phabricator.wikimedia.org/T299401 [10:07:31] T296064: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 [10:12:58] Merged the two changes. [10:14:42] super [10:15:30] Did we do a rolling-restart of kafka-main within the past 6 days? varnishkafka-statsv.service is showing kafka disconnects in its status output, although it's running fine now. [10:16:45] not that I know of, I upgraded the kafka-main os to buster but it was earlier than that [10:17:48] Not to worry, just interested. Puppet applied. All three varnishkafka instances restarted and running. [10:19:39] on cp3050? [10:19:55] (I just fixed the instance selection in the grafana dashboard, it wasn't working) [10:20:30] Yes, only on cp3050. I also see `ch_ua`,`ch_ua_mobile`,`ch_ua_platform` in the events. Although I had to add the `-o end` offset to kafkacat first. [10:20:50] perfect [10:20:58] let's do it on another node as well [10:21:07] just to be sure, 3050 had some changes already applied IIRC [10:22:24] I see the new fields as well, the values are getting in [10:22:39] there seems to be a weird double quoting [10:22:40] "ch_ua_platform": "\"Android\"" [10:23:27] Yes, I see what you mean. [10:25:28] o/ [10:25:30] o/ [10:25:36] 10Analytics, 10Wikidata, 10Wikidata Analytics: dumps.wikimedia.org access logs on stat1007 are incomplete since May 2021 (possibly earlier) - https://phabricator.wikimedia.org/T299358 (10Lucas_Werkmeister_WMDE) That seems to have fixed the logs; using the same `zgrep` pipeline from the task description (now... [10:25:53] phuedx: Ben deployed the change to one caching node, all good but we see stuff like [10:26:01] "ch_ua_platform": "\"Windows\"" [10:26:11] is the double quoting expected? I mean, part of the specs [10:26:35] (one set of quotes are added by the json message of varnishkafka, but the second pair seems to be the value of the header) [10:27:09] btullis: something interesting is that cp3050 increased its cpu usage [10:27:51] weird [10:27:59] elukey: Yes. The Sec-CH-UA header will contain quoted values, e.g. Sec-CH-UA: "";v="", ... [10:28:13] phuedx: perfect thanks for confirming, all good then :) [10:28:18] Other examples are given here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Sec-CH-UA#examples [10:31:45] elukey: I'm not sure that I understand what the axes represent on this graph: https://grafana.wikimedia.org/d/000000253/varnishkafka?viewPanel=42&orgId=1&var-datasource=esams%20prometheus%2Fops&var-source=webrequest&var-cp_cluster=cache_text&var-instance=cp3050 [10:32:37] btullis: it should be the cpu time spent in the cgroup that runs the varnishkafka instance [10:32:38] I see what you mean about an uptick, but if you compare it to the host CPU overall (https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cp3050&var-datasource=thanos&var-cluster=cache_text&from=now-1h&to=now) I can't make out what the effect represents. [10:32:55] Ah, OK. Good to know. [10:33:03] nothing really, and the other two instances that you restarted are ok [10:34:12] maybe it is a temporary weirdness [10:34:15] Yeah, looks like it is dropping as well, so perhaps the restarts just caused it to spike. I'll wait a few more minutes to see if it drops back to what it was, before re-enabling puppet on all cp-* nodes. [10:34:50] even if it stays at that level it should be fine, it is comparable with other instances [10:39:13] OK, re-enabling puppet now. [10:39:38] it will be interesting to observe if other instances raise their cpu-usage as well [10:39:49] and if the tls latencies to the brokers will stay the same etc.. [10:40:24] Yeah. Interestingly, the eventlogging one seems to have reduced its CPU usage slightly, but this varies more over time anyway. https://grafana.wikimedia.org/d/000000253/varnishkafka?viewPanel=42&orgId=1&var-datasource=esams%20prometheus%2Fops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3050 [10:41:16] btullis: congrats for your first vk deployment :) [10:41:48] 10Analytics, 10Wikidata, 10Wikidata Analytics: dumps.wikimedia.org access logs on stat1007 are incomplete since May 2021 (possibly earlier) - https://phabricator.wikimedia.org/T299358 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE The file size also went back up starting from t... [10:42:07] Thanks for being patient and super-helpful. [10:43:59] super happy to help, the more people know how these things work the better :) [10:58:14] ^ [10:58:45] Congratulations btullis! [10:59:10] Thank you. :-) [11:00:19] 10Analytics-Radar, 10Data-Engineering-Radar, 10Event-Platform, 10Patch-For-Review: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 (10elukey) The last clients to move should be eventstreams and eventgate! Next steps: - deploy https://gerrit.wikimedia.org/r/7534... [12:48:54] FYI, I'm rolling out apache security updates on the various analyics web UIs in the next 1-2 minutes, there might be brief (and unvoidable) blips [12:50:08] those are done now [12:55:15] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [13:07:33] moritzm: many thanks [13:47:57] (03PS1) 10Joal: Add user-agent client hints to webrequest tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757445 (https://phabricator.wikimedia.org/T299402) [13:48:43] mforns, milimetric, btullis - I just sent --^ after the update of varnishkafka for the CH-UA headers [13:48:57] I'll deploy today if you give me + :) [13:51:25] (03CR) 10Btullis: [C: 03+1] "Looks good to me. :-)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757445 (https://phabricator.wikimedia.org/T299402) (owner: 10Joal) [14:02:05] thanks btullis for the quick review - I'm gonna wait a second +1 and that'll make +2 :) [14:17:06] (03CR) 10Milimetric: [C: 03+2] Add user-agent client hints to webrequest tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757445 (https://phabricator.wikimedia.org/T299402) (owner: 10Joal) [14:30:01] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:36:08] ottomata: o/ hiii since you are in eventgate-land, do you have time for the ca-bundle deployment ? [14:36:21] les do it [14:36:22] (checking that staging looks good etc..) [14:36:29] oh i should wait for my restarts then. [14:36:36] ah whatevs i'll finish eventgate-analytics and do it again [14:36:45] elukey: did we merge already? [14:37:19] ottomata: nope just done :( [14:37:28] kay [14:37:39] finishing my roll restart, we can apply changes after [14:37:45] super [14:38:37] you shouldn't get the new changes up to the next puppet run but check with helmfile diff before syncing just in case [14:39:10] done, okay, also...i have meetings starting in 20 mins [14:39:17] can i start with you, and if needed you finish? [14:39:26] sure sure [14:39:34] ok [14:39:41] i'm looking at eventgate-logging-external staging diff [14:39:43] i see the change [14:39:49] ok to deploy there in staging? [14:39:52] (thanks for the time and patience) [14:39:53] + ssl.ca.location: /etc/ssl/certs/wmf-ca-certificates.crt [14:39:55] +1 [14:40:01] ah wait a sec [14:40:10] there should also be an image change [14:40:21] oh yes [14:40:22] image cahnge too [14:40:30] + image: "docker-registry.wikimedia.org/wikimedia/eventgate-wikimedia:2022-01-11-142353-production" [14:40:33] elukey: do we have a ticket [14:40:37] want to attach it to log message [14:40:44] ? [14:40:52] lemme check [14:41:00] T296064 [14:41:01] T296064: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 [14:42:37] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:43:51] I think that --^ failed due to the deploy [14:46:03] maybe, i think separate issue, is the reason i was restarting eventgate-analytics [14:46:14] yep yep [14:46:17] I've restarted it [14:46:23] ottomata: o/ [14:46:31] o/ [14:46:41] if you have a minute, could you +1 that small patch of me so that I could deploy please? [14:47:03] Mwarf ottomata - didin't notice milimetric already did - my bad [14:47:21] (03CR) 10Ottomata: [C: 03+1] Add user-agent client hints to webrequest tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757445 (https://phabricator.wikimedia.org/T299402) (owner: 10Joal) [14:47:30] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/757445 (https://phabricator.wikimedia.org/T299402) (owner: 10Joal) [14:47:43] elukey: eg logging exteranl staging looks good [14:47:52] ok if i continue to codfw and eqiad? [14:48:19] a-team: I'm gonna deploy refinery now - anything you'd like me to add to the current 2 things (webrequest CH-UA and edit-hourly update) [14:48:34] Nothing from me, thanks. [14:48:55] ottomata: +1 [14:49:34] sorry I had no time to deploy, jo [14:49:47] np milimetric - it's actually good we waited for today :) [14:50:18] 10Data-Engineering, 10SRE, 10observability, 10serviceops: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10Ottomata) > A better and more stable Kafka Mirror Maker (even if after all the work that Andrew did we have something very stable as well now) This really does look great, and has s... [14:50:28] train is on the way! [14:52:25] !log Deploy refinery with scap [14:52:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:52:40] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:54:15] elukey: since i have your attention! :) [14:54:17] https://phabricator.wikimedia.org/T296543#7647470 [14:54:41] summary: i want to make skein start a yarn AppMaster, that will run spark-submit [14:54:50] airflow will be doing this, so it will need to be able to use a keytab [14:55:16] from what I can tell, it should be ok to use yarn localresources to upload the keytab to the yarn appmaster [14:55:47] (eg logging-external done and looks good, moving on to eventgate-analytics-external) [14:56:01] !log elukey@cp4035:~$ sudo systemctl restart varnishkafka-webrequest.service - metrics showing messages stuck for a poll() [14:56:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:56:47] ottomata: yes it should be good, everything is encrypted and authenticated [14:57:04] and it should be a temporary per-user cache so I think it is ok [14:57:08] wow that is the answer I wanted to hear! joal ^ :) [14:57:12] yeah it a per app cache [14:57:24] but it is not shared by users right? [14:57:26] no [14:57:39] i mean, not even a same user in a different app coudl acces it [14:57:50] exactly yes this was my understanding [14:58:00] when spark uploads the keytab it uses the same IIRC [14:58:10] ya [14:58:14] that's my understanding as well, but I'm always concerned when moving secrets around [14:58:26] ok all good then - let's do it :) [14:59:17] 10Data-Engineering-Kanban, 10Airflow: Tooling for Deploying Conda Environments - https://phabricator.wikimedia.org/T296543 (10Ottomata) Beautiful words from Luca re uploading keytab: > ottomata: yes it should be good, everything is encrypted and authenticated :) [15:00:15] okay elukey eg analytics done. [15:00:22] i gotta go to meeting [15:00:28] can you do eventagte-analytics and eventgate-main? [15:02:36] 10Data-Engineering-Kanban, 10Airflow: Tooling for Deploying Conda Environments - https://phabricator.wikimedia.org/T296543 (10elukey) >>! In T296543#7652986, @Ottomata wrote: > Beautiful words from Luca re uploading keytab: >> ottomata: yes it should be good, everything is encrypted and authenticated > > :)... [15:04:16] ottomata: we can do it later on or tomorrow together, no rush, I'd be happier with you around :) [15:06:06] !log elukey@cp4035:~$ sudo systemctl restart varnishkafka-eventlogging.service - metrics showing messages stuck for a poll() [15:06:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:06:16] I don't like this [15:08:16] but it seems limited to a couple of hosts in ulsfo [15:08:55] in the logs for all instances I see [15:08:55] KAFKAERR: Kafka error (-195): ssl://kafka-main1004.eqiad.wmnet:9093/1004: Disconnected (after 1199322ms in state UP) [15:10:46] !log elukey@cp4036:~$ sudo systemctl restart varnishkafka-statsv [15:10:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:10:50] !log elukey@cp4036:~$ sudo systemctl restart varnishkafka-eventlogging [15:10:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:19:56] mmmm cp403[5,6] show no traffic [15:20:02] they are maybe new/special nodes [15:23:05] ok elukey lets do later [15:27:01] !log Deploy refinery to HDFS [15:27:02] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:40:24] !log Kill-restart edit-hourly oozie job after deploy [15:40:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:40:51] so folks on some ulsfo caching nodes we have been dropping vk messages for a while :( [15:40:58] even before today's changes [15:41:14] vk was failing to get the shm handle of varnish [15:41:31] hm - we have almost not seen errors of missing data elukey - one yesterday is all I recall in the past weeks [15:42:47] joal: I hope to be wrong, I'll open a task [15:44:19] !log Kill-restart webrequest oozie job after deploy [15:44:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:46:27] joal: i think if a vk node is not sending any data, we wouldn't have any data loss info aout it [15:46:29] about * [15:47:16] ottomata: if we're missing full vk hosts, yes - if a host misses some messages is what I was thinking [15:49:49] joal: I think that due to some weird cp node statuses we have dropped traffic from a lot of cp nodes completely [15:50:09] cp3045 for example [15:50:16] Valentin just restarted it with the correct settings [15:50:18] okey :S do we have an idea from how long elukey? [15:50:54] joal: see https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&from=now-3h&to=now&var-datasource=ulsfo%20prometheus%2Fops&var-source=webrequest&var-cp_cluster=cache_text&var-instance=All [15:51:10] still not a precise idea, the last varnishkafka upgrade was in late 2020.. [15:52:58] ack elukey [15:54:06] !log Add new CH-UA fields to wmf_raw.webrequest and wmf.webrequest [15:54:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:01:50] ok traffic restored on a lot of nodes [16:03:03] the problem was cross-instance, so not all webrequest-related [16:03:14] Going to make a post-mortem so we can understand the damage [16:07:27] 10Data-Engineering, 10Data-Engineering-Kanban, 10Metrics-Platform: Add user agent client hints to the `webrequest` table - https://phabricator.wikimedia.org/T299402 (10JAllemandou) [16:08:01] thank you very much elukey [16:13:21] elukey: is it just those 2 nodes? [16:15:15] even so...that is 1% of webrequest_text? is that right? [16:15:29] about 1000 reqs/sec handled by those 2 nodes? [16:16:19] webrequest_text around 100K / second? [16:17:02] the affected nodes are [16:17:03] cp1087.eqiad.wmnet,cp[4021,4033-4034,4036].ulsfo.wmnet [16:17:08] and cp4035 [16:18:11] looking at chart it seems cp1087.eqiad.wmnet has not come back sending data [16:18:12] and elukey, we lost the requestt logs then, right? or was varnish not serving any reqs? [16:18:22] the former yes [16:18:41] nasty [16:18:48] yeah we should quantify [16:18:53] joal: checking yes [16:18:58] that might be significant [16:19:06] not all webrequest, to be clear [16:19:11] various instances [16:19:26] yes, but all webrequests served by those instances, right? [16:19:32] 3 hosts in upload from ulsfo and 2 in text from ulsfo [16:19:44] ok upload slightly less important [16:20:00] nono I need to verify, on some node it may have hit say only eventlogging/statsv [16:20:04] oh [16:20:07] k [16:20:11] and the one from eqiad, but I don't yet have it back to check [16:22:17] joal: should be working now [16:22:22] I see logs from kafkacat [16:25:09] so for webrequest, I'd say 3 vk instances: cp403[56] and cp1087 [16:25:11] 3 text nodes [16:25:47] the rest *should* be related to eventlogging and statsv [16:28:34] https://phabricator.wikimedia.org/T290694 [16:29:00] so this is re-assuring - my theory is that these hosts are new nodes (like pooled during the past couple of months) [16:29:11] and for some reason, they came up with the wrong version of vk [16:29:16] and never sending data [16:31:48] again elukey - thank you <3 [16:52:56] 10Data-Engineering: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10elukey) [16:53:14] joal, ottomata - draft of the post-mortem in --^ [17:01:02] 10Data-Engineering: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10Ottomata) > varnishkafka-webrequest on cp1087, cp4035 and cp4036 (text) > varnishkafka-webrequest on cp4021, cp4033, cp4034 (upload) We... [17:33:14] folks are we doing the sre sync? [17:33:30] elukey: we're in planning session, you'll be alone :S [17:33:32] Sorry. we're in quarterly planning. [17:33:41] ahh okok np [17:47:16] 10Data-Engineering: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10elukey) The main issue is: ` varnishkafka | 1.0.14-1 | buster-wikimedia | main | amd64, source varnishkafka | 1.1.0-1 |... [17:49:27] (03PS11) 10Phuedx: [WIP] Metrics Platform event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [17:51:18] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Metrics Platform event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [17:53:40] 10Data-Engineering: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10elukey) >>! In T300164#7653547, @Ottomata wrote: >> varnishkafka-webrequest on cp1087, cp4035 and cp4036 (text) >> varnishkafka-webreque... [18:02:38] ottomata: any idea how long after receiving events I should see my table appear in the event db in hadoop? [18:02:45] or did I miss another step :P [18:09:39] addshore: if there was no error on validation etc, I think it's something like a few hours (less than 6 for sure) [18:10:15] ahh, maybe i should check the validation logs ;) [18:20:25] (03PS12) 10Phuedx: [WIP] Metrics Platform event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [18:26:41] (03PS1) 10Joal: Integrate SparkSQLNoCLIDriver and HiveToCassandra [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/757495 (https://phabricator.wikimedia.org/T297934) [18:30:51] (03CR) 10jerkins-bot: [V: 04-1] Integrate SparkSQLNoCLIDriver and HiveToCassandra [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/757495 (https://phabricator.wikimedia.org/T297934) (owner: 10Joal) [18:32:38] addshore: yea, usually 2ish [18:32:51] u got no table? lets see [18:32:53] are the events in kafka? [18:34:28] yes i see some [18:34:54] addshore: you hve a table [18:34:57] event.mwcli_command_execute [18:35:15] *looks again* [18:36:25] nice, the table list just went form 253 to 255 for me :D [18:38:15] (03CR) 10Ottomata: Integrate SparkSQLNoCLIDriver and HiveToCassandra (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/757495 (https://phabricator.wikimedia.org/T297934) (owner: 10Joal) [18:38:19] (03CR) 10Ottomata: [C: 03+1] Integrate SparkSQLNoCLIDriver and HiveToCassandra [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/757495 (https://phabricator.wikimedia.org/T297934) (owner: 10Joal) [18:38:36] :) [19:56:07] (03PS2) 10Joal: Integrate SparkSQLNoCLIDriver and HiveToCassandra [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/757495 (https://phabricator.wikimedia.org/T297934) [21:11:15] (03PS28) 10AGueyte: Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) [22:10:13] (03CR) 10Ebernhardson: [C: 03+2] rdf-streaming-updater: add a "reconcile" operation [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/737429 (https://phabricator.wikimedia.org/T279541) (owner: 10DCausse) [22:10:40] (03CR) 10Ebernhardson: [C: 03+2] rdf_streaming_updater: add a reconcile event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/756536 (https://phabricator.wikimedia.org/T279541) (owner: 10DCausse) [22:11:32] (03Merged) 10jenkins-bot: rdf-streaming-updater: add a "reconcile" operation [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/737429 (https://phabricator.wikimedia.org/T279541) (owner: 10DCausse) [22:11:39] (03Merged) 10jenkins-bot: rdf_streaming_updater: add a reconcile event schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/756536 (https://phabricator.wikimedia.org/T279541) (owner: 10DCausse) [22:33:53] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [22:45:15] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [22:58:59] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers