[09:16:00] <gehel>	 https://www.irccloud.com/pastebin/5z4MHQEE/
[09:16:06] <gehel>	 dcausse: ^
[09:26:17] <dcausse>	 gehel: actually these logs are quite old, would you have something that's around ~20:30 UTC yesterday? (ryankemper@cumin2002 START - Cookbook sre.wdqs.data-reload reloading wikidata_full on wdqs2023.codfw.wmnet from DumpsSource.HDFS (hdfs:///wmf/discovery/wdqs-reload-cookbook-test-T349069/ using stat1009.eqiad.wmnet))
[09:26:18] <stashbot>	 T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers - https://phabricator.wikimedia.org/T349069
[09:34:59] <gehel>	 dcausse: looking
[09:38:13] <gehel>	 most recent logs on cumin2002 are from June 4. Or I'm not looking at the right place
[09:38:19] <gehel>	 cd ..
[09:40:28] <gehel>	 found it!
[09:41:05] <gehel>	 https://www.irccloud.com/pastebin/i70TOXHy/
[09:46:10] <dcausse>	 thanks!
[10:02:28] <dcausse>	 lunch
[12:31:46] <dcausse>	 gehel: is this the full log? is there anything happening before this?
[12:33:17] <gehel>	 There is probably more 
[12:33:30] <gehel>	 I'll send it in 10'
[12:33:52] <dcausse>	 thanks!
[12:41:45] <inflatador>	 <o/
[12:43:41] <inflatador>	 dcausse if you want me to put the logs in your homedir on people or something like that LMK
[12:43:53] <dcausse>	 o/
[12:44:02] <dcausse>	 inflatador: yes that'd be great! :)
[12:44:43] <inflatador>	 dcausse ACK, will do. gehel ^^ I can take the log stuff from here
[12:45:14] <gehel>	 inflatador: thancks! I'll let you do that!
[12:50:28] <inflatador>	 dcausse I copied the logs to `wdqs-cookbook-logs` in your homedir on people1004. If you need any more LMK
[12:50:41] <dcausse>	 thanks!
[12:51:37] <inflatador>	 np
[12:56:14] <dcausse>	 inflatador: do you have logs that are more recent (from yesterday) these ones seem old
[12:58:28] <inflatador>	 dcausse oops, let me check
[13:01:52] <inflatador>	 dcausse I'm not sure why, but I'm not seeing that matches "wdqs" or "hdfs" that's newer than June 4
[13:02:22] <dcausse>	 inflatador: on cumin2002?
[13:02:30] <inflatador>	 dcausse correct
[13:02:56] <dcausse>	 gehel: found some part of them in https://www.irccloud.com/pastebin/i70TOXHy/ but not sure what's the location
[13:03:31] <gehel>	 in ryan's home dir
[13:12:47] <inflatador>	 in mtg but will pick it up once I'm out
[13:24:38] <inflatador>	 gehel might be late to 1x1, working w/Ben on helm chart
[13:24:51] <gehel>	 inflatador: ack
[13:24:54] <gehel>	 dcausse: 
[13:24:59] <gehel>	 https://www.irccloud.com/pastebin/kZDsuWWm/
[13:25:12] <dcausse>	 gehel: thanks!
[13:25:51] <dcausse>	 ah, seems to only log the last command?
[13:28:06] <dcausse>	 actually I want to see what happened before :)
[13:28:12] <dcausse>	 the full log would be great :)
[13:32:12] <gehel>	 dcausse: full log in your homedir on elastic1090
[13:32:22] <dcausse>	 thanks!
[13:40:40] <inflatador>	 gehel I ran out of time...have to take my son to camp ;(
[13:41:15] <gehel>	 ack, we can reschedule or see next week
[14:33:40] <inflatador>	 taking my other son to camp...back in ~30
[14:48:41] <dcausse>	 ryankemper: made further adjustments to the reload cookbook (https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1038904/23..24/cookbooks/sre/wdqs/data-reload.py), if you get a chance to make another attempt today, could you copy the full logs in e.g. people.eqiad.wmnet:~dcausse/? thanks! 
[14:57:32] <inflatador>	 back
[15:27:38] <dcausse>	 ebernhardson: fyi, ended-up creating https://gitlab.wikimedia.org/repos/search-platform/cirrus-rerender (copying most of the build system from the reindexer), somehow it felt wrong to add all this to cirrus-reindex-orchestrator 
[15:28:24] <dcausse>	 not super happy with the overhead that creates for such a small script, but not sure what other options we had
[15:28:59] <ebernhardson>	 dcausse: makes sense, and yea i had pondered shuffling the subdirs so it was more of a `cirrus` module, but then it starts to seem like there should be more of a shared architecture...i dunno it's probably better to keep them separate for now
[15:37:33] <pfischer>	 dr0ptp4kt: dcausse: should we schedule some time tomorrow? The RDF router is coming along, and so far I didn’t break any of the existing tests. However, I still have to write tests for the routing and resulting patches, which is going to be tedious. Also, ran into a few questions regarding the expected (stub) outcome.
[15:38:31] <dcausse>	 pfischer: sure, we have the pairing the session at 2pm UTC, might work?
[15:48:07] <dr0ptp4kt>	 if we can use that slot that works for me. lemme know if I should schedule a different slot though
[16:06:16] <ebernhardson>	 i keep saying the elastic percentiles grahs arent right ... i'm going to change the set of 4 graphs (p99+qps,p95,p75+qps,p50) into three the same everywhere (qps, p95, p50)
[16:20:06] <dcausse>	 thanks!
[16:21:06] <btullis>	 inflatador: Do we know what's up with elastic2088 and elastic2099 at the moment? They're down, but I don't know if that's expected.
[16:26:20] <inflatador>	 btullis no, that is not expected. Let me check
[16:28:23] <btullis>	 inflatador: Thanks. 
[16:29:45] <inflatador>	 btullis did you get an alert for these? I don't see anything yet
[16:31:03] <btullis>	 I did not get an alert. I only noticed because I was running some cumin commands against 'A:owner-search-platform' and these hosts timed out.
[16:31:09] <inflatador>	 ryankemper are you running the roll-reboot cookbook for codfw ? ^^
[16:31:34] <ryankemper>	 inflatador: no, codfw reboot's been done for a few days
[16:32:12] <inflatador>	 ryankemper ACK, will reboot both hosts via cookbook and get back
[16:32:25] <ryankemper>	 let's check what row they are, I think there was some sort of row E/F maintenance this week? I forget timing then
[16:32:40] <inflatador>	 nah, that was for eqiad
[16:32:59] <btullis>	 https://usercontent.irccloud-cdn.com/file/fvceGbP3/image.png
[16:34:49] <inflatador>	 would be cool if we had a website or MOTD that listed all the cookbooks in progress
[16:36:42] <inflatador>	 regardless, looks like the reboot cookbook doesn't work when SSH isn't up
[16:36:49] <inflatador>	 DRAC time ;)
[16:37:24] <volans>	 how would that help? that said is easy to make one, we have all the locks, I think I have a draft locally (and I can check them manually)
[16:37:39] <volans>	 inflatador: untrue
[16:38:01] <inflatador>	 volans see above, I have to check w/ others to make sure there's not a maintenance already going on
[16:38:03] <volans>	 or you mean the rolling reboot?
[16:38:41] <volans>	 inflatador: why? make the cookbook locks in a way that prevents you to run conflicting maintenances
[16:38:44] <volans>	 even better than checking a list
[16:38:48] <volans>	 that is error prone
[16:39:29] <inflatador>	 The use case is not about running conflicting maintenances, see above
[16:40:08] <ryankemper>	 Cookbookwise you'll want to use `./cookbooks/sre/hosts/reboot-single.py` btw
[16:40:37] <inflatador>	 as for the rebooting stuff, I'm running `sudo cookbook sre.hosts.reboot-single elastic2088.codfw.wmnet` and getting `cumin execution failed`  
[16:41:03] <inflatador>	 maybe I'm invoking it wrong? I tried `elastic2088*` and `elastic2088.codfw.wmnet`
[16:41:34] <volans>	 reboot-single tries a clean reboot from the host yes, not via mgmt
[16:42:08] <ryankemper>	 Ah scratch that then
[16:42:50] <volans>	 I don't think we have a simple reboot via mgmt cookbook, but we do reboot via mgmt on other cookbooks like reimage
[16:43:15] <inflatador>	 ACK. maybe a --force option or something? Not a huge deal
[16:44:16] <volans>	 yeah could make sense, feel free to propose a patch
[16:44:48] <inflatador>	 will do, thanks
[16:50:28] <ebernhardson>	 elastic percentiles dashboard updated: https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles
[16:50:39] <inflatador>	 cooooool
[16:52:57] <ebernhardson>	 a bit more consistent now, although i only changed the top percentiles section.  Somehow the other 2/3 of this percentiles dashboard isn't percentiles :P
[16:53:51] <inflatador>	 I think you mean 33.33% of the percentile dashboard IS percentiles ;P
[16:54:39] <ebernhardson>	 lol
[16:55:31] <inflatador>	 2088 is back up. Still not sure why I didn't get an email or ping in IRC (I did get a ping for it coming back up)
[16:55:46] <ebernhardson>	 i did get an instance not indexing alert for something, lemme check
[16:56:21] <ebernhardson>	 huh, but not for these, weird
[16:56:30] <inflatador>	 probably for the host we banned for the switch maintenance this morning
[16:56:47] <ebernhardson>	 is it because they stopped submitting metrics all together, intead of submitting 0's?  might want to change some alerts
[16:57:08] <ebernhardson>	 or maybe not:P i dunno
[16:57:09] <inflatador>	 they appeared as down in icinga, so not sure yet
[17:01:12] <ebernhardson>	 unrelated curious thing. I merged the patch to remove zk network bits from SUP helmfile, since admin_ng can now provide that functionality.  The apply clearly showed changes, but it didn't restart any pods. I guess network changes get applied without restarts?
[17:01:53] * ebernhardson will probably force a restart anyways, to be sure
[17:02:36] <inflatador>	 Yeah, I think the network policies just manipulate iptables/ipvs
[17:05:46] <ebernhardson>	 makes sense
[17:06:26] <inflatador>	 thanks for doing that regardless, I probably should've done it already
[17:06:52] <ebernhardson>	 i was looking through open patches, this has apparently been waiting almost a month now. Seemed like something worth finishing
[17:09:53] <dcausse>	 thanks!
[17:10:13] <ebernhardson>	 lol, promising error message from gitlab cu: fatal: unable to access 'https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater.git/': Could not resolve host: gitlab.wikimedia.org
[17:10:29] <dcausse>	 :)
[17:11:03] <inflatador>	 oopsie
[17:12:36] <dcausse>	 speaking about old patches I guess we could apply https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/966902 too
[17:12:41] <dcausse>	 inflatador: ^
[17:13:25] <inflatador>	 dcausse ACK, I see that one on my gerrit page and then..I keep ignoring it ;(
[17:14:01] <dcausse>	 np! should be safe to merge, nothing runs there
[17:14:22] <inflatador>	 I'll merge after I finish this phab ticket, thanks for the reminder
[17:17:28] <ebernhardson>	 made me randomly curious about the oldest open patches, as i should probably abandon a bunch of things to make the gerrit generic reports more useful.  Apparently the oldest open and not-WIP patch is Sept 2014. https://gerrit.wikimedia.org/r/q/status:open,7100
[17:43:07] <ebernhardson>	 hmm, where might SRE have seen all the requests coming from SUP and noticed they didn't have a user agent? About to deploy the fix to provide http agents but wondering if i can see it fixed in any dashboard or report
[17:44:45] <ebernhardson>	 claime: You noticed on 5-31 we were sending some requests without a user agent, i'm about to ship a fix and wondering where you were seeing them so i can verify the fix?
[17:54:10] <dcausse>	 dinner
[18:00:04] <inflatador>	 OK, merged the flink operator patches in dse-k8s
[18:00:24] <inflatador>	 now I eat! Back in ~40
[18:00:38] <inflatador>	 err.r...back in time for pairing in :30
[18:41:48] <ebernhardson>	 inflatador: https://phabricator.wikimedia.org/T366363
[19:20:47] <ebernhardson>	 user agent update looks to have worked, can see the apache http user agent replaced with our custom agent
[22:01:33] * ebernhardson avoids temptation to write a tiny expression language for blazegraph throttling via headers.
[22:30:18] <inflatador>	 {◕ ◡ ◕}
[22:43:34] * ebernhardson didn't avoid the temptation :P But i made it way simpler than my first idea