[00:23:24] <urandom>	 mutante: oh, cool, thank you
[10:11:38] <jynus>	 I am guessing install2004 is a new/upgrade host that is being setup, right?
[10:43:28] <volans>	 Emperor: to answer your question of input redirection you can just do the subshell on the cumin host with the right quotes and have them "inlined": "echo '$(cat /path/to/patterns)' | grep -f - /var/log/file_to_grep"
[10:52:20] <moritzm>	 jynus: yes, I applied the role to it earlier the day, previously it was installed with an insetup role
[10:52:45] <jynus>	 thanks, no problem, it just showed up in bacula as a new host
[10:57:01] <moritzm>	 there'll be five more coming as well
[11:52:05] <Emperor>	 volans: thanks. I ended up with the snappy and memorable: cumin -x --force --no-progress --no-color -o txt O:swift::proxy 'zgrep -hFf <(curl -s https://people.wikimedia.org/~mvernon/all_sadobjects) /var/log/swift/proxy-access.lo*' 2>/dev/null | sort -b -k 11,11 -k 3n,3 -k 4,4 >allsad_output
[18:12:02] <sukhe>	 has anyone attempted to update the iDRAC on the R440 on a very old version? (3.15)
[18:12:12] <sukhe>	 (iDRAC firmware)
[18:12:22] <sukhe>	 trying to do an update from 3.15 to 6.10 and it's failing with:
[18:12:24] <sukhe>	 "RED008: Unable to extract payloads from Update Package."
[18:12:30] <sukhe>	 wondering if I should do an intermittent one perhaps
[18:12:33] <sukhe>	 like a 5.x
[18:13:42] <mutante>	 sukhe: seems likely that you need to go to 3.30 first
[18:13:44] <mutante>	 "We noticed that we had to do a manual, intermediate update with iDRAC9 in order to update to 4.00 if the existing version was old enough.  If you manually update to 3.30, then it can update to 4.00 without those issues."
[18:13:53] <sukhe>	 oh thanks!
[18:14:06] <sukhe>	 mutante: where is this from?
[18:14:26] <mutante>	 but also mention it in -dcops channel. I think they do most of these so have the most experience
[18:14:29] <mutante>	 sukhe: https://www.dell.com/community/Dell-OpenManage-Enterprise/Message-Unable-to-extract-payloads-from-Update-Package/td-p/7498144
[18:14:36] <sukhe>	 yeah, I should ask there
[18:14:37] <sukhe>	 thanks!
[18:14:40] <mutante>	 yw
[18:15:10] <sukhe>	 if anyone is interested in this, posted to the -dcops channel
[19:29:47] <urandom>	 Does anyone have any thoughts on https://phabricator.wikimedia.org/T327675?  Basically, on (alternately) de-pooling datacenters for sessionstore in order to perform Cassandra restarts?  My assumption is that it will mean some minutes of elevated latency for some requests (elevated by whatever it takes to go between DCs), and nothing else.  I want to make sure I'm not missing anything though.
[19:46:06] <rzl>	 hmm, I was going to say you could maybe coordinate it along with the DC switchover to minimize overall disruption, but if the certs have less than a month left, that won't be possible
[19:46:50] <cdanis>	 urandom: that seems reasonable enough to me. eqiad<>codfw one way is nominally 36ms
[19:46:58] <cdanis>	 https://wikitech.wikimedia.org/wiki/Network_design#/media/File:Wikimedia_network_overview.png
[19:48:21] <cdanis>	 so that will add a fair bit of latency, given the p95 for sessionstore is ... about 2.5ms for reads and about 10ms for writes?
[19:48:42] <urandom>	 wait, 10ms?
[19:49:17] <cdanis>	 that was my guessing based upon https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1&viewPanel=52&from=now-6h&to=now
[19:49:49] <cdanis>	 most of the time writes are <=2.5ms but there are deviations from that
[19:50:02] <urandom>	 yeah, ok
[19:50:26] <urandom>	 I wonder what's making the distribution wider there... 🤔
[19:50:57] <urandom>	 anyway... yeah, it would make it look closer to the DELETE latency (precisely because it uses a quorum that includes the other DC)
[19:52:08] <urandom>	 ...and I think the latency distribution is so low on reads because it's all in-memory (now that I think about it)
[19:55:17] <cdanis>	 makes sense
[19:56:18] <urandom>	 I have a hunch these restarts wouldn't create an outage at all...I'm starting to think it happens when the host is restarted