[10:22:53] <dcausse>	 errand+lunch
[12:38:55] <claime>	 Do cirrusSearch jobs hit parsoid?
[13:08:36] <inflatador>	 <o/
[13:09:42] <inflatador>	 claime not that I know of... gehel dcausse any thoughts on ^^ ?
[13:09:43] <claime>	 nevermind, it's something else
[13:09:46] <claime>	 heh
[13:09:57] <claime>	 It's a big commons pagebundle reparse
[13:10:48] <claime>	 which would probably explain the time correlation with a bump in cirrusSearch jobs at about the same time
[13:13:21] <inflatador>	 no problema, good luck!
[13:59:51] <dcausse>	 o/
[14:00:08] <dcausse>	 claime: sorry I'm late, no, CirrusSearch still uses the old php parser
[14:00:52] <claime>	 dcausse: ack, ty
[15:06:59] <ebernhardson>	 \o
[15:08:36] <inflatador>	 <o/
[15:09:15] <inflatador>	 according to #wikimedia-sre , flink in eqiad was accidentally depooled for a short time...still checking dashboards, but it doesn't seem to have affected anything
[15:10:18] <ebernhardson>	 kk
[15:18:33] <dcausse>	 o/
[15:24:21] <inflatador>	 trying to dig up elastic settings checks...looks like they exist per https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp#25 , but I can't find that file on a prod codfw host
[15:25:37] <gehel>	 inflatador: settings are global to the cluster, so they are not checked on every host. I think we check on all master eligible nodes.
[15:27:21] <inflatador>	 gehel ACK, looks like you're correct (as usual ;P )...I see them on a CODFW master host
[15:31:53] <inflatador>	 we need to figure out how to do this with blackbox/remote polling as well...but for now, will just use icinga
[16:15:01] <inflatador>	 workout, back in ~40
[16:41:02] <dcausse>	 errand
[17:31:31] <inflatador>	 OK, so the icinga check is going to get nasty...wondering if we could maybe just run a systemd timer to set Elastic running config to the rendered config in cirrus_settings_check.yml every hour/day/whatever?
[17:43:26] <inflatador>	 I guess we could render the health checks dynamically thru puppet a la https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/icinga/manifests/monitor/elasticsearch/cirrus_settings_check.pp#9 . Looks nasty though
[17:43:37] <ebernhardson>	 hmm
[17:44:00] <ebernhardson>	 there generally wouldn't be any harm just applying the config every hour. 
[17:44:21] <ebernhardson>	 I don't know if elastic is smart enough to throw out cluster state updates that don't do anything, but even if it didn't...it's once an hour
[17:46:08] <inflatador>	 Yeah...I don't like either option. I guess we're going to have to move this logic out of icinga anyway since it's going away. hmm...
[17:46:28] <inflatador>	 would be nice to have a source of truth outside of a config mgmt repo, but of course I don't have any helpful suggestions ;)
[17:46:39] <ebernhardson>	 nmap? :P
[17:46:54] <inflatador>	 packets don't lie!
[17:47:18] <ebernhardson>	 lol
[17:48:20] <inflatador>	 anyway, grabbing a quick lunch, back in time for pairiing
[18:04:35] <dcausse>	 dinner
[18:32:21] * ebernhardson is mildly surprised a java instant can represent year -1 billion
[18:45:35] <inflatador>	 back
[19:24:54] <inflatador>	 I'm hitting _cluster/settings on the Elastic API. Can I give it a query string so it only gives back the CCS settings, or do I have to filter that client-side?
[19:25:58] <inflatador>	 hmm, looks like client-side https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cluster-get-settings.html
[19:33:48] <inflatador>	 ryankemper gehel SRE pairing if y'all wanna join
[20:44:04] <inflatador>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1012703 (CCS monitoring) is ready...this is just for the short-term fix
[21:12:51] <inflatador>	 another small change to turn off alerts for a host that's about to be decommed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1012751
[22:13:02] <inflatador>	 thanks for the reviews! Heading out for the day...
[22:30:45] <pfischer>	 ebernhardson: are you around? Would you have a moment to help me with airflow/discolytics (T358472)?
[22:30:46] <stashbot>	 T358472: Search dag image_suggestions_weekly failed with: Empty dataframe provided - https://phabricator.wikimedia.org/T358472
[22:31:00] <ebernhardson>	 sure
[22:31:25] <ebernhardson>	 pfischer: certainly, whats up?
[22:32:18] <pfischer>	 meet.google.com/qhu-iezx-bab