[11:08:33] <dcausse>	 lunch
[11:36:28] <gehel>	 lunch 2
[13:57:06] <gehel>	 dcausse: just to check, will you be there for the Mobile search schema meeting in 1.5h? You have not responded to the invite
[13:57:28] <dcausse>	 gehel: yes I think I'll make it, will ack
[13:57:34] <gehel>	 thanks!
[14:04:03] <inflatador>	 <o/
[14:04:14] <dcausse>	 o/
[14:24:25] <inflatador>	 The data reloads on wdqs1009 and wdqs2009 might be stuck? Nothing logged in spicerack since 2023-01-08 00:43:18
[14:25:00] <dcausse>	 looking
[14:25:59] <dcausse>	 wdqs1009 is importing file:///srv/wdqs/munged/wikidump-000000251.ttl.gz
[14:26:06] <dcausse>	 can't log in wdqs2009
[14:26:48] <inflatador>	 dcausse thanks, was stracing on 1009
[14:28:00] <inflatador>	 will check on wdqs2009, I can't get in either
[14:37:12] <inflatador>	 dunno what to make of wdqs2009 so far. No SSH, but it does ping. mgmt console unresponsive, but NFS traffic is going back and forth between it and clouddumps.
[14:38:04] <dcausse>	 I vaguely remember seeing few alerts flapping on the new codfw hosts, ssh was one of them
[14:50:51] <inflatador>	 I think let's wait until the munge is finished on 1009 and check it again, since NFS traffic still seems to be flowing
[14:51:51] <dcausse>	 you mean 2009? wdqs1009 is done with the munging as it's importing to blazegraph now
[14:52:52] <inflatador>	 dcausse sorry, just saw the word 'munge'. Should have said wait until the reload is complete on wdqs1009
[14:53:23] <dcausse>	 ok
[14:58:34] <inflatador>	 nm, I'm gonna reboot it. No log messages on centrallog for more than 24h
[15:03:51] <inflatador>	 OK, host is rebooted, SSH back up, logs coming in to centrallog
[15:04:04] <inflatador>	 just started reload again
[16:01:28] <gehel>	 dcausse, pfischer: triage meeting: https://meet.google.com/eki-rafx-cxi
[17:01:08] <ebernhardson>	 hmm, i guess first is to check into what all these new airflow complaints are about
[17:10:13] <dcausse>	 ebernhardson: all refinery jobs stopped during the week end
[17:10:47] <ebernhardson>	 ahh, that would explain it. yea poking at the failed task instances seemed like you already cleared it up
[17:10:54] <dcausse>	 I tried to clear the failed ones that I could identify
[17:11:27] <ebernhardson>	 imo easiest way to find failed task instances is from browse->task instances, set a filter on state failed and sort by date
[17:11:45] <dcausse>	 oh
[17:12:16] <dcausse>	 indeed that's better than the dag view, esp on hourly jobs
[17:12:50] <dcausse>	 they just finished the backfill apparently so things will start moving again hopefully
[17:13:04] <ebernhardson>	 i think you only missed one, i saw one in there but everything else is from november or earlier
[17:13:25] <ebernhardson>	 did the elastic plugins get shipped since i was out, for the LTR feature collection bug?
[17:13:35] <dcausse>	 no I don't think so
[17:13:42] <inflatador>	 ebernhardson no, it's on our radar though
[17:13:49] <ebernhardson>	 ok
[17:14:02] <inflatador>	 we can probably finish that in the next day or so
[18:42:21] <gehel>	 ryankemper: is there anything urgent we need to discuss in our 1:1? I'm not feeling super well, so could we reschedule for tomorrow?
[18:44:37] <inflatador>	  lunch , back in ~1h
[18:56:21] <ryankemper>	 gehel: that's fine with me!
[19:01:32] <gehel>	 Thanks!
[19:05:47] <dcausse>	 dinner
[19:09:24] <ebernhardson>	  Error: Class 'Onoi\MessageReporter\ObservableMessageReporter' not found :S
[19:14:10] <ebernhardson>	 seems otherrs have simply been ignoring CI in REL1_39 to get Cirrus patches through, antoine poked it a little in november but doesn't seem to have gone anywhere. 
[19:18:53] <ebernhardson>	 seems that while onoi/message-reporter is in master branch, it's not in the REL1_39 vendor repo.  Probably just add it?
[19:40:46] <inflatador>	 back
[19:45:21] <inflatador>	 ryankemper let's work on https://phabricator.wikimedia.org/T324247 during our pairing if that is OK
[21:05:31] <inflatador>	 ^^ New LTR plugin is packaged. It's currently deployed in relforge and cloudelastic is in progress
[21:21:04] <inflatador>	 quick break, back in ~15
[21:42:04] <ebernhardson>	 \o/ thanks
[21:45:06] <inflatador>	 back . Cloudelastic is finished, starting CODFW shortly
[23:01:28] <inflatador>	 ryankemper about to head out, are you able to keep an eye on the codfw restarts? It's up in my user's tmux window on cumin1001
[23:01:48] <inflatador>	 window is called 'codfw-ltr-plugin'
[23:13:06] <inflatador>	 we went red for a second on chi and omega in codfw, but it recovered
[23:14:55] <ryankemper>	 inflatador: watching now
[23:15:30] <inflatador>	 ryankemper ACK , thanks