[11:21:23] sigh... these "Executor reports task instance finished (failed) although the task says its queued. Was the task killed externally?" airflow failures do not sound good :/ [11:21:50] seems like a race that has not been solved yet (https://github.com/apache/airflow/issues/10790) [12:17:01] lunch [15:36:29] sorry, been here for awhile [15:36:40] Have a simple PR if anyone has a chance to take a look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/868418 [15:42:21] o/ [15:59:26] thanks for the review dcausse ! [15:59:46] yw! [16:27:17] one more PR if you or anyone else has time ;) https://gerrit.wikimedia.org/r/c/operations/puppet/+/868440 [16:27:30] this one should open the FW [17:04:39] errand [17:57:27] inflatador, ryankemper: do we have an incident report for this Monday WDQS outage? [18:01:08] gehel I haven't done one, ryankemper have you worked on it? If not we can probably do it at pairing session or afterwards. Will be very similar to https://wikitech.wikimedia.org/wiki/Incidents/2022-11-22_wdqs_outage [18:11:13] lunch, probably back in time for pairing but may be a little late [18:22:16] No I didn’t make one [18:22:53] There is a google doc with a brief timeline so we can use that when making it [18:23:22] https://docs.google.com/document/d/1Ypnu5_NtQZH0rcMLGIuvVpU2NFzvDQslWq9WjNr4uGo/edit [18:23:38] it definitely does not need to be much, but having a trace of the problems we had would be nice [19:17:32] back in time for pairing! [19:17:49] and full of pozole...you guys might have to keep me awake [21:32:06] not feeling so hot...taking a 30m break [22:19:16] Feeling gross, gonna take the rest of the day off. See you tomorrow!