[10:53:16] Hmm [10:53:31] I'll rebase all of them on master, my bad [11:03:55] nemo-yiannis: Done, I can merge it as an apology for the false start :P [11:04:04] ahaha, no worries [11:04:14] feel free to go ahead if you deploy other stuff, i can do it later if not [11:04:30] No I'm not deploying anything rn, so feel free to do it at your convenience [11:18:59] ok [17:01:19] hi folks. I am trying to debug https://logstash.wikimedia.org/goto/51df7c09287c8b0a6a31f17c12f0b14e for hcaptcha folks. tl;dr is that starting August 28 around midnight UTC and continuing till around ~18:00 UTC (?) on August 31, there was an increase in this error: [17:01:29] > POST https://api.hcaptcha.com/siteverify HTTP/1.1 - NULL cURL error 28: Resolving timed out after 1000 milliseconds (see https://curl.se/libcurl/c/libcurl-errors.html) for https://api.hcaptcha.com/siteverify [17:01:55] the errot itself is fairly self-explanatory in that it seems like DNS quries were timing out, taking more than 1s, which of course is too long [17:02:12] was there any DNS related changes during this period on k8s or the related realm? [17:02:31] (I do not understand this world and therefore the question) [17:03:00] this error then itself resolved itself, so there was no change I am aware of that resolved this [17:03:56] happy to file a task on phabricator if it is better but yeah, I thought I will start here and see if there was an obvious reason [17:46:55] hmmmmmmm did we point MW DNS to the internal resolver maybe? [17:56:06] no I don't think so -- definitely shouldn't have, and we'd have seen a lot more breakage, but if it happened that should be a quick error and not a timeout [17:57:19] can't rule out some of j.ayme's other coredns work but I don't think the timing lines up in any obvious way [18:42:42] my go-to "is core dns sad" detector is EtcdConfig fetch errors: https://logstash.wikimedia.org/goto/9d9dbc80295efd729be84151fea598e5 [18:43:32] yeah, looks like things were sad between the 28th and late on the 31st, with a recurrence perhaps ongoing [18:43:54] ... aaaand upstream resolver health checks, per-pod: https://grafana.wikimedia.org/goto/sghx5c?orgId=default [18:44:18] swfrench-wmf: that certainly matches the timing, yep [18:44:34] looks like there might be two or three pods consistently acting up [18:45:31] we still don't understand this behavior well, and generally our solution is to delete the pods =/ [18:45:48] I can do that shortly [18:46:05] no worries swfrench-wmf. I am also happy writing it up if that helps. but for now, the issue seems to have resolved fwiw. [18:46:08] (itself) [18:46:41] I see the early symptoms of it coming back, so I might just go ahead and take of it right now [18:47:04] ... and advocate separately for getting to the bottom of this [18:47:15] thanks <3 [18:47:46] it's very rare that I see an error on MW/logstash that I can actually understand. "DNS query timing out because it took more than a second, that seems fair" :) [18:48:44] ... hunh [18:53:10] the fact that we often see this develop as correlated behavior (i.e., multiple pods) suggests an external trigger - e.g., a query workload (more likely) or perhaps something upstream [18:54:03] or a shared node or rack maybe? [18:56:14] (can't imagine why that would be, coredns isn't a daemonset or anything like that, but easy to check) [18:58:32] oh *dns* pods, I thought we were talking about correlated behavior between MW pods. okay less likely [18:59:05] ah, yeah these were coredns pods - in C3, D8, and C6 this time [18:59:12] *on workers in [18:59:30] 👍