[12:47:58] dns5004 has sometimes timed out its dns and NTP service in the last hour :-/ [12:48:51] thanks I'll have a look [12:50:08] tcp/attemptfails potentially high? https://grafana.wikimedia.org/goto/_TeDnw2Sz?orgId=1 [12:51:23] or maybe it is something UDP-only? [12:51:26] mmm interesting, potentially related to another issue had yesterday with a huge number of requests (cacheable) from single ips [12:53:09] is it a vm? [12:53:46] baremetal [12:54:42] s/ips/ip/ [12:57:08] I don't see anything else super-obvious, the host seems to have generally low load [12:59:44] calm day today, so far, denisse [13:00:57] Thanks jynus . I see some emails from Arelion regarding a fiber cable cut. [13:34:28] puppet is failing to run on idp1002, that's me. I'm on it. [13:41:33] fixed [13:47:56] I made a change in one of the debian-installer scripts, I'm going to test it but let me know if you're having issues during re-images - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1003464 [14:01:29] looks like the test is successful ! (cc moritzm) [14:03:21] nice! [19:36:26] Hello, sre-collab! wmf-debci is causing traffic a little bit of confusion: It seems that the CI process has changed and we're unable to get it working again. Emperor was kind enough to fix up https://gitlab.wikimedia.org/repos/sre/dnsdist/-/ci/editor some time ago but that's failing right now, as are our simpler jobs: [19:36:28] https://gitlab.wikimedia.org/repos/sre/pybal/-/blob/bullseye-wikimedia/.gitlab-ci.yml?ref_type=heads [19:39:58] sre-collab: I'm also interested in proposing the idea of separating the debian packaging from codebase repos, much like how arch/alpine/fedora et al do it - combining the two is kinda difficult to manage [19:40:45] I know that this is following what Debian salsa is doing - and it might be better to keep that consistency - but dang if it isn't awkward to keep cherry-picking everything around [19:41:55] An example repo just for building logic: https://git.alpinelinux.org/aports/tree/community/sway?h=master or https://src.fedoraproject.org/rpms/sway/tree/rawhide [19:44:35] sre-collab: We could continue using the per-distro branching strategy for building logic in each Debian release - but the dgit CI/actual code would reside in its own repo so that the dgit CI/actual code management could be managed separately [19:46:23] it's possible that my brain is too smooth and I need to just git gud with Debian-land [19:48:59] brett: would you mind pasting the above into a phab task and tagging us, we'll take a look at it [19:49:14] sobanski: Sure! Thank you for doing that :)