[00:00:34] Hmm, working again now [07:07:26] krinkle@mwlog1002:/srv/mw-log/archive$ head JobEx [07:08:02] JobExecutorapi.log-20231206.gz JobExecutorapi.log-20231206.gz JobExecutoxff.log-20231128.gz JobExexff.log-20231030.gz JobExxff.log-20231130.gz [07:08:15] there are dozens of seemingly corrupt filenames there [07:08:22] containing entries in an unrelated channel [07:08:42] well, not entirely unrelated. E.g. JobExecutorapi.log-20231206.gz contains an event that was clearly destined for api.log [07:09:08] Have other people noticed this? Some kind of regex gone wrong? [07:13:08] Hm.. and the dates in the filenames seem to be off by one. [07:13:23] inflatador: I use thelounge as my self hosted bouncer [07:13:46] archive/JobExecutor.log-20231211.gz contains 2023-12-10 and JobExecutor.log-20231212 contains 2023-12-11 [07:19:23] Krinkle: o/ I'd suggest to open a task so people can investigate [07:20:16] was there an outage around 12:00 UTC for k8s mw appservers? [07:21:39] https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&var-dc=codfw%20prometheus%2Fk8s&var-service=mediawiki&var-namespace=mw-api-ext&from=1702338100787&to=1702341453508 [08:27:54] {{done}}, ref T353220, T353221 [08:27:54] T353220: mwlog/archive sends some events to corrupted file names - https://phabricator.wikimedia.org/T353220 [08:27:55] T353221: mwlog/achive dates in filenames are off-by-one - https://phabricator.wikimedia.org/T353221 [08:58:22] fyi, I deployed the change to not require a homer commit to add BGP peers (k8s, dns, lvs), doing a full check now to make sure it behaves as expected [09:06:42] yay, thanks! [09:21:37] yay, virtually a NOOP [15:26:12] moritzm the deb pkg build pipeline isn't in prod yet, is it? [15:26:36] <_joe_> the building part is [15:26:52] <_joe_> I'd ask eoghan about importing it automatically into apt [15:27:25] Excellent, thanks _joe_ . Reading https://wikitech.wikimedia.org/wiki/Debian_packaging now [15:44:48] Emperor for the `dgit` commands in https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI , are you running those from your local machine? [15:58:49] I guess these instructions are for CI config as opposed to manual building [16:17:03] inflatador: yes, run them locally [16:17:31] (or on whatever machine you want to be doing Debian packaging on) [16:23:32] Emperor OK, thanks for clarifying...I didn't see dgit installed anywhere (checked debmonitor) so I figured it would be in CI [16:25:54] inflatador: CI uses dgit for some operations (basically, checking for updates to a package from Debian)