[08:08:04] hello folks! [08:08:26] As FYI the docker-reporter failures on build2001 are due to a bug in debmonitor that I have introduced, the fix is going to happen soon-ish [08:27:42] elukey: would you happen to know what /srv/.hardsync.published.jqCCxZoo4mGJ/datasets/one-off/caption_competition/training is (on an-web1001) ? It looks like a ML dataset [08:27:54] it's taking 30% of the disk space, which is hovering at 99% used [08:29:51] o/ [08:30:15] it seems a leftover of the script that aggregates the various published dirs on stat100x if I have to guess [08:30:35] I checked and /srv/published/datasets/one-off on all the stat100x doesn't list "caption_competition", so maybe it was already removed [08:32:25] brouberol: yeah I don't see it in here too https://analytics.wikimedia.org/published/datasets/one-off/ [08:34:02] Thanks! I'm not seeing anything in codesearch either [08:34:34] Who do we know who could give me the thumbs{up,down} to delete it, if it's no longer needed, according to you? [09:08:07] <_joe_> brouberol: who owns the file? [09:08:24] stats:wikiedev [09:08:34] *wikidev [09:08:36] <_joe_> oh nice, how helpful [09:08:37] <_joe_> :D [09:08:40] no user in particular on the an-web host [09:08:51] <_joe_> so next up.. is it opened by a process? [09:09:15] <_joe_> if not, next thing you can do is look at the crons for the stats user maybe [09:09:34] it is not, AFAICS. I think it's up for being served by the webserver. I've reached out to ML and research [09:12:21] they seem to be pretty old files apparently [09:12:21] Birth: 2021-09-22 02:44:41.565769980 +0000 [09:12:36] Although it does not mean that they're not valuable in some way [09:57:43] * fabfur afk for a little [11:24:06] * fabfur lunch [11:37:10] marostegui, fabfur, _joe_: my patch finally passed CI, I'm ready to deploy if it's ok with you. [11:37:27] <_joe_> duesen: go on, our window finished in the meantime :D [11:37:31] duesen: go for it [11:37:37] <_joe_> duesen: 40 minutes for a merge? [11:37:45] <_joe_> wow [11:38:13] no, 20. got a CI failure that i need to fix. just a silly thing. [11:50:15] go for me [14:23:07] * fabfur afk [15:54:23] * fabfur off [17:03:30] !oncall [17:04:23] !oncall-now [17:04:23] Oncall now for team SRE, rotation business_hours: [17:04:23] a.rnoldokoth, c.white [17:04:36] maybe that should be an alias :) [17:05:09] lol thanks i never do get it right [19:05:26] mutante you around? Just wondering if the hostname is automatically added to y'all's ProbeDown tickets (such as T352084 ) or if it's necessary to add the hostname yourself [19:05:27] T352084: ProbeDown miscweb1003 - query.wikidata.org/bigdata/ldf - https://phabricator.wikimedia.org/T352084 [19:07:11] inflatador: https://phabricator.wikimedia.org/T356140 :/ [19:07:33] TLDR: manual but would be awesome if it was automatic [19:08:13] the instance name is already in the body of the message, not in the title though [19:08:46] last time I looked it wasn't as trival as just adding a variable to a string though [19:09:17] mutante ah, I see you're feeling the same pain ;) [19:09:50] feel free to upvote my ticket:) [19:10:44] despite the manual work we still like having the automatic tickets though. we found quite a few things that would have gone unnoticed otherwise [19:10:44] How do I do that? Award token? [19:11:02] yeah, I'd like to try the solution suggested by go-dog in the ticket [19:11:09] yea, token and just subscribing and commenting that you would like it [19:11:49] or mentioning it on observability chanel [19:12:23] yeah, let me do a little recon on this...curious to see if the dc ops stuff mentioned is in alerts repo or puppet repo [19:14:03] I also need to detune some of our probe alerts, they're a bit sensitive ATM [19:14:11] cool!:) [19:16:44] * inflatador went ahead and asked in #observability