[11:26:38] Is codesearch.wmcloud.org down for anyone else? I see some spikes and missing data here, but I'm not currently in the right project to look in any more detail: https://nagf.toolforge.org/?project=codesearch [12:08:13] It failed (timeout) once, it worked another [12:08:15] (for me) [12:09:34] I think that Amir1: is looking into it as we speak. I'm happy to chip in if that will help though. [12:09:48] 👍 it seemed to get out of memory a few times [12:10:42] yeah, I restarted it [12:11:11] I'm seeing errors of failing to load git repos but nothing major [12:11:31] the memory seems decent now [12:11:39] Ack, thanks. [12:11:41] thanks! [12:35:26] weird, we haven't gotten any vendor maintenance update since the 18? It is just thanksgiving or something bad on our side? [12:35:50] I will send a test email, just in case [12:38:07] email is working so shouldn't be that [16:32:04] there is a scheduled incident ritual in aprox 30 minutes, I recommend people attending it [16:32:16] (it should be on your calendars) [16:38:47] Lots of work for mr or ms nan this week: https://wikitech.wikimedia.org/wiki/SRE/Oncall/Schedule ! [16:38:58] trying to find who should be there [16:39:53] it should be replaced by NaP (Not a Person) [16:39:55] I am guessing that is denisse [16:39:56] :-P [16:40:13] Mr Null is kinda busy with clinic duty as well [16:46:32] I guess on clinic duty "Dev Null" should be translated to "No backup clinician" [16:46:54] or just drop the column entirely [16:48:43] it comes from this "backup/shadow" shift: https://usercontent.irccloud-cdn.com/file/9ZBtw3YK/Screen%20Shot%202022-11-21%20at%2011.48.04%20AM.png [16:48:59] which yeah, we could just delete [16:50:34] would require some clinic-specific logic but the single clinic duty column could just say "Firstname Lastname (backup: Firstname Lastname)" when there's actually a backup scheduled, and omit it otherwise [16:51:14] plenty of good solutions though :) the nan thing is more important probably [16:52:16] that could work, we could simply set an override in the event of a backup/shadow and call that good enough for the clinic duty roster [16:54:51] yeah let's just do that, and we can always make it more complicated if need be [16:54:59] oh I didn't even mean change anything on the VO side, just render it differently on wikitech [17:01:15] its too late! so long dev null. the simple override really makes more sense, since we would have to override dev null anyway to set a backup [17:10:44] jynus: Yes, I'm on clinic duty this week but today it's a public holiday in Mexico. :) [17:49:50] no worries, denisse|m [17:50:05] this was about the schedule automation having issues [17:50:29] take your well desrved rest! [18:59:55] _joe_: > as for build cache, there's a command line option to build [19:00:18] i don't see it? [19:01:36] ah i think i see it in the source, but not in the --help message [19:02:02] OH yes i do, need, --help after build subcommand [19:02:03] got it. [19:02:03] ty [19:14:44] is there a way to force a rebuild of an image in development without rming the image and without bumping the version in the changelog? [22:25:49] anyone else having phab connection issues? [22:26:08] ah nm, I see the ping in operations [22:40:18] inflatador: is back now [22:41:55] ACK, thanks RhinosF1 ! [22:42:07] we are still working on some issues but in general it's back [22:43:41] It looks good for me [22:43:56] I'm going to sleep now before fire [22:45:19] there are some yellow exclamation marks about some issues [22:45:24] but you only see them as admin I think [22:45:29] we are debugging those [22:49:46] diffusion also seems to be having issues still, from CI cloning: fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/': The requested URL returned error: 500 [22:51:34] mutante: ^ [22:51:50] ebernhardson: [22:51:50] Command failed with error #1! COMMAND /usr/bin/sudo -E -n -u phd -- git ls-tree -z -l 9665dd9ab7287d80c9c0c3752cf99e8e841d0d10 -- STDOUT (empty) STDERR sudo: a password is required [22:52:01] Maybe a missing / broken sudo rule [22:52:14] Or group [22:52:42] I sleep time 3 [23:11:12] ebernhardson: we fixed an issue related to that, hope it's gone [23:11:43] ok, trying CI again [23:14:03] mutante: nope, still failing clones from diffusion (same manually cloning from here)