[04:47:17] !log library-upgrader rebooting libup-web02 after I fixed the full root partition :/ [04:47:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [04:51:09] !log library-upgrader rebooting libup-db02, unsure what's wrong [04:51:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [04:55:00] it's aliveeee [07:18:42] We noticed that one of our toolforge tools isn’t displaying some fonts, probably after switch the docker image to php7.4. Is there any way to update the missing fonts without building an image from scratch, as that doesn’t seem to be supported yet? [07:26:24] what exactly are you trying to do? usually, for web apps, the fonts come from the system that the browser is running on and not from the server [08:42:26] I assume the languages for each image are included to be able to display in all required languages. We have a bug where Korean doesn't get displayed correctly.https://phabricator.wikimedia.org/T310435 [08:56:48] We were only guessing that this happened because different images include different font packages. [09:16:23] Hey all, PAWS does not work for me, it just doesn't load in the browser. Is this a problem only for me, or a larger one affecting everyone? [09:26:59] let me take a look [09:27:31] I think it might be https://phabricator.wikimedia.org/T319366 [09:28:40] at least the public side seems to work [09:28:46] https://public.paws.wmcloud.org/ [09:30:20] * dcaro it seems to timeout loading for me (https://hub.paws.wmcloud.org, after redirecting to login) [09:30:46] same here [09:54:47] !log paws removing labstore mounts on control-2 [09:54:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:56:16] !log paws removing labstore mounts on control-3 [09:56:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:57:41] !log paws removing labstore mounts on acme-chief [09:57:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:58:55] !log paws removing labstore mounts on ingress-3 [09:58:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:00:09] !log paws removing labstore mounts on ingress-4 [10:00:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:00:26] !log paws note that ingress were not mounting labstore already [10:00:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:01:35] !log paws checking db-backup for labstore mounts (it should have according to hiera) [10:01:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:02:42] !log paws checking the prometheus instance for mounts from labstore [10:02:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:18:21] !log paws aborrero@paws-k8s-control-1:~$ sudo -i kubectl -n prod rollout restart deployment/proxy (T319366) [10:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:18:24] T319366: PawsJupyterHubDown alert flapping - https://phabricator.wikimedia.org/T319366 [10:20:52] MisterSynergy: can you verify if now it works for you? [10:40:35] !log paws extended the /srv volumes for both prometheus nodes to 15G to give some space for prometheus to shuffle data (the tsdb size is set to 10G, but it needs a bit more space than that) [10:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:47:20] dcaro: yes it works now. thanks a lot! [14:28:14] !log admin adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 (T316284) [14:28:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:28:18] T316284: Replace cloudnet100[34] with cloudnet100[56] - https://phabricator.wikimedia.org/T316284 [16:56:27] !log tools.stewardbots ./stewardbots/SULWatcher/manage.sh restart # All bots disconnected [16:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [16:57:46] kindrobot: thanks a lot for your comment, it really helps to see that what you are working for is appreciated :) [18:24:13] !log tools.lexeme-forms deployed 93ebb772c5 (more Spanish templates) [18:24:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:26:38] !log tools.lexeme-forms deployed 953b553968 (translate Hebrew adjective template label) [20:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL