[00:33:53] https://www.youtube.com/watch?v=rksCTVFtjM4 [00:34:48] #wikimedia-cloud today [00:36:04] actually https://www.youtube.com/watch?v=5UT8RkSmN4k is the better clip [08:40:44] PAWS seems to be down for a while. A colleague has a training session later today (about PAWS). Does anyone know what might be the issue and/or when it might be back up again? [08:44:58] nevermind, it's working! [09:44:24] !log tools.pb webservice restart ref. cloud-l [09:44:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pb/SAL [12:18:08] bd808: thank you. I didn't realize this was related as one was a 500 error in this was a 503. Platonides: I have no idea. I have not been able to trigger any segfaults on my end, or the wmcloud VMs operating IABot. From my experience PHP segfaults are usually from PHP itself, and that I can at best work around them, if I knew what part of the code was even triggering them. [12:28:26] !log tools.lexeme-forms deployed 4fa53fae89 (l10n updates: pt-br) [12:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [15:09:14] !log tools try to rescue docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:eebb155bd1116e3b67e2ce43244f9c9958df0cbb75a84c231565fae2ed87c9f4 back into the registry from a k8s worker local cache (T321188) [15:09:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:09:17] T321188: [tbs] fix/improve the updating of the buildpack/tekton images in the local repo - https://phabricator.wikimedia.org/T321188 [15:37:34] !log tools root@tools-docker-registry-05:~# curl -sS -X DELETE localhost:5000/v2/toolforge-distroless-base/manifests/sha256:2d4d28e45bbe4e38177fd4fdc922dbfaf95e607b06bbc4187a90410d895b4491 (T321188) [15:37:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:37:37] T321188: [tbs] fix/improve the updating of the buildpack/tekton images in the local repo - https://phabricator.wikimedia.org/T321188 [15:38:06] !log tools root@tools-docker-registry-05:~# docker-registry garbage-collect /etc/docker/registry/config.yml (T321188) [15:38:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:51:50] Out of curiosity before i start doing googling, is there any easy way to push an archive of files on all the cloudvps hosts in a project to a volume, would rsync work? [15:57:26] rsync wouldn't give you an archive, it's basically a faster cross-host recursive copy. [15:57:44] so you'd end up with a directory at the target which matches the source directory [15:58:10] But would it be its own copy or would it act more like a symlink? [16:00:10] Effectively what im trying to do is copy files on all the instances to the volume so I can rebuild the instances with the most recent OS [16:02:35] it'd be it's own copy [16:03:23] alright thanks! [16:03:48] I'm in the habit these days of not having any persistent data on the instances themselves, with the exception of a MariaDB instance. [16:04:21] everything else (application, config, package dependencies, etc) is deployed from config management/git [16:04:36] Yeah thats the long term goal, but im still fiddling with alot of things, so its just easier to have it just locally instead of messing with git [16:05:37] I do all of my fiddling and experimentation locally on my own machine, not on WMCS :) [16:06:18] (well, that's not quite true, but it might as well be for the purposes of this discussion - *cough* terraform *cough*) [16:06:43] Lol [16:36:01] So, I appear to no longer have sudo access to ANY of my VMs on the cyberbot project. [16:36:47] cyberpower678@cyberbot-db-01:~$ sudo mysql [16:36:47] sudo: a password is required [16:36:47] Sorry, user cyberpower678 is not allowed to execute '/usr/bin/mysql' as root on cyberbot-db-01.cyberbot.eqiad1.wikimedia.cloud. [16:37:34] * taavi looks [16:40:28] Cyberpower678: try now? [16:40:48] Hey it works. What changed? [16:41:51] I am not sure. the correct rules existed in https://horizon.wikimedia.org/project/sudo/, so I just restarted the service responsible for loading that (and all other authentication data) [16:47:09] !log lta-tracker delete puppet3 and bots instances (prep for OS upgrade) [16:47:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Lta-tracker/SAL [16:48:22] taavi: cool. Thanks. :D [16:58:23] !log lta-tracker create bots2 instance (OS: debian-11.0-bullseye) [16:58:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Lta-tracker/SAL [18:29:34] !log tools.wd-image-positions deployed 0e0deb1a49 (reimplement “add new depicted” form in Codex) [18:29:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [18:29:54] took a bit longer but there we go :) (re @lucaswerkmeister: (I also wanted to migrate one of my tools to it, let’s see if I can make that happen and use it as a second example in the email...) [18:30:01] I’ll write that email now [18:47:38] done: https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/thread/BJKLALZUZ3WCKJVWE4JJHOICAS4LUNH3/ (also to wikitech-l)