[02:30:47] I replied on Leaderboard's talk page [05:30:22] legoktm: we are going to need to rethink the scratch NFS share soonish. The last physical NFS server is close to being decommissioned. I don’t know that we have planned for a new cross-project replacement yet. All the other NFS shares besides scratch and tools have already migrated to in-project instances backed by ceph. [05:31:16] We can build a virtual scratch, but we probably need a better idea of it’s value. [05:41:10] ack, that makes sense [10:41:45] !log admin restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503) [10:41:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:15:48] Hi. I've tried to ssh into a Cloud VPS instance that has just been created for me. I'm getting the error "Connection closed by 185.15.56.87 port 22" and "kex_exchange_identification: Connection closed by remote host". Using the verbose setting it says the server accepts my public key, but then after that the connection is closed. [13:17:52] Following the steps on the page for an error like this hasn't fixed it [14:28:40] what's the hostname of the cloud vps instance? [14:36:50] does someone need to be in project-bastion in order to connect to a cloud vps instance? Because it seems to me that dreamy isn't in that ldap group from looking at https://ldap.toolforge.org/user/dreamyjazz. [14:38:27] I had the same question, but I'm not sure if it's a requirement or not [14:39:51] https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#Be_a_member_of_a_Cloud_VPS_project [14:40:03] "you need to be a member of at least one project (then the project-bastion LDAP group will be added automatically)" [14:47:35] I only added them as projectadmin [14:47:45] so perhaps they need user membership as well [14:48:16] arturo: yes, you need to add them as an user too [14:48:28] doing that now [14:50:13] the docs tell that it's needed but doesn't explain why, maybe we need to fix the docs per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Creating_a_new_project [14:51:37] Dreamy_Jazz: try now! [14:51:52] taavi: also, now that we have a cookbook, for it, it can be easily automated [14:53:37] Thanks [14:53:40] I will try now [14:54:06] Thanks it worked! [14:54:24] Didn't see messages until the ping [14:54:36] now I see the user is also a member of project-bastion in https://ldap.toolforge.org/user/dreamyjazz, so maybe that's triggered by adding users as "user" of a project, but not by adding them as "projectadmin"? [14:54:50] dhinus, taavi https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/879816 [15:01:55] dhinus: a user will always need both roles to function. There are a lot of magic hooks associated with the 'user' role. [15:02:09] That could be reworked, or we could bury that in a cookbook. [15:02:24] hm, I'm late, everyone has already said all these things :) [15:02:26] I'm fine with burying it in a cookbook for now :) [15:02:41] not just said but arturo has already updated the cookbook too :D [15:03:18] I haven't found where the magic hooks are stored yet, any hint? [15:05:42] arturo: for that patch, what if the user already had the user role? [15:05:57] taavi: noop? [15:06:38] taavi: confirmed, is a noop [15:12:40] T301949: I have just started a test backup on ToolsDB *primary* (so far I always tested on the replica). I'm keeping an eye on stats to see how much it impacts on performance, hopefully not much but let me know if you see any slowness and I'll stop it. [15:12:40] T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4 - https://phabricator.wikimedia.org/T301949 [19:15:03] hey all i'm having a lot of trouble with a vagrant instance today https://skins-demo.wmflabs.org/ is redirecting to an unrelated https://skins-2022.wmcloud.org/ - could this relate to the current upgrades? I tried to add https://skins-demo.wmcloud.org/ to point to port 8080 but that's not working [19:15:22] I'm also having some issues with disk space which may be related [19:19:22] the static part of the site on port 80 is working fine (https://skins.wmcloud.org/#/) [19:20:53] !log tools.refill-api Recreated refill-api deployment [19:20:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL [19:23:18] Jdlrobson: which 'current upgrades' are you referring to? [19:27:04] Jdlrobson: skins-demo is showing a php syntax error [19:27:16] It sounds like a broken git rebase [19:32:28] ok no idea but ive somehow fixed it [23:23:13] I'm getting 500 errors on iabot.toolforge.org [23:23:23] 2023-01-12 03:10:04: gw_backend.c.2275) response not received, request sent: 982 on socket: unix:/var/run/lighttpd/php.socket.iabot-0 for /api.php?action=statistics&format=flat, closing connection [23:23:23] 2023-01-12 03:11:35: http-header-glue.c.1499) read() 9 11: Connection reset by peer [23:23:40] Is what I see for these errors. Not sure what this means though.