[14:37:13] sorry for Friday ping so feel free to ignore or suggest i file a phab ticket. there's a Cloud VPS project that no one on my team can ssh into right now (but instances on our other projects are all accessible via ssh). the project was upgraded with additional resources about a week ago (T301121) and was working fine after that and even as recently as this Wednesday when I created a new instance. the instances are all running (some of [14:37:13] them host web proxies) but no one can access via ssh to make changes. i'm curious if someone could help diagnose the issue. i get `Connection closed by UNKNOWN port 65535` when I try to ssh in and other members get a `channel 0: open failed: administratively prohibited: open failed; stdio forwarding failed; ssh_exchange_identification: Connection closed by remote host`. an example instance would be: [14:37:13] `edit-types.research-collaborators-api.eqiad1.wikimedia.cloud` [14:37:15] T301121: Request increased quota for research-collaborations-api Cloud VPS project - https://phabricator.wikimedia.org/T301121 [14:38:35] happy Friday, is clouddb2001-dev.codfw.wmnet in circulation? It's the only db host I can find that is on stretch https://gerrit.wikimedia.org/r/c/operations/puppet/+/761927 [14:43:04] Amir1: yes, afaik it holds the labtestwikitech database (but my wmcs-roots/labtest-roots apparently don't let me to log in and verify) [14:43:25] isaacj: let me have a quick look [14:43:27] isaacj: I can't get into any instances in that project with my root key either. I wonder if something bad has happened to the project's security groups [14:44:47] !log research-collaborations-api Added BryanDavis (self) to project to debug ssh connectivity issues [14:44:48] isaacj: you have the project name in the hostname wrong, the correct hostname is `edit-types.research-collaborations-api.eqiad1.wikimedia.cloud` [14:44:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Research-collaborations-api/SAL [14:45:33] oh gosh thanks taavi! so embarrassing! [14:45:52] lol. nice catch taavi. I had cut-n-paste and not really read the hostname and then kept just changing it as I walked the project. :) [14:45:54] thanks bd808 -- sorry for wasting time. sigh... [14:46:02] taavi: okay, what I can do make it migrate off stretch [14:46:06] isaacj: no worries. we all need other eyes some days [14:46:36] heh. it becomes rather obvious after a ctrl-f of the project name doesn't find it on the horizon dropdown [14:47:14] yep -- and i was pasting the example project in for my other team members to check too so none of them caught it. thanks both. maybe a sign i should go make myself some tea or something [14:47:21] !log research-collaborations-api Dropped BryanDavis (self) from project [14:47:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Research-collaborations-api/SAL [14:51:32] Amir1: good question! I suspect you want to ask a.ndrewbogott about that, but in general I don't imagine a few hours of labtestwikitech downtime is a big deal [14:51:54] yeah [14:52:12] okay, I'll ask [14:52:17] oh also I think deployment-prep and cloudinfra still have stretch mariadb instances. I can try to get the cloudinfra ones upgraded like this weekend, but not sure about deployment-prep [14:57:15] are they using mariadb role in puppet? [14:58:26] yes, role::mariadb::cloudinfra and role::mariadb::beta [15:01:21] 😭 [15:03:29] oh fun, apparently cloudinfra-db02 has the mariadb service stopped which means that it hasn't been replicating off db01... since late 2019, apparently? [15:03:52] that's an another reason to get that upgraded/fixed asap [15:13:10] you need to reclone from primary, binlogs don't stick around for that long (production is usually 30 days I think) [15:14:07] I need to do that anyways, no in-place reimages in cloud [15:18:08] !log cloudinfra switch floating ip 185.15.56.27 from ntp-02 to ntp-04 [15:18:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:41:52] !log cloudinfra switch floating ip 185.15.56.3 from ntp-01 to ntp-03 [15:41:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [21:11:29] !log quarry switching shared nfs project dir (again) to internal nfs server quarry-nfs-1 [21:11:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [23:05:57] !log tools.lexeme-forms deployed b4624e0bbc (l10n updates) [23:06:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL