[05:24:00] federico3: I created https://phabricator.wikimedia.org/T430104 and set it to high priority, if you can take a look that'd be appreciated [05:25:00] I've updated the masters table manually in the mean time [06:53:19] @marostegui ack, i'll fix it in a bit [06:54:00] thanks! [07:00:52] the reason is that this returns nothing: SELECT name FROM instances WHERE server like 'db1290%'; [07:01:17] Fixing that! [07:01:49] first time i see this glitch. At least the script should check before doing the run [07:02:03] yeah [07:02:12] and maybe tell the operator to fix that [07:02:17] I am fixing db1290 entry [07:02:32] so I can check adding it via zarcillo UI will also add an entry on servers as we discussed yestrerday [07:03:37] adding it to zarcillo makes it show up on serverrs table with the correct rack, very nice!!! [07:03:40] yes (FWIW it works on my testbed) [07:04:16] yep it detects from netbox on the fly [07:04:23] yeah, that's very nice [07:06:10] BTW I found https://phabricator.wikimedia.org/T272954 - at some point it could make sense to make an inventory of places where we do queries directly on zarcillo's DB [07:07:15] for now I'll add the safety check as part of T430104 [07:07:15] T430104: db-switchover failed when insering into zarcillo - https://phabricator.wikimedia.org/T430104 [07:07:22] +1 thanks [07:41:31] m3 codfw master (unused) is down https://phabricator.wikimedia.org/T430116 [08:02:46] dhinus: I am going to give some wait to the new clouddb1026 (in s1) [08:34:51] marostegui: you mean before pooling it? [08:36:46] dhinus: sorry i meant weight, i gave it 10 and it is there [08:37:21] haha ok, makes more sense :) [08:50:42] federico3: I added clouddb1026 via zarcillo gui, and it shows in servers, but on the instances table, it shows badly, it shows as clouddb1026 but it should show as cloudb1026:3311, I guess the UI isn't prepared for multi-instance hosts? [08:50:53] If it is added to the labs section the instance should be hostname:port [08:51:00] Want me to create a task? [08:51:18] same for section_instances [09:15:09] @marostegui when adding it from the UI did you put the port number? [09:15:17] if so yes please create a task [09:15:21] yep [09:17:57] thanks I will do one more test and if it happens I'll create it! [09:19:55] dhinus: I gave it 100 weight, but I still see no connections from tools. Does the LB actually allow more than 1 instance? I can depool clouddb1013 and check if it gets connection, but I am wondering if this is expected. [09:58:36] marostegui: that's weird, I don't think we have any special config that prevents having more than 1 instance [09:58:46] let me check the config in the lb [10:00:43] dhinus: ok thanks [10:01:00] dhinus: right now the host has weight 100, same as clouddb1013, and I see no connections at all from users, which is strange [10:01:17] I can depool clouddb1013 and see, but we should check first if this is expected or not [10:01:35] yes let me check first [10:01:41] thanks you [10:05:26] this is the live config in the lb: https://phabricator.wikimedia.org/P94466 [10:05:35] looks correct, and uses roundrobin [10:05:44] let me try some manual connections from toolforge [10:08:11] thanks [10:08:20] Grants and roles look ok [10:08:23] And views ofc [10:11:25] from the LB logs: "Health check for server wikireplica-db-web-s1/clouddb1026.eqiad.wmnet failed" [10:11:31] maybe a firewall issue? [10:12:35] hm but I can open a connection from the LB [10:13:02] nope, I was using clouddb1013, and it works. but "nc" fails with clouddb1026 [10:13:37] "nc clouddb1026.eqiad.wmnet 3311" from cloudlb100x fails [10:16:09] mmm [10:16:15] pupeptwise they seem to have the same config [10:16:46] I have depooled it for now [10:17:17] thx, I have asked in the cloud channel if anybody remembers related firewall settings [10:17:28] I will open a tracking task [10:17:39] thank you, please cc me [10:23:05] done, T430132 [10:23:05] T430132: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132 [10:25:27] ta [11:06:58] XioNoX: can you confirm clouddb1026 is in the same vlan as clouddb1013 and/or clouddb1015? [11:07:53] I see it being the same on netbox, as in the private vlan [11:07:57] but just to triple confirm [11:52:10] marostegui: yep [11:52:16] thank you [11:55:33] marostegui: do you know what's up with cephosd2001 ? I don't know if it needs to be depooled or not for today's maintenance [11:56:14] XioNoX: Emperor ^ [12:16:39] Right he is out [12:16:44] maybe urandom? [12:23:33] brouberol, btullis ^ maybe you can help with that? [12:24:34] dhinus claime https://phabricator.wikimedia.org/T430102 do you think this can be just dropped from wikireplicas? [12:25:49] @marostegui did you open a task for the clouddb1026 issue perhaps? maybe I missed it [12:27:22] federico3: Not yet, but it requires a bit more because I think you'd need to check if the host is set to be to labsdb, you'd have to assume that the instance would be hostname+port, but what goes to server is just the hostname. And if I add a different instance, because they have more than 1, then what? [12:27:33] I will add all that in the task, but I thought it is a bit more complex than we think [12:27:40] like some logic will need to be put in place for this [12:28:06] marostegui: yes I think they can be dropped [12:28:21] do you want to drop the views first, so we don't break it? [12:28:33] currently we can add multiple instances using the UI/API and the server row will be created on the first one - but I think we need to sit down and discuss this a bit more on monday [12:28:54] yeah, but the server table won't be touched because it is only one row per server [12:29:09] I will add all the thoughts to the task [12:29:32] yes it's meant to be added only if needed (so for the first instance creation, then the server is already there) [12:29:43] cool [12:30:31] I mean the query is literally ""INSERT IGNORE INTO servers (fqdn, hostname, dc, rack)" :D [12:30:37] :_) [12:30:44] marostegui: yes I can drop the views, but I'm afk, will do it as soon as I'm back [12:30:51] dhinus: no worries [12:30:58] dhinus: I think it is better to drop the views and THEN the db [12:31:06] agreed [12:52:10] XioNoX: as stated on #-operations, we should be fine even w/o depooling [12:52:18] sorry for the slow response [12:53:44] brouberol: no pb! I should have seen sooner that we didn't get an answer for that host