[09:26:30] <arnaudb>	 will restart orchestrator service on dborch1001
[09:30:45] <marostegui>	 ok!
[10:01:10] <arnaudb>	 We ( moritzm btullis and I ) checked a bit more of what could be the root cause of Orchestrator x509 issue → it might come from the way Go handles reading/loading the ca chain in memory. We're not sure, this will require some further code testing/debugging. Also, it's maybe a symptom that we should consider a course of action regarding Orchestrator maintenance 
[10:01:10] <arnaudb>	 https://github.com/openark/orchestrator/pull/1478/commits/2f032d8af3e65de0c99806cb6e17071ff1f6f2a6
[10:05:23] <marostegui>	 Yeah, we are aware of that :)
[10:20:30] <marostegui>	 arnaudb: Why would that change with a puppet migration? if it is memory related, I wonder why that's not solved by a daemon restart and/or host reboot?
[10:21:45] <arnaudb>	 I think it comes from a bug handling a ca bundle instead of a single one, not directly from puppet itself 
[10:23:16] <marostegui>	 Ah I see what you mean
[12:25:30] <jynus>	 @Amir1 cloudb hosts should not have production root password, if they have, that would be considered a breach
[12:26:11] <Amir1>	 it doesn't have the password from what I'm seeing
[12:26:23] <Amir1>	 it might be password hash
[12:26:28] <marostegui>	 they have a different one
[12:26:33] <jynus>	 ah, ok
[12:26:49] <Amir1>	 and that could be why db-mysql is failing then
[12:27:00] <jynus>	 then, that is a good thing
[12:27:02] <jynus>	 :-D
[12:27:11] <marostegui>	 no, .my.cnf is correct in cumin102
[12:27:11] <jynus>	 better than the alternative
[12:27:25] <marostegui>	 My guess is it is a password in the old format, I will check when I am finished with something I am doing now
[12:28:07] <jynus>	 either that or an issue with db-mysql indeed
[12:31:04] <marostegui>	 I just fixed clouddb1021
[12:31:52] <jynus>	 so new client stopped supporting it, likely?
[12:33:38] <jynus>	 "mysql_old_password support was removed from the mysql client libraries in 5.7.5"
[12:34:16] <jynus>	 but pymysql in theory still supports it
[12:36:46] <marostegui>	 All fixed
[12:48:54] <Amir1>	 marostegui: Thanks, shall we close it?
[12:49:04] <marostegui>	 not yet, I am checking the other two hosts which are not clouddb
[12:49:09] <marostegui>	 I will close it when fixed
[12:49:11] <Amir1>	 ah okay
[12:49:13] <Amir1>	 thanks
[15:23:50] <marostegui>	 I can't believe in 2024 megacli options are still so terrible
[15:57:31] <topranks>	 urandom, Emperor: I created a master ticket to kick off discussion on the task ahead of renumbering servers in codfw row A/B 
[15:57:36] <topranks>	 T346428 Re-IP hosts on codfw row A and B to new per-rack vlans/subnets
[15:57:36] <stashbot>	 T346428: Netbox: Add support for our complex host network setups in provision script - https://phabricator.wikimedia.org/T346428
[15:58:03] <topranks>	 I also created 3 basic sub-tasks for the hosts we discussed here yesterday, where we can discuss the specific approach that might work 
[15:58:38] <topranks>	 feel free to add any detail/thoughts, and add subscribers who may also have input.  thanks!
[16:02:46] <Emperor>	 topranks: thanks, will try and get some thoughts on that tomorrow
[16:07:31] <claime>	 marostegui: I goofed up a select that'll probably run way too long, what dO?
[16:08:25] <claime>	 I killed it client side, can't remember if it'll kill it server side?
[16:16:20] <marostegui>	 claime: no, it will keep running in the server assuming it was already running 
[16:16:29] <claime>	 crap
[16:16:53] <topranks>	 urandom, Emperor: seems I linked the wrong task sorry - it's T354869 
[16:16:54] <stashbot>	 T354869: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869
[16:17:24] <claime>	 marostegui: query killer server side engages at some point?
[16:19:03] <marostegui>	 Yep
[16:19:28] <marostegui>	 Unless it's not coming from the wikiuser 
[16:19:36] <marostegui>	 Which I assume it's not the case 
[16:23:04] <claime>	 It's coming from the user our sql script on mwmaint uses
[16:29:00] <marostegui>	 If it's wikiadmin then we need to kill it on the server side 
[16:33:09] <claime>	 it is
[16:37:11] <claime>	 I think it's ok, I checked the mysql processlist from the server I was querying and my queries aren't there anymore
[16:38:14] <marostegui>	 Right :)