[07:03:26] btullis: I assume it is intentional, but db1108 has mariadb stopped [07:03:36] Just letting you know in case you forgot to bring it back up after the migration [08:31:37] marostegui: Ack, thanks for the reminder. I'm going to move straight to decom for db1108. [08:31:47] ok [08:32:04] btullis: My advice would be to leave it like that for 3-4 days before going for a full decom [08:32:53] OK, will do. [08:52:30] btullis: My point was to leave mariadb stopped for more days, maybe it wasn't clear. So just in case something unexpected is trying to connect to it, so you can know before fully decommissioning the host :) [08:54:08] Oh, sorry. I thought you were suggesting that I start it, even though the replication from its sources won't work. I'll stop them again and downtime the services. [08:56:00] jayme: I just saw puppet is disabled on cloudcontrol1005 with (Reason: 'rollout 937573 - jayme') [08:56:17] but checking the patch, I don't this host has anything to do with the affected code? [08:57:27] btullis: Yeah, usually before decommissioning a database it is a good idea to leave the process stopped for a few days, you never know which service might be using it behind the scenes and unexpectedly :) [08:57:34] arturo: if it's running confd it has :) [08:58:20] arturo: if you need puppet enabled rn, you may do so. I will re-enable in a couple of minutes anyways [08:59:57] jayme: I can wait. I think I just discovered something new about my hosts [09:00:00] :-) [09:00:55] ahh I think the firewall profile may fetch files from confd [09:01:11] modules/profile/manifests/firewall.pp: confd::file { '/etc/ferm/conf.d/00_defs_requestctl': [12:58:59] <_joe_> sorry I have a brainfart [12:59:13] <_joe_> what was the git repo that mirrors all the hiera settings in horizon? [13:10:10] _joe_: cloud/instance-puppet.git [13:10:58] <_joe_> taavi: <3 thanks [14:47:08] jbond: hi! happy friday [14:47:25] definitely not urgent but fabfur and I are trying to reimage lvs1016, an offline experimental LVS host [14:47:35] the cookbook runs fine, d-i starts and finishes and then the host reboots [14:47:48] but the cookbook doesn't detect d-i and thus fails saying it doesn't think it is in d-i [14:47:51] any idea what can be wrong here? [14:49:09] some context: lvs1016 is one of the four host moved (T341992) this week and will be used for some testing, they are not serving traffic. The reimage went fine on the first 3 but fails on this [14:49:10] T341992: Relocate lvs1013-lvs1016 to rows E & F - https://phabricator.wikimedia.org/T341992 [14:50:11] checked the firmware and other obvious stuff, did a powercycle and racreset [14:51:01] sukhe: i have seen the issue but not been able to work out whats wrong see here https://phabricator.wikimedia.org/T342345 [14:51:15] oh [14:51:17] thanks jbond [14:51:20] that's certainly comforting [14:51:26] * sukhe brings out the rubber duck [14:52:33] I am going with "go" for now [14:53:18] i have not had much time to look at it and im out mon-wed next week but i pingd vol.ans to see if they can take a look on monday [14:53:28] makes sense, thanks ! [14:53:34] sukhe: yes i think its fine to use go, however yuo will need to do a manual reboot which is a pain [14:53:49] yeah [14:53:56] for the experimental hosts it's fine probably [14:54:00] +1 [14:54:01] but for anything prod, the OCD won't allow that :) [14:54:51] lol, hoenstly its not a big issue all the steps of the cookbook still run [14:57:37] :D [16:00:38] herron: arnoldokoth: nothing to report from today [16:01:20] jbond: Cool, thanks. [17:45:18] Is there anyone around who can tell me how much free space is available in `/var/lib/nginx/body` on the host running nginx which fronts docker-registry.discovery.wmnet ? [18:22:49] um. I'm looking at registry.2003 (which I think ?? is one of the nginx front ends? but I coul dbe wrong). it looks like /var/lib/nginx is 2G in size (tmpfs) and that's all available. now that I've said the wrong thing, someone will come along and correct it, right? [18:30:14] Thanks!