[08:48:08] FYI, I'm temporarily disabling Puppet on scap hosts for the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/789146 [09:09:02] I am about to upgrade firmware and upgrade os of the main backups hosts [09:09:57] If you are planning to delete accidentally production data, please don't do it anyway, but now specially wait a few hours until the service comes back [09:10:33] lol [09:11:02] * marostegui stops his delete from revision; [09:12:52] marostegui: actually, db recoveries are not affected by this [09:13:07] so feel free to accidentally delete prod data :D [09:13:14] I can proceed then! [09:13:31] as long as you add a where to that clause, yes [09:13:36] XD [09:14:23] BTW, this confirms the need to have high availability on the bacula director: https://wikitech.wikimedia.org/wiki/Bacula#Architecture_update_(2020) [09:14:57] I would like to have 2, one on each datacenter, with separate stacks so next time we never lose backup coverage [09:27:26] expect secundary temporary error such us prometheus monitoring of bacula, gerrit backup delays, or storage not communicating with the director (expected) [10:13:42] Things looking great so far, any volunteers to test data recovery to a buster host? [11:19:45] Error: /Stage[main]/Profile::Conftool::Requestctl_client/File[/usr/local/bin/requestctl-checkip]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/profile/conftool/requestctl-checkip.py [11:20:19] ^ there are a few hosts failing puppet: https://puppetboard.wikimedia.org/nodes?status=failed [11:26:28] the contint ones seem to be from failing puppet code related to helm [11:28:41] jynus: looking [11:28:47] * jbond is looking [11:54:22] <_joe_> jbond: that file sounds new to me, but I'm happy to help if needed [11:56:01] _joe_: yes its a simple script i wrote so we can search to see if an ip is in a specific ip-block in the requestctl private repo. the error was refrencinf the file with and underscore vs hyphen. its fixed now [11:56:18] <_joe_> ha [11:57:00] its only about 5 lines of actuall implmentation code https://github.com/wikimedia/puppet/blob/production/modules/profile/files/conftool/requestctl_checkip.py [17:23:21] klausman are you OK with me puppet-merging your ores change? [17:23:28] yes, pleasedo [17:23:50] Cool, it's merged [17:23:55] Merci bien [19:04:18] What's the difference between puppet-merge.sh and puppet-merge.py in puppetmaster/files? They seem to have both been last touched at around the sametime and feature similar docstrings. [19:06:45] brett: partial rewrite of puppet-merge that was bash, the current puppet-merge run by users (us) is the bash, that then in turn runs the python one, but the automatically-triggered part on the other puppetmasters, run as gitpuppet and runs directly the python version only [19:11:22] Is this doubling intentional or was the rewrite just never finished? [19:12:11] for that you should ask jb.ond ;) [19:12:39] (added dot to prevent ping at this time) [19:12:49] volans: Thanks for the help :) [19:37:22] For those interested, it does appear that the intention is to fully move to only the Python file: https://phabricator.wikimedia.org/T254249 [20:14:00] I'm curious what https://wikitech.wikimedia.org/wiki/Category:Nova_Resource is for; It looks like they're largely just mirroring OpenStack documentation. A few articles have small bits of information peppered but they largely seem to just be the same template [20:20:08] brett: the pages in that category are auto-generated when a project is created. mostly for having a separate "SAL" log for each project. if you do "!log foo" on the -cloud channel it's like using !log on -operations but only for your project and ends up on these pages. [20:20:33] so that would be encouraged when making changes in shared project [20:20:43] mutante: Makes sense. Thanks! [20:21:20] also can be useful to check who are the admins of a project without having to join it [20:21:58] well it just links to these: https://openstack-browser.toolforge.org/project/incubator [20:22:22] but more background is.. before we even had openstack-browser and before we even had horizon.wikimedia.org .. wikitech wiki was the UI for all these things [20:22:33] even managing the VMs etc [20:30:38] I tried to capture that in the cat page [20:35:29] In days long past, but not quite forgotten, wikitech served the role of Horizon as well. We managed all of the OpenStack projects, membership, instances, etc directly on wikitech. The Nova Resource namespace is a still useful artifact left over from that time. [20:37:19] * brett stokes the campfire [20:37:25] https://openstack-browser.toolforge.org/ provides a more complete view of the OpenStack projects than the Nova Resource pages on wikitech do these days. But it is handy for bigger/complex projects to use pages like https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep to document themselves. [20:41:37] a.ndrewbogott and I have gutted most of the magic out of the old extension, but https://www.mediawiki.org/wiki/Extension:OpenStackManager was the Horizon like thing embedded in wikitech. The extension is actually still installed there, but has been reduced to a small amount of customization for Extension:LdapAuthentication [20:50:30] !log resuming last 6 esams cp host firmware updates via T243167. cp306[01] going offline [20:50:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:50:35] T243167: Upgrade BIOS and IDRAC firmware on R440 cp systems - https://phabricator.wikimedia.org/T243167 [21:15:22] !log cp306[01] returned to service, cp306[23] coming down for firmware update via T243167 [21:15:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:15:27] T243167: Upgrade BIOS and IDRAC firmware on R440 cp systems - https://phabricator.wikimedia.org/T243167 [21:34:08] win 27 [21:43:12] !log cp306[23] returned to service, cp306[45] coming down for firmware update via T243167 [21:43:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:43:18] T243167: Upgrade BIOS and IDRAC firmware on R440 cp systems - https://phabricator.wikimedia.org/T243167