[08:40:16] vgutierrez: what to talk about https://gerrit.wikimedia.org/r/c/operations/puppet/+/924342 ? [08:53:16] want* [09:17:14] It isn't feasible sadly [09:17:32] Not easily anyways [11:01:03] can I please get an additional pair of eyes here? https://gerrit.wikimedia.org/r/c/operations/puppet/+/924481 [11:01:13] trying to don't break LVS for upload.w.o [11:04:07] vgutierrez: ^ [11:09:07] Arturo: looks good. I can take care of merging / restarting pybal after I get home. Maybe sukhe comes online earlier than that [11:09:28] thanks vgutierrez, really appreciated [12:18:52] jbond: [12:18:58] aborrero@cumin2002:~ $ sudo install_console cloudcontrol2005-dev.mgmt.codfw.wmnet [12:18:58] /usr/local/bin/install_console: line 23: $2: unbound variable [12:19:47] jbond: https://gerrit.wikimedia.org/r/c/operations/puppet/+/922559 may be the cause [12:21:33] arturo: yes very, looking [12:30:11] godog, bblack: I'm disabling puppet [12:30:16] on 'R:package = cadvisor' [12:30:29] arturo: should be fixed now [12:30:36] volans: ack [12:30:41] jbond: thanks! [12:30:45] and that's now done [12:31:06] ok I'll fixup the mistake while we're at it [12:31:22] let the revert: revert: dance begin [12:31:56] you can also just fix it forward [12:31:58] if it's trivial [12:32:56] indeed, I'll do that [12:34:19] https://gerrit.wikimedia.org/r/c/operations/puppet/+/924500 [12:35:35] godog: the docs says it's a single - [12:36:09] but I see the existing parameters with -- [12:36:16] so I guess that also works? [12:36:21] yeah indeed, works with - and -- [12:36:59] cool, then lgtm [12:37:07] cheers [12:39:41] I'll reenable puppet on R:package = cadvisor [12:40:22] I used "cadvisor broken" [12:40:25] as reason [12:40:36] thank you [12:40:54] feel free to run https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed [12:41:01] if you want a quicker recovery [12:42:19] *nod* will do [12:46:29] I'm reimaging a host with --new (because it was renamed, changed FQDN, etc) and is now stuck in [12:46:30] [26/50, retrying in 78.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb..poll_puppetdb' raised: Nagios_host resource with title cloudcontrol2005-dev not found yet [12:46:49] volans: actually in this case --failed-only won't work because afaics the puppet run is successful, the unit failed [12:47:01] godog: ack [12:47:09] I'll run puppet instead [12:47:21] arturo: that could mean that the NOOP puppet run on the host failed [12:47:29] and didn't generate even exported resources [12:47:33] so failed to compile entirely [12:47:44] ok [12:47:58] try to run puppet agent with the noop option on the host [12:48:04] how can I ssh into the host if not even the root password works? [12:48:14] puppet agent -t --noop [12:48:20] the install_console script [12:48:23] from a cumin host [12:48:26] or puppetmaster [12:48:59] let me see [12:50:02] ok, I constantly forget about that access, I always the install_console for the .mgmt. interface and watch the installer [12:50:28] thanks volans I see now the error and seems trivial to fix :-P [12:50:40] good :) [12:50:40] no puppet role has been assigned to this node. (file: /etc/puppet/manifests/site.pp [13:43:45] https://www.documentcloud.org/documents/23819681-law-enforcement-working-party-document-encryption (Law Enforcement Working Party document - encyption) [13:44:19] the full document re: https://www.wired.com/story/europe-break-encryption-leaked-document-csa-law/ [13:55:09] (relevant: https://www.schneier.com/blog/archives/2005/12/computer_crime_1.html) [14:35:57] if only governments could learn to healthily distrust themselves. [15:09:19] jbond: I no longer know who to vote in my country :-) [15:11:17] arturo: yes its (spanish polotics) something i need to do a lot more reading about. we cant vote in the national and just missed the locals. but we where out eating the other day and there is a lot of support for vox where we are so i think i need to get a bit more involved and understanding the local scene a bit better