[13:08:58] Does anyone have hosts to reimage? I'm looking for reimages to do to try the new reimage cookbook with. [13:09:13] volans: I will have next week [13:09:44] ack, by next week I hope that will be already stable enough to be the default way to do reimages :) [13:10:45] ok, I will ping you anyways just in case you want to test something [13:11:24] thanks, that would be great [13:45:58] volans: you can pick a mediawiki server from codfw [13:46:23] I can pick one for you [13:46:28] effie: yepI already did :) [13:46:36] I can pick it again to make sure nothing changed [13:46:49] but I was hoping to get some host that needs reimage already for real [13:46:50] ah, you are just looking for fresh victims [13:47:24] ah, I need to strip something that mwmaint2002 has [13:47:37] so reimaging it would be alright [13:47:50] I will ping you when I do it [14:24:09] effie: thanks! [15:54:57] > the ops team spent some time trying to find [the Dwimmerlaik page] during an outage, […] change our dummy titles to something very clearly explaining what's going on [15:55:14] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/201678 [15:55:56] Context: Before 2015, in MediaWiki when a piece of wikitext was parsed not for a page view content (e.g. all the interface labels and such), then the parser used the dummy title [[Dwimmerlaik]] as its value. [16:23:25] Krinkle: ha, I love it [18:48:32] Anyone arround that can update PCC facts for me? Specifically i'm trying to pull the facts from the puppetmaster at wdqspuppet.wikidata-query.eqiad.wmflabs [19:10:50] After merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959 we're getting errors like this: `Confd template for /srv/config-master/pybal/codfw/wcqs on puppetmaster2001 is CRITICAL: File not found: /srv/config-master/pybal/codfw/wcqs`. It makes sense why the error is cropping up since that puppet patch adds the `conftool-data` files for `eqiad/wcqs` and `codfw/wcqs` [19:11:24] What's not clear to me is what actually creates `/srv/config-master/pybal/codfw/wcqs`. Anyone have any context (or even educated guesses) here? [19:11:44] Or rather, what's actually supposed to create wcqs* [19:13:31] ryankemper: looks like it should by pybal::web, invoked from profile::configmaster. It works by looping the sites in each svc. In theory that means the PCC for the patch should include the file [19:25:59] ryankemper: after pulling a compiled catalog, i would suspect some sort of ordering issue. /srv/config-master/pybal/codfw/wcqs is defined in the catalog [19:29:09] * ryankemper throws a couple of darts randomly [19:29:12] mutante: herron: are either of you lucky winners around to add me as a user to https://openstack-browser.toolforge.org/project/puppet-diffs? I need it for access to the compiler hosts for https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs#How_to_update_the_facts_for_cloud_VMs?_(e.g._INFO:_Unable_to_find_facts_for_host_util-abogott-stretch.testlabs.eqiad.wmflabs,_skipping) [19:29:37] ryankemper: it would be my honor [19:29:41] 1 min [19:29:45] thanks! [19:32:21] ryankemper: are you able to get in now? [19:33:02] herron: nope [19:33:28] (Specifically `ssh compiler1001.puppet-diffs.eqiad.wmflabs` fails, and also I don't see myself in the users yet) [19:34:56] hmm, I added user ryankemper via horizon to puppet-diffs as a project member [19:35:18] openstack-browser is heavily cached [19:35:33] herron: I just saw it pop up in the web ui now, I'm gonna blame caching as always [19:35:35] ^ yeah, that [19:36:33] I'm still seeing `Connection closed by UNKNOWN port 65535` when I `ssh compiler1001.puppet-diffs.eqiad.wmflabs`, not sure if being added as a user wasn't the problem or if there's a delay on being able to ssh into the instance, perhaps I need to wait for an automated puppet run? [19:36:35] I see you on the users list now on the browser, fwiw [19:36:47] (Not sure if we maintain the every 30 minutes cadence for the cloud instances too) [19:37:01] I think it should be instant [19:37:32] yeah I'm seeing a PAM access denied in the auth.log on the host, but afaik it should be instant too [19:37:35] majavah: herron: Nevermind the ssh just worked now. Not sure the cause of the slight delay between the web ui and ssh given that I see puppet hasn't been run [19:38:02] herron: majavah: thanks! [19:38:10] haha ok note to self -- add project member, go for a short walk, then try to use it! [19:38:12] np! [19:38:19] :P [19:38:34] maybe sssd cache or something? [19:38:45] it certainly doesn't need a puppet run [20:51:03] yea, it's sometimes instant and sometimes it's just not.. not sure why, but it's liek that