[04:39:30] <_joe_> ryankemper: dunno if someone fixed it in the meanwhile, but pcc works fine for me [04:40:05] <_joe_> I see some facts weren't re-imported, it seems [05:01:58] legoktm: ok to merge your change? [05:03:40] marostegui: yes :) [05:54:00] good morning! yes I re-imported facts yesterday evening [05:55:13] checking the past pcc run in jenkins to see which one broke [05:56:18] I think that Ryan referred to https://puppet-compiler.wmflabs.org/compiler1003/31181/ [05:57:50] indeed it still breaks [05:58:18] maybe puppet didn't run on that node when I updated the facts [06:02:04] I can re-import facts to quickly check [06:12:58] mmm no still broken [06:13:43] I tried it with an-launcher1002, that fails for another fact-related issue [06:13:50] https://puppet-compiler.wmflabs.org/compiler1002/31188/an-launcher1002.eqiad.wmnet/change.an-launcher1002.eqiad.wmnet.err [06:14:04] it is in hadoop commons, we read the facts['partitions'] map [06:18:47] I am reading https://gerrit.wikimedia.org/r/c/operations/puppet/+/722653/1/hieradata/role/common/puppetmaster/puppetdb.yaml, in theory 'partitions' should be removed as well, but numa should already be working for all nodes [06:23:07] and in fact on compiler1001 if I check [06:23:18] /var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet/facts/lvs2010... I don't see the numa entry [06:25:41] but the one for puppetmaster2001 shows the numa facts [06:25:59] but [06:26:00] -rw-r--r-- 1 jenkins-deploy wikidev 43041 Sep 19 04:49 puppetmaster2001.codfw.wmnet/facts/lvs2010.codfw.wmnet.yaml [06:35:26] so in https://puppetboard.wikimedia.org/fact/numa (that IIUC uses puppetdb) I see a lot of nodes with numa facts, but not lvs2010 [06:36:10] so it may be a corner case [06:44:36] elukey: interesting, so the numa fact has 1039 entries, a random fact (say manufacturer) has 1731 entries, so not too much of a corner case I'd say [06:45:03] ah I didn't count all nodes :( [06:45:15] it's at the bottom of the page :-P [06:45:35] yes I got it, I mean I didn't check a random fact to compare [06:45:50] and facter on the host ofc reports it correctly [06:45:58] so I need to double check the way we filter them [06:52:19] elukey: from a cursory look at the doc: "All other settings, which go in PuppetDB’s configuration file(s) and take effect after the service is restarted." [06:53:17] and I don't see a restart there, and that wouldn't explain how we got some numa and some not :/ [06:54:10] let's wait for jbond to understand the next steps [06:56:05] if it's not urgent yes I'd prefer to [06:59:26] elukey: and the fun part is that from the LVSes, we got eqiad/eqsin with numa and all the rest without [06:59:57] ah wait a sec [07:00:35] ok, those that are missing go to puppetmaster2001 AFAICT,so puppetdb2002 [07:01:43] ----- not related - I am going to refactor secrets/tokens for helmfile, the maintenance should last some mins, please sync with me first if you need to run helmfile :) ---- [09:12:57] volans: elukey: i sent puppetdb a HUP to reload the config, this is AFAIK undocument but there is some puppet issues which indicates it works for most config settings and from testing it worked with the facts_blackilist config. will look specifically at this issues, it could be because there are multiple facts files for each host (one frome puppetmaster2001 and one from puppetmaster1001) and we just [09:13:03] need to do some clean up. leave it with me and ill ... [09:13:06] ... let you get back to you [09:13:27] jbond: from what I can tell puppetdb2002 didn't get the new config [09:13:30] while 1002 did [09:14:04] volans: ack let me check [09:14:24] I've opened a bunch of related tasks fwiw [09:14:55] oh great thanks [10:37:13] apergos: hi, do you think you can merge these puppet patches? I can't merge them yet :D https://gerrit.wikimedia.org/r/c/operations/puppet/+/721811 https://gerrit.wikimedia.org/r/c/operations/puppet/+/722026 [10:38:34] elukey: fyi pcc should be working again now, sorry for the delay and ping me if its still borked [10:40:51] d'oh! would you hurry up and be a regular staff person? sheesh [10:41:08] anyways, merged, they won't be live until the next run (up to 30 minutes from now) [10:41:39] :P :P [10:42:04] namely says in 40 days [10:55:45] too long :-P [13:03:25] jbond: you rock thanks [13:07:51] lol :) too kind [20:51:22] I seem to remember a website to do code search across mutliple wmf projects at once...anyone happen to know what I'm thinking of? [20:51:50] http://codesearch.wmflabs.org/ [20:52:15] better than download everything and `grep` :-) [21:21:26] hauskatze: indeed it is :P check out `ripgrep` if you haven't, it's a faster version of `grep -R` [21:21:31] cdanis: that's it, thanks! [21:21:51] ryankemper: does it really exist, ripgrep? LOL [21:22:42] hauskatze: https://github.com/BurntSushi/ripgrep indeed it does. it's `rg` on the cli [21:29:29] Any infos on , please? [21:32:31] hauskatze: you broke something [21:32:54] 2021-09-22 21:28:56 [afe54c4c-f00e-4cea-9048-d0b85f8ca264] mw1347 tcywiki 1.37.0-wmf.23 exception ERROR: [afe54c4c-f00e-4cea-9048-d0b85f8ca264] /w/api.php MWException: Format text/plain is not supported for content model wikitext {"exception_url":"/w/api.php","reqId":"afe54c4c-f00e-4cea-9048-d0b85f8ca264","caught_by":"entrypoint"} [21:33:03] What were you doing? [21:33:26] trying to update a script that uses &action=parse in API [21:33:58] Reedy: https://meta.wikimedia.org/wiki/User:MarcoAurelio/stewardlinks.js#L-14 [21:34:21] basically {{NUMBERINGROUP:sysop}} returns the number in the content language [21:34:36] which for the purposes of regex, if not in arabic numbers, says 'error' [21:35:03] I was trying to fix that... unsuccessfully :)