[07:20:28] PSA SOSP 2021 videos are online https://www.youtube.com/channel/UC6jgDZsRWpE9uJKe202F0iw/videos [08:41:49] Thanks godog! Some of those look really interesting :) [08:43:27] hehe indeed topranks, I've added a few to watch later myself but can't vouch yet [10:38:44] elukey, btullis: I'm ready to handle T292563, we're mostly waiting on your side I think [10:38:45] T292563: Request creation of data-engineering VPS project - https://phabricator.wikimedia.org/T292563 [10:55:45] Thanks arturo. I haven't really had a chance to talk to the rest of my team much about a deprecation schedule for the 'analytics' project in WMCS yet. I think that the new one only really needs a very low quota of instances/CPU/RAM at the moment though, so if you could set it up in parallel with the analytics project I think that would be really helpful. Something like 4 or 6 small-ish instances would be about the max right now. [10:56:27] btullis: ok, if you remark that in the phab task, I can get the project created today [11:01:35] Thanks. Will do. [11:23:23] arturo: Done. Many thanks. [11:35:58] btullis: thanks, you are all set [11:39:09] Awesome. Cheers. [13:27:39] <_joe_> elukey: I see you've removed a few images from production-images; should we also clean up the registry? [13:29:21] _joe_ ah snap I cleaned up only docker on deneb, and forgot to ask how to do the final clean up [13:29:24] yes yes [13:30:07] <_joe_> elukey: docker-registryctl -h [13:30:15] <_joe_> on deneb :) [13:30:21] ack thanks, will check in a sec [13:32:45] and if you want to be really nice debmonitor too :D [13:32:46] https://wikitech.wikimedia.org/wiki/Debmonitor#Manually_remove_an_image_from_DebMonitor [13:36:05] <_joe_> volans: don't they get garbage-collected oif they're not around for too long? [13:36:33] yes, but stick around for another 3 months or so IIRC [13:36:59] 90 days [15:03:42] herron, godog, cwhite, we suspect https://gerrit.wikimedia.org/r/c/operations/puppet/+/731976 of breaking puppet on cloud VMs. jbond proposes something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/732715/ [15:04:55] andrewbogott: SGTM [15:20:04] godog: that doesn't seem to have helped. I'm in the middle of something so haven't really thought about the issue, hoping that herron will appear to fix it [15:26:57] andrewbogott yup I'm looking at it [15:27:08] great, thank you! [15:52:46] andrewbogott: I'm just going to revert the original change and work on fixing this while things aren't broken [15:57:12] andrewbogott: done [16:04:46] herron: ok, thanks -- let's revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/732715/ too so we have a clean slate [16:10:26] will do, although I may leave in place for a short time while working out a fix [17:30:57] andrewbogott: do you have any ideas why the cloud compiler wouldn't reflect the same error on the patch that was reverted? I was expecting https://puppet-compiler.wmflabs.org/compiler1003/31826/ to fail, but it's showing NOOP [17:32:45] herron: I don't have total trust that the hiera search path is the same in pcc as on actual VMs, but I don't know anything smarter than that. This task was meant to articulate that lack of trust... https://phabricator.wikimedia.org/T255787 [17:34:58] andrewbogott: thanks that helps, hmm I wonder how to best test this without live hacking. guess I'll need to borrow a project puppetmaster [17:36:00] Use mine: abogott-puppetmaster.testlabs.eqiad1.wikimedia.cloud and abogott-puppetclient [17:36:11] if there are local diffs there just reset, nothing interesting is happening there [17:37:02] herron: ^ [17:37:29] andrewbogott: nice thanks! [17:52:45] herron: its an intresting question i havn;t looked in depth but i do notice that the diif https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/31826/console looks wiered i wonder if this is because the change has now been reverted [17:54:15] also worth noting that i had highlighted this class as one that probably dosn't need to be on all cloud instance i.e. moved to profile::base::production (see email, hoping to send on monday :)) [17:57:12] jbond: yeah it's weird for sure, since the original patches and attempted fixes have been reverted I was expecting to reproduce the error in the compiler but it's a NOOP. does seem like the cloud compiler is using a different hiera config or something like that [17:57:27] ah yes that would be nice as well [17:57:49] herron: have you created a revert of the revert and tried that? [17:58:15] last i looked pcc is using the same hiera striuctre, including the use of the enc to pull stuff from horizon [17:58:32] i rember because i was suprised :) [17:58:52] no I haven't but thanks will try that. right now I'm experimenting on andrew's puppet master/client [17:59:08] np ill create a revert revert, im curious :) [18:00:01] jbond: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732701 [18:00:23] https://gerrit.wikimedia.org/r/c/operations/puppet/+/732700 lol just beet you ;) [18:00:33] ah you beat me to it! dueling reverts haha [18:00:45] https://puppet-compiler.wmflabs.org/compiler1003/31831/ [18:00:52] it dose show the error [18:01:38] ah! TIL [18:01:42] ill make a note of the other case, it probably pcc and git getting confused because the change had been reverted [18:02:02] makes sense yeah, thanks that's really good to know [18:02:16] its new to me as well, guss dosen; come up much [18:26:40] jbond: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732700/ should do the trick [18:30:43] herron: can yuo run a pcc with 're:.*deployment-prep.*cloud$' i think there may be some instances there tha have this class enabled? [18:31:02] of to eat food in a sec but will be around later, however feel free to merge if things look good [18:31:07] jbond: sure [18:31:12] ok [19:04:17] cleaned up prod Icinga CRITs. it's clean again, except that one thing: the netbox reports [21:29:46] why does Google Groups let us search for conversations with "No action needed" and "Is complete" and "Is duplicate" but not the opposite of that? Who wants to know what stuff does NOT need work.. we want the stuff that needs action and is not complete of course.. duh? [21:30:26] or how does the negation work, I tried to stuff a ! into multiple places but not what I expect [21:36:24] adding "essential subsea cable repair operation" on November 1st. location: "between Bude, UK and Brookhaven, US" :p [22:10:43] what really is a "formal collaborator" of the WMF? Not employee, not contractor, not volunteer? [22:20:28] mutante: try it with - instead of ! [22:22:27] or use `is:unresolved` which I think is what the wikitech instructions link to [22:24:29] rzl: thanks, will try -. meanwhile I found out if I just close that tab and reopen that link from wikitech another time, I actually see the updates and "no action needed" _does_ translate to "resolved", so I am good [22:25:19] yeah, there is a fair bit of caching and/or eventual consistency [22:28:29] I dont think - is working but I got it, what I really needed was just the caching and then the wikitech docs are still correct unlike I thought