[07:45:10] https://berthub.eu/articles/posts/practical-peg-parsing/ looks interesting (but not alas in Debian yet) [11:40:42] jynus: Please feel free to merge my prometheus patch with yours. [11:41:03] oh, true, I forgot to hit enter [11:41:57] btullis: it is the T362904 one, right? [11:41:57] T362904: Scraper: track with production Prometheus - https://phabricator.wikimedia.org/T362904 [11:42:06] Yes, that's the one. Thanks. [11:42:16] ongoing... [11:43:08] Done. [11:43:16] Thx [12:57:24] I'm seeking a simple seal of approval for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1023423 [12:59:46] +1 for me [13:01:34] thank you fabfur, appreciate it [13:02:12] hello on-callers [13:02:32] I am about to move the Cassandra instances on AQS eqiad nodes to PKI, lemme know if there is a reason not to [13:21:05] ok proceeding :) [13:45:17] roll restart in progress! [13:45:21] so far all good [13:45:30] Cc: urandom: [13:49:00] 👍 [13:55:10] note: the probe firing right now for aqs are related to the new blackbox exporter checking instances not yet on PKI [13:55:24] I have open nodetool status and all is UN [15:05:45] completed! [15:16:39] nothing to report for oncall! [15:42:58] The Puppet style docs ( https://wikitech.wikimedia.org/wiki/Puppet/Coding_and_style_guidelines#Hiera ) say "A good example is, in our codebase, ganglia_clusters" but AFAICT that is no longer true; is there a suitable replacement example? [15:49:40] Emperor: wikimedia_clusters [15:54:50] thanks [16:49:34] Over the weekend I discovered https://github.com/brookhong/Surfingkeys - I'm a pentadactyl refugee and never liked vimium/tridactyl et al. This one seems to be working pretty well for me so I'm tentatively recommending it for anyone in the same boat [16:50:02] (and while I like qutebrowser a lot it's never really been a good fit though I haven't used it in a few years) [16:52:01] brett Thanks for sharing! I always feel like I'm way behind in discovering/using desktop optimizations [16:52:48] inflatador: I've been a little more behind in certain ways since I've been totally happy with sway et al, so I get that. [16:53:16] I've never been particularly happy with the browser situation: Pentadactyl was horrendous for firefox performance but the webextension equivalents are difficult sometimes... [16:53:30] So it's possible I'll just throw my hands up with whatever problems arise and give up again [16:56:45] Is there a test gerrit instance I can use the API for? I see https://wikitech.wikimedia.org/wiki/Gerrit/test_instance but it's flagged as OOD and it seems https://gerrit.devtools.wmflabs.org/ is no more [17:01:59] "What's your top 3 GUI app/CLI app/browser extensions" would make a nice SRE survey [17:02:00] brett: that was deleted just last week [17:02:22] brett: there is another test instance in that same project but I guess the proxy isnt setup properly [17:02:31] mutante: Bad timing! [17:02:51] Thanks for the info [17:03:52] unfortunately both have been deleted :/ [17:04:12] we should recreate one with the right OS [17:04:32] That would be lovely. Anything I can do to help with that? [17:04:54] I'll file a ticket [17:06:22] yea, just that because I have something to point to that there is an actual need. thank you. will make sure it's seen [19:03:09] dwisehaupt: Are you around to coordinate about T361595? I've built you a new puppetserver but switching things over will require a bit of tinkering with the puppet git checkout and you seem to have quite a lot of work in progress there. [19:03:10] T361595: Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595 [19:08:20] andrewbogott: howdy. i'll have a look at that. we are getting really close to migrating the community-crm instance into the prod VPS realm. (hopefully this week) then we can look at what we can do there. it's entirely possible that we will be able to destroy it and start fresh. [19:09:03] there may still be a reason for us to run our own puppetmaster in cloudvps but i'm hoping to eliminate those reasons. [19:10:00] ok -- can you accept and comment on that task so I know I can ignore it next time it crosses my screen? I'll remove the server that I just built if you dont' want to adopt it so things don't get out of sync. [19:10:35] sure thing will do when i'm out of this meeting. thanks for the follow up. [19:13:07] thx [22:17:59] got a system with a user who doesn't otherwise need full root but would like to read puppet logs. if you are in SRE you can just read that without using sudo or root. the reason is we are all in the "adm" group. options: a) add non-SRE user to adm but only on their system b) change mode on puppet.log making it readable by anyone on that box (no other users) c) add sudo privs line for their [22:18:05] actual admin group just to read puppet logs d) give them full root on their machine through a new group ... ? I think it's c) but I have to ask for group change review in almost any case [22:25:34] do puppet logs ever contain secrets that you *should* need root to access? like sensitive config file contents, or hiera values, or whatever [22:25:46] I can't think of any reason why they would but I think that's the operative concern [22:26:26] assuming we're okay with it overall though, I agree C or D is the right mechanism [22:26:53] if a file has secrets, you need to explicitely set `show_diff => false` to not log them, and there've been plenty of instances where we've not done that [22:27:10] yeah [22:28:09] I'll leave it to mo.ritz but puppet logs might contain credentials in the diff, it happens... [22:29:39] I think in the long run it Would Be Nice™ to have a log we can safely share with unprivileged users, so we can proceed in a least-privilege kind of way, but under the circumstances it sounds like that should probably be a request for full root on that machine [22:30:07] or, at least, we should follow the same approval process that we would for full root, even if we're "only" granting access to that file [22:33:08] rzl: I think we limited it because there could be a diff that leaks a secret [22:33:31] I have another option meanwhile: [22:33:55] that would be this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1023517/1/modules/admin/data/data.yaml [22:34:09] add the admin group to adm, not just SRE [22:34:23] but of course it would still be just on the systems that this admin group is on [22:34:26] that would do it globally, not just on the stewards* boxes [22:34:43] the group only exists on steward boxes [22:35:11] ah, right [22:35:30] direct yaml anchors aren't affected by that [22:36:41] well, the whole question was to avoid "give up and just give them root" but seems like that it is .. like everyone else [22:37:49] could still just add sudo line for the one log file.. have to get review from IF for that though as well [22:37:52] thanks