[00:59:44] https://gerrit.wikimedia.org/r/c/operations/puppet/+/721644 is the patch to fix that [01:00:17] Earlier today, after dcausse started to rerun the script removing deleted Wikidata items from the Query Service, I reran the query populating https://www.wikidata.org/wiki/User:MisterSynergy/sysop/empty_items and now items deleted in March(!) are showing up [01:11:24] ryankemper : will a new contributor like me have access to any toolsforge VM via bastion ? [01:12:17] mahir256: that's bizarre! I'm taking a brief look, I don't have a lot of context on that side of wdqs unfortunately so we might need to wait till europe wakes up to hear an answer [01:13:27] SDineshKumar: hmm, I would think so after you've followed these steps: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Quickstart#Getting_started_with_Toolforge_-_Quickstart but not entirely sure [01:14:08] SDineshKumar: btw, I deployed your / our patch and it worked great! I'll have a followup patch to remove those cron resources now that the `ensure => absent` has done its job [01:15:30] thanks ryankemper - i can send the followup patch as its already for you [01:16:06] SDineshKumar: sounds great! [01:17:47] ebernhardson: sorry for the now-way-too-big irc backlog, but I found something else we broke: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721646 Turns out the switch to `/var/log/query_service` might have been a bad idea with how we use `$deploy_name` currently. in any case hacking in a ternary operator will suffice in the meantime [01:35:15] SDineshKumar: I realized I wanted to clean up a couple other things in that file - especially the weird `default` stuff I mentioned ~15 lines above - so I got that followup patch up, feel free to take a look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721647 [01:37:09] looking. [01:44:51] ryankemper : lgtm and nice catch on the comment /type part [01:44:59] am i allowed to review code ? [01:45:13] and publish comments? [01:45:33] SDineshKumar: absolutely, +1 it if it looks good, -1 if shouldn't be merged in current state [01:48:57] SDineshKumar: for example multiple people can give their +1s; often for a certain task or area of the code there's an (informal) key individual whose +1 you really want before merging [01:49:45] So if you were to +1 a patch, and say the patch submitter also wanted to hear back from someone in traffic to look over some caching layer stuff or something complicated like that, they wouldn't merge it until they'd gotten the +1 from someone from the traffic team, etc [01:50:03] i see.. [01:50:20] basically what I'm saying is, don't be afraid to review if you don't know everything going on in a patch, as long as you've got something worth contributing :) [01:55:25] got it. that's really inviting. thanks [01:55:30] but where do i +1 ? [01:56:10] i don't think there's such a button for new contributors. maybe ? ryankemper [01:58:12] yay.. got it. it's well hidden under reply button [02:01:02] ryankemper if its not too late can you please direct me towards some page which says how to contribute to SRE team or CI/CD team - i see this page just says how to contact SRE if help is needed. After some good fishing, I am yet to find a page saying "here's how you can contribute/help DevOps/SRE team" [02:01:17] https://wikitech.wikimedia.org/wiki/SRE/SRE_Team_requests [02:03:41] SDineshKumar: glad you found it, gerrit's ui (and general workflow) can be quite confusing [02:03:59] for example the fact that if you reply to someone's inline comments, you have to manually click the big reply button as well to actually send them out, always trips me up [02:04:33] it makes sense - since it lets you get a bunch of inline draft comments ready, edit them, then fire them off in one batch reply - but was confusing [02:04:58] yes, i was looking for final "publish" button at the end of the screen and then immediately started fishing from the page source - lol [02:05:09] hahaha [02:05:50] SDineshKumar: as for helping SRE or CI/CD, yeah the reason the documentation you're finding is the other way around is because a lot of operational stuff requires access privileges, so there's various things that people need to submit a request to SRE to get done for them (which is understandably frustrating) [02:06:30] The short answer is that, you're already doing SRE work! All the config management stuff is owned by SRE, although devs will often submit patches (especially on search team, since our devs end up doing a lot of the heavy lifting) [02:07:57] But yeah you naturally won't be able to run puppet on production instances (puppet is automatically ran every 30 minutes, but hosts don't uptake changes merged to the puppet repo until an SRE runs `sudo puppet-merge` on a `puppetmaster` host) [02:08:19] so yeah my advice is pick up what tasks you can / that interest you or involve an area you want to learn more about [02:08:54] hmm, interesting. In general, search and research is also heavy work. The next thing that i wanna pick up is - https://phabricator.wikimedia.org/T290608 [02:09:32] as for the longer answer, I'm not super familiar with all of our processes, I know there's a general concept of "trusted contributors" and for example there are community members who have signed the NDA that we (employees) all have to sign, giving them the ability to see wmf private phab tasks (we only use those for sensitive info since we do all our work in public), that kind of stuff [02:10:21] I think trusted contributors might get more access to WMCS as well but not sure [02:10:36] i under - the secret stuff - that needs to be secret for the orgs and everyones' safety. [02:11:51] i understand" - we can't just share our passwords / private keys by saying "sharing is caring". [02:12:05] So I'd probably ask in #wikimedia-tech for the general procedure on becoming a trusted contributor; https://wikitech.wikimedia.org/wiki/Volunteer_NDA has some info on the general process [02:12:56] As that document notes, stuff like requesting production host access is a separate (additional) process, and is presumably only granted to a tiny # of non-wmf individuals, based on the work they're doing, etc [02:13:09] i've signed up an NDA already. is this one more NDA ? [02:14:10] oh.. that was L3 link yesterday. now its L2 link. granting more privi access. cool [02:14:40] i meant the URL to phab to request access. [02:14:53] SDineshKumar: I'm not sure what NDA you've signed already, nor am I familiar with the process in general unfortunately, beyond the fact that I know there's a trusted contributors NDA :P [02:15:17] https://phabricator.wikimedia.org/L2 - I found today - thanks to you [02:15:26] great! [02:15:28] https://phabricator.wikimedia.org/L3 - i've signed this yesterday. [02:16:00] for the L2 link - i just get Access Denied: Restricted Legalpad Document [04:28:31] ryankemper: there are multiple different ndas, and sometimes things are rather confusing [08:29:17] dcausse, zpapierski: I have to record the WikidataCon presentation, and it's really hard to do without an audience. Would one of you be available to just listen while I talk over 2 slides? [08:29:29] sure thing [08:29:35] sure [08:29:48] meet.google.com/eiu-kkuu-muv [09:29:05] Errand [09:38:03] Lunch [10:00:31] lunch [10:20:23] gehel: (when you're back) I have a spicerack part more or less ready and I see that broker data is available in puppet (hieradata) - I'd like to know how to access that data [10:40:39] relocating + lunch break [12:39:54] zpapierski: we'll need a puppet change to write that data to a config file in the appropriate format on the cumin servers [12:40:11] for now, you can just assume that the data will be there [12:40:17] and document in which format you want it [12:40:41] or we can pair to see if we can figure out together how to create that config file [12:42:47] the entry point in puppet is going to be profile::spicerack [12:42:59] which already defines a few config files [13:18:03] ryankemper dcausse any luck finding out why items deleted in March are being returned at https://www.wikidata.org/wiki/User:MisterSynergy/sysop/empty_items ? [13:20:12] mahir256: it's sadly a known issue of the current system that we plan to replace mid-october, I'm not sure that the deletion log has such old items so I don't think I can run the script on that period, but I could suggest running your query to extract this list against https://query-preview.wikidata.org/ (which is the test server used to evaluate the new updater) [14:08:13] gehel: thx! I'll use that to find examples and code around accordingly [14:14:49] hmm, kafka brokers in hieradata are not divided by clusters... [14:15:43] I could divide them by subnet [14:16:06] gehel: are you aware of any clean way of doing something like this in puppet? [14:17:49] zpapierski: there's a function named kafka_config: $kafka_brokers = kafka_config($kafka_cluster)['brokers']['string'] [14:19:02] $kafka_cluster should be 'main', reading the doc you can also pass the 'site' param to get eqiad or codfw [14:20:56] ah, great, sounds what I need, thx [14:21:34] I wonder if there's a place I can get cluster names as well [14:21:54] or is it just jumbo and main? [14:22:17] zpapierski: for us this should be set in "profile::query_service::streaming_updater::kafka_cluster" [14:22:35] test's the logging and test cluster as well IIRC [14:22:44] s/test/there [14:23:04] and each of them (if available) can have eqiad and codfw [14:23:43] yes, but scratch the profile::query_service::streaming_updater::kafka_cluster idea, the cumin host might not have the role required [14:23:48] I'm doing a spicerack change, I assume I should do a complete configuration, since it's not streaming updater specific [14:26:46] zpapierski: sorry, was in interview, reading backlog [14:27:10] no need to apologize, it's an async communication method by design :) [14:28:12] looks like David already mostly answered your question [14:28:15] yep [14:28:37] in the context of spicerack, we won't depend on any of the query_service* stuff, it has to be intependant [14:28:45] this kafka_config method looks good, I just need to get the list of clusters, but I think I can get the keys somehow via lookup [14:29:06] sites I don't know how to get, I'm guessing we're not getting a new one soon, but rather not hardcode them [14:32:43] dcausse: zpapierski, anywhere you can, please use tls kafka connection [14:32:48] one day we'd like to make everything use it [14:32:53] so better to start early :) [14:33:32] I have no idea how to force one python kafka clinet [14:33:37] but I will try :) [14:39:03] ryankemper: https://gerrit.wikimedia.org/r/c/operations/puppet/+/719931 has been reviewed by Cole and is ready to be merged (maybe not on a Friday) [14:39:27] ah, I see that kafka cluster names are hardcoded everywhere as well [15:01:26] gehel, dcausse or anybody who can help: any hint on how to map to a hash in puppet? Particularly I try to achieve something like this (this code isn't correct, of course): https://www.irccloud.com/pastebin/w4vKbAgg/ [15:02:42] no clue :/ [15:03:18] I'm not even sure I understand what you are trying to do :/ [15:03:28] That's a case for writing a unit test! [15:03:38] \o [15:03:39] in a real TDD sense! [15:03:40] that's actually broken, should one map less [15:04:00] I'd love to do a TDD, I have no idea how to do one in puppet [15:04:13] otoh, It's apparently not really possible with map [15:04:15] o/ [15:04:31] I need to finish interview notes first, but can jump in a meet afterward to at least help you setup a first test [15:06:22] if you're confident that we can do it before unmeeting, I'll gladly use some help on this, I'm still scared by puppet :) [15:07:58] zphttps://meet.google.com/qzm-hrgr-xgm [15:08:02] zpapierski: https://meet.google.com/qzm-hrgr-xgm [15:08:14] not confident that we can get it working before unmeeting, but we can try [15:09:40] * ebernhardson also found puppets collection handling atrocious...many failures to get it working right in the elastic cookbooks [15:19:59] puppet is horrible for any kind of data structure [15:20:32] * gehel might be late for unmeeting [15:20:52] * gehel forgot to buy milk and needs to get it before the shop closes [15:21:14] * gehel does not want to imagine how the kids would behave at a breakfast without milk [15:22:54] lol [15:23:21] i suspect if liam didn't have milk at breakfast...much less cereal would be stuck the table when he's done [18:04:32] ryankemper: lol, the 'If this was merged to master please fix' sounds troubling, but indeed the commentary included makes it look pretty benign. I agree the change is to make it always be the per-instance title. I suspect without looking closely that the 'default' case was dead code [18:07:46] ebernhardson: yeah it was probably already clear but I mentioned it because the comment indicated to me that it was vestigial, not that it sounded like an actual problem haha [18:07:58] yeah all the existing resources we have use the new name and not the old one which is another strong indicator that it's dead code [18:14:06] ryankemper: regarding the find command and gc_log_subpath, i think the problem there is find by default doesn't follow symlinks, two other options would be to make it /var/log/query_service/ which does resolve the symlink (by explicitly starting inside the directory) or turn on following symlinks [18:14:19] * ebernhardson is slowly reading backscroll and looking at things :P [18:15:18] i've always wondered, there isn't really a best practice around if a dir should be referred to as /a/b/c or /a/b/c/ [19:13:24] I usually try to use /a/b/c/ (with final slash) as a defensive move, depending on how it is going to be used–cut'n'paste or concatenated by code—because if mistakes are to be made, it's better to end up with /a/b/c//d than /a/b/cd