[04:27:34] RhinosF1: correct, they don't do strict host key checking [09:03:53] fyi. I'm going to start merging a bunch of cleanup patches for puppet, if you see anything breaking let me know [10:35:58] is something wrong with ldap? https://phabricator.wikimedia.org/T298508 [10:39:43] there's some non-usual traffic starting around 8-9 hours ago: https://grafana-rw.wikimedia.org/d/DnxQ26qmk/ldap?orgId=1&from=now-2d&to=now&forceLogin=true&var-dc=eqiad%20prometheus%2Fops&var-instance=seaborgium [10:40:18] oh, indeed [10:42:46] is there anything interesting in seaborgium's ldap logs? [10:52:29] dcaro: fyi i just got this error in horizon https://phabricator.wikimedia.org/P18363 possibly related? [10:52:50] jbond: that sound related yes [10:56:39] I don't see anything useful in the slapd logs :/ [10:57:58] would https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/751405/ help? [10:58:26] me neither (other then it may be usefull to index modifyTimestamp) [10:58:46] taavi: maybe? xd [10:59:01] it's weird that it does not log error responses (or I don't see them) [11:03:16] jbond: hi, I might need your expertise, I set up tendril-legacy.wikimedia.org as replacement for tendril.wikimedia.org (the legacy will be shut down in a month) but it seem CAS doesn't recognize it [11:03:26] "Application Not Authorized to Use CAS" [11:04:21] I suspect it needs its own certificates again for idp but I have no idea how that works, TLS certs for web are working fine [11:07:43] * jbond looking [11:07:58] Thanks [11:11:25] Amir1: this is all working just needed to run puppet on the active idp server (idp2001). the fix was https://gerrit.wikimedia.org/r/c/operations/puppet/+/751380/5/hieradata/role/common/idp.yaml, however i wonder if the following may make more senses https://gerrit.wikimedia.org/r/c/operations/puppet/+/751406 [11:12:16] ooh, I thought that's standby [11:12:38] jbond: the follow up? I don't think so, we are removing the whole thing in a month [11:12:42] jbond: this is making alert1001 fail to run puppet (from the cleanup) https://gerrit.wikimedia.org/r/c/operations/puppet/+/751407 [11:12:58] currently it's just a static page: tendril.wikimediaorg [11:13:06] tendril.wikimedia.org [11:13:29] Amir1: currently the active one is dicteded byt the cnam in dns so always best to check. its on idp2001 at the moment due to an ongoing upgrade. [11:13:41] I see, thanks [11:13:53] re the follow up, wqithout it then the current tendril will have the same error untill its removed. if thats fine no problem ill abandon [11:14:25] ahh i see yes ill abandon my CR [11:14:51] dcaro looking [11:14:53] Thanks for the idp note. [11:15:14] jbond: On that patch, I'm not sure if setting source to undef just ignore source [11:15:15] Amir1: no problem [11:16:16] dcaro: yes with that it will use the default [11:17:18] its, imo, one of the stupidest choices made in puppet (and that list is not short ;)) [11:19:04] * jbond just noticed that is an absent should be easy to fix, checking the actual error on alerts [11:23:50] dcaro: tbh its two files on two serveres i would just remove them manually [11:24:03] xd [11:24:08] fair enough [11:25:14] jbond: updated the patch [11:26:14] +1 thx [11:31:08] it seems it's cleaning up the config 'Notice: /Stage[main]/Nagios_common::Commands/File[/etc/icinga/commands/check_graphite_freshness.py.cfg]/ensure: removed [11:31:09] ' [11:31:29] but not the plugin script xd [11:31:33] so only one file to remove [11:32:55] jbond: btw for mediawiki/* repos you're expected to +2 approved patches instead of +1 and waiting for the author to merge [11:33:14] hot take: absenting resources properly requires more effort than it's worth [11:35:44] taavi: noted thanks [11:36:29] joe: completely agree [12:11:24] taavi, jbond: about the ldap, I'm not sure this will help, but it might https://gerrit.wikimedia.org/r/c/operations/puppet/+/751417 [12:11:39] I'll have a look after the current mw backport window [12:20:14] dcaro: lets give it a go and see, simple to revert [12:46:41] it did not help with the horizon issue :/ [12:47:45] dcaro: for account creation the error code is apparently "Constraint violation" [12:51:28] taavi: interesting! hmm, still matches the size_limit issue though (I think, will investigate) [13:40:18] as this seems to have happened on the year change it could it be a 2022 date int overflow somewhere? [14:06:39] jbond: https://mobile.twitter.com/Ax_Sharma/status/1477332270730158084 ? [14:06:48] Could be something similar? [14:07:26] RhinosF1: thats what i was thinking of yes but i dont see anything obvious [14:15:12] jbond: I don't know anything about ldap sadly [14:15:28] or how wikitech's works [14:16:08] RhinosF1: np, thanks anyway [14:26:38] wow, that's scary [14:28:18] btw. jbond can you ensure you were able to get the list before? [14:30:00] dcaro: i cant be 100% on that specific project but i knwo that i view the puppet-dev list recently and that gets the same error [14:30:53] ack [14:31:02] (just making sure xd) [14:49:57] joe: currently our envoyproxy module sets a hardcoded LimitNOFile of 65k.. that's far from enough for Traffic needs... I'm wondering if I should refactor envoyproxy to allow a custom systemd unit or just increase globally that LimitNOFile limit [14:52:09] it's pretty easy to add a systemd override in the profile where you're using envoy [14:52:51] that should be the easiest route [14:53:34] actually I'm already doing that for the OCSP stuff [14:53:40] so yeah [14:53:46] LimitNOFile= [14:53:55] LimitNOFile= [14:54:03] ack, thx <3 [14:54:36] vgutierrez: what envoy version are you installing rn? [14:55:01] I should get to finish the v3 transition for discovery outside of k8s and we should move to the 1.20.x branch at least [14:55:14] 1.18.3 from our envoy-future component [15:57:23] ccccccvkvhgbvcbefjiiuuvcjirjvbrdreltndviruki [15:57:33] Sorry. :( [16:01:46] yubikey? xd (been there) [16:06:26] * bd808 considers adding a stashbot function to say "Hello yubikey!" whenever a message contains "cccccc" [16:07:08] 🐈? [16:22:03] Yeah. :'( [20:37:04] Who would be most likely to know about/ set up bullseye apt mirrors? [20:44:15] andrewbogott: what are you trying to mirror? [20:44:34] was about to paste https://gerrit.wikimedia.org/r/q/owner:jhathaway%2540wikimedia.org but there he is :) [20:44:49] :) [20:45:35] jhathaway: Maybe nothing? I was going to ask why we don't have a mirror for the bullseye security repo but now I'm not so sure we have a mirror for any of the other versions either [20:46:12] I'm gathering context for https://phabricator.wikimedia.org/T291168 -- at the moment I think that it's just cloud-init putting invalid values into sources.list (and nothing to do with mirrors existing or not existing) [20:46:51] looking... [20:47:32] I imagine this is why we don't mirror security: https://www.debian.org/security/faq#mirror [20:48:51] Yeah, I think 'for some reason, there are no Wikimedia mirrors for the Debian security repos' is just a red herring there [20:49:02] so it sets the template to be used to apt/base-apt-conf-new.erb if something is bullseue [20:49:20] but values inside that template don't get filled if it's not in prod [20:49:45] I believe taavi is correct on why we don't mirror security [20:49:48] which would result in files looking like that [20:51:06] mutante: I'm not following... what is the 'it' in your initial sentence? cloud-init? puppet? [20:51:26] as far as I can see apt/sources.list isn't puppetized at all (at least on cloud-vps) [20:51:39] andrewbogott: puppet, modules apt. as in https://phabricator.wikimedia.org/rOPUP96a3cc3f689919a1f7034ab7cbc80a3f7f794331 [20:51:56] ah yeah, so probably unrelated to the cloud-vps issue [20:52:22] although we /could/ puppetize that file and maybe that would make this all moot. taavi what do you think about that? [20:52:48] If it's not puppetized then why does it have "deb {{security}} {{codename}}/updates main" with those placeholders [20:52:52] mutante: we're talking about /etc/apt/sources.list? [20:53:09] `profile::apt::manage_apt_source: false` [20:53:19] depends on how many people have custom things in the main sources.list file [20:53:23] is set in cloud.yaml [20:53:40] a quick phab search reveals https://phabricator.wikimedia.org/T264311 [20:53:45] * andrewbogott doesn't have to run 'git blame' to know he is to blame [20:54:21] hm [20:54:25] andrewbogott: yes, that's right. it's directly sources.list, not dropped into sources.list.d. in production [20:55:11] taavi: what do you think? sed, or properly resolve T264311? The latter would probably require some systematic way to detect stray entries in that file and move them to sources.list.d [20:55:12] T264311: Prepare for puppetizing /etc/apt/sources.list - https://phabricator.wikimedia.org/T264311 [20:55:52] I'd prefer doing it "properly", but at the same time I don't know how much effort that needs [20:56:21] It would be nice to reuse the existing puppetry if possible [20:57:13] taavi: and 'properly' is integrate with prod puppet? [20:57:18] yes [20:57:20] ok [20:57:37] take a look at "class apt" [20:57:41] well I'm already looking for a procrastination task so this is as good as any! [20:57:52] you already include apt::noupgrade here? modules/profile/manifests/base/labs.pp: include ::apt::noupgrade [20:57:59] maybe you can just include all of apt [20:58:16] mutante: the only concern is clobbering existing customizations in sources.list [20:58:39] specifically, it looks like anytime anyone tries to use docker they start by adding an entry to sources.list which it would be rude to remove [20:59:22] andrewbogott: the nicest solution I can think of would be.. first write something to detect non-standard sources files.. then tell those users to move their entries to custom files dropped in sources.list.d or move them for them.. then puppetize sources.list [20:59:39] yeah [21:00:41] maybe first just detect how many people do it and what kind of stuff they use [21:01:16] if it turns out there are 2 or 3 standard cases.. can puppetize that too [21:01:55] mutante: I think everything you're saying is already discussed on T264311 but now I'm nervous that you're saying something else unrelated that I'm missing :) [21:01:55] T264311: Prepare for puppetizing /etc/apt/sources.list - https://phabricator.wikimedia.org/T264311 [21:03:07] andrewbogott: no, it does sound pretty much exactly like what I said without me being aware of that ticket [21:03:26] great :) [21:03:39] I think taavi linked it but maybe that was in a different channel [21:04:21] I am thinking maybe you can greatly reduce the number of "different" sources.lists by doing some | uniq | sort and removing whitespace etc [21:04:37] like.. are they REALLY all different in what they fetch..or just format [21:06:04] the docker thing does indeed sound like the most common one, yea [21:06:18] Other than docker I bet they're pretty much all the same, modulo changing order and cloud-init comments [21:06:45] yea [21:07:14] and as moritz mentioned on the ticket, changing sources.list doesn't break any existing software, it only breaks future by-hand installations so it's a pretty safe thing to change [21:07:26] worst case is that a user will think, huh, I thought I already added this [21:08:04] yea, but if the template has the "this file is puppetized comment" or something even more obvious for those who never heard of puppet.. then that is good enough [21:08:28] maybe just link to the ticket in a comment in that file [21:08:41] * andrewbogott nods [21:10:37] I think a # MANAGED BY PUPPET comment plus a link to the source template is always helpful in a config file [21:11:08] docker's docs use a sources.list.d file these days [21:12:21] the problem is that running a copy-pasted `echo foo | sudo tee /etc/apt/sources.list` doesn't show you any comments [21:13:49] apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common [21:13:52] curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - [21:13:54] add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" [21:13:57] this is the "official" way [21:14:00] that I use [21:14:24] https://docs.docker.com/engine/install/debian/#install-using-the-repository is what I found [21:14:35] so you could exec add-apt-repository nowadays [21:14:41] and not worry about the details below that [21:14:45] mutante: cool, I already have a list of affected hosts so I will run that shortly :) [21:15:29] https://phabricator.wikimedia.org/P18392 [21:17:19] just as soon as I can figure out how to use a hostfile with cumin :( [21:18:45] you'll want to add some -y to that so it doesn't ask you interactive questions [21:20:33] well.. andrewbogott I don't know if you want to follow that for all of them. see how it first removes the docker and docker.io packages. that is to make sure to use "docker-ce" instead, community edition and not conflict with that. while I would do that for fresh VMs.. messing with existing install of users is different [21:20:35] can I give puppet a commandfile or do I need to do some echo | thing [21:20:48] s/puppet/cumin/ [21:21:53] I think all I need is the middle three lines [21:21:55] https://www.irccloud.com/pastebin/59GTdZAb/ [21:22:58] *nods* yea, seems reasonable [21:24:36] Hey, I have great news: add-apt-repository addes an entry to sources.list [21:24:40] it doesn't create a .d file [21:25:10] so none of this helps, I will just use 'echo' :) [21:25:10] :) the best part is you don't have to care where it puts them, I guess [21:26:01] andrewbogott: add-apt-repository - Adds a repository into the /etc/apt/sources.list or /etc/apt/sources.list.d or removes an existing one [21:26:04] or [21:27:08] Is there a switch to force it to create a .d file? I sure don't see one [21:28:13] it seems like it would do that if you give it a "ppa:.." format for those PPA repos as popular in Ubuntu [21:28:22] hrmm [21:28:53] In the second form, ppa:/ will be expanded to the full deb line of the PPA and added into a new file in the [21:28:56] /etc/apt/sources.list.d/ directory. The GPG public key of the newly added PPA will also be downloaded and added to apt's keyring. [21:29:07] sounds cool about the key, but only for PPA stuff [21:29:42] * andrewbogott uses echo [21:29:54] yea, just using echo is the same in this case. but this would be smart if you wanted to _remove_ repos the proper way [21:30:25] yeah, that's how a read the docs as well [21:39:08] it depends on profile::apt::purge_sources in Hiera whether it's true that users can drop their own files in sources.d or those get purged as well. But only deployment_prep is different in Hiera in the repo at least. [21:39:14] cloud/eqiad1/deployment-prep/common.yaml:profile::apt::purge_sources: true [21:39:17] cloud.yaml:profile::apt::purge_sources: false [21:41:23] huh, dang [21:41:46] I guess I need to make my comment longer and more confusing [21:43:52] updated [21:52:22] thanks for your help with this, all. I will probably switch it on in the morning [21:53:16] yw! sounds good. I added Moritz as well.