[13:28:34] hi, does anyone know how to have a toolforge built image that has read only files? [13:28:34] (I'm at Wikimania physically) [13:28:36] my problem is for pywikibot the config needs to be read only but it is [13:28:37] -rw-rw-rw- 1 heroku heroku 362 Jan 1 1980 user-config.py [13:55:34] is pywikibot complaining about the permissions of the file? or do you want it to be read-only for other reasons? [14:01:26] yep, pywikibot says `Skipped '/workspace/user-config.py': owned by someone else.` [14:05:30] in [user-config](https://gitlab.com/carlinmack/addletterboxdfilmid/-/blob/main/user-config.py?ref_type=heads) I access the password file (which is read only) via the TOOL_DATA_DIR environ, but I pretty sure I can't do the same thing for the user-config [14:05:31] we should create a sample-pywikibot tool xd [14:05:35] that would help [14:06:17] there's a pre-built pywikibot image from https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/ [14:06:36] (use as https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts) [14:10:43] carlinmackenzie: there's a few commits there to adapt and allow using pywikibot from within the image, https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/-/commits/10406db40/?ref_type=undefined [14:11:24] this one fixes that specific issue https://gitlab.wikimedia.org/toolforge-repos/pywikibot-buildservice/-/commit/06b051dcbbf8525d46d69d743b461d7f261804c2, but you might want all of them too (or use that image directly if you don't need any specific changes/versions/etc) [15:51:15] !log admin upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044 [15:51:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:22:54] Could somebody help me get logged into https://wikitech.wikimedia.org/ I've tried all the passwords I think it might be, but haven't been able to find the right combination, I tried going through the recovery flow at https://idm.wikimedia.org/wikimedia/password/ but can't get that to work either. [16:43:11] roy649: I don't know if this will help you find your way, but your Wikitech username is RoySmith and the password is the same for all uses of your Developer account (Wikitech, Horizon, Gerrit. GitLab, Phabricator with LDAP credentials, IdM, etc). [16:58:46] roy649: based on this question in another channel. if you get actual error messages on idm.wikimedia.org when trying to reset it.. would be good if we report them to the relevant team [18:15:59] our toolforge app emits structured logs to NFS but the files often seem to be corrupted in weird ways. at the moment the first ~250k bytes of my logs are nulls. I'm guessing this is a weird NFS issue? is there a better place to emit logs to? [18:24:42] i guess https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/data/project says it's explicitly not a good place for logs... do we have a place that is good? !help [18:25:35] For toolforge nfs is your only real option, we just ask that you rotate periodically and keep the files from getting to huge. [18:26:12] It's unlikely that nfs is corrupting things because that would break many many things... I don't have much theory about what happened. [18:26:51] we do rotate daily [18:27:00] great! [18:27:22] are all the NFS shares equally bad for logs? [18:28:03] scratch is probably slower... putting them in your tool home is standard, it's where others would look [18:28:58] ok. i guess i need to figure out what else could be zeroing this file [18:29:36] maybe https://wikitech.wikimedia.org/wiki/Help:Shared_storage#/data/project should be updated to reflect the reality that logs are put there [18:29:36] Yeah, or possibly you just logged a bunch of zeros :) [18:29:59] That page is mostly intended for cloud-vps users since toolforge users don't really get options about storage [18:30:17] so one thing we do is during rotation `rm` the log file. i think maybe writes that happen while that is happening could cause weirdness [18:31:46] I added a note to the doc [19:32:37] derenrich: your problem sounds a lot like https://stackoverflow.com/q/64871670/8171 [19:34:25] it seems like puppetmaster.cloudinfra.wmflabs.org isn't getting updates from upstream prod puppetmaster. my VM using it is still at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060399 from yesterday [19:34:42] derenrich, andrewbogott: ^^ logrotate can cause null head padding when the log writing app is not using O_APPEND [19:35:15] that certainly fits [19:38:29] mutante: still? I ran the job that runs every minute and it did something but seems happy (so the timer should've been working too) [19:38:53] mutante: /srv/git/operations/puppet on cloudinfra-cloudvps-puppetserver is showing https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060877 as the current HEAD ¯\_(ツ)_/¯ [19:39:15] maybe it was stuck and I unstuck it just now? I'll watch and see if it lags behind again [19:40:20] it is still at the change from yesterday, yea [19:40:43] mutante: what's the fqdn of the VM? [19:41:00] wikistats-bookworm.wikistats.eqiad1.wikimedia.cloud [19:41:14] grep master /etc/puppet/puppet.conf [19:41:14] server = puppetmaster.cloudinfra.wmflabs.org [19:43:28] the change linked by bd808 is from today but also a couple hours ago [19:43:40] so if that runs every minute it should be even newer [19:46:15] that change in production for puppetmaster->puppetserver is only scheduled for August 12.. otherwise I would be suspicious it's that [19:49:09] mutante: HEAD on the cloudvps puppetmaster is now just one commit behind https://github.com/wikimedia/operations-puppet/commits/production/. The remote is the canonical https://gerrit.wikimedia.org/r/operations/puppet which makes me think the delta is just because the timer hasn't fired again yet [19:49:30] This is probably something messed up with the gitpuppet user... I haven't had the bad manner to push https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056010 through yet [19:50:05] alright, it's not like I am particularly blocked or anything [19:50:21] just wanted to know my merge didnt break it.. but if it does I will get email that tells me later.. it's ok for now [19:51:07] it was about letting puppet manage my cinder volume [19:59:25] I tested deleting my puppet agent cached catalog [19:59:29] but it made no difference, fwiw [21:23:16] @bd808, I eventually got it sorted out. The problem turned out to that lastpass had gotten confused about which credentials to use with which wiki because they all end in wikimedia.org; unless you override it, lastpass guesses they're the same domain. [21:24:27] As for idm, I just wasn't getting any mail, probably because I was entering the wrong username. Why can't it just ask you for your email address, like all other account recovery systems? [21:25:16] I've got several different usernames on several different WMF systems, in part forced because different systems have different constraints on what's a valid username, i.e. case sensitive or not. [21:26:35] I know I'm not the first person to say this, or even the first time I've said it, but the whole WMF auth system is a bit of the dog's breakfast. [21:29:03] I've got accounts on wikipedia, and wikitech, and cuwiki, and ombwiki, and VRTS (and I think there's an associated wiki with that, maybe), and IRC, and Discord, and maybe some other things. It would be really nice if one username/password worked on all of those. [21:29:36] But, anyway, thanks for your help. [21:37:16] roy649: glad you got going again. https://phabricator.wikimedia.org/project/view/6542/ is the place to file feature requests/bog reports for "Bitu" the software behind https://idm.wikimedia.org/ [21:37:23] There is little chance that IRC and Discord are going to become linked to either SUL accounts or Developer accounts. Many things inside the Wikimedia sphere are converging on those two account types however. [21:39:16] SUL is already generally a single sign-on system. Developer accounts are headed that way with the https://idp.wikimedia.org/ CAS-SSO system, but adoption is still spotty [21:39:27] Yeah, I know, those are run by outside entities. My understanding why it can't happen doesn't make it less irritating, however :-) [21:39:56] And having worked on rolling out SSO systems, I understand that it's a non-trivial problem, which also doesn't make it any less irritating :-) [21:40:52] I have credentials for ~15 Developer accounts for various bots and other tools, so I think I understand some of your pain [21:41:53] I think something like "investigate hooking up VRTS to Bitu" could be valid project [21:42:18] and phab? [21:42:42] phab already let's you pick between using either SUL (wiki) or Developer account [21:42:42] or maybe phab already is on my developer credentials? [21:42:43] mutante: s/bitu/cas/ [21:42:59] roy649: with phab it's basically your choice [21:43:12] bd808: ack [21:43:14] Pahb can use a Developer account for same sign-on, but not single sign-on [21:44:03] TBH, part of the problem is lastpass (substitute your choice of pw manager) handles most of this automatically. Which means when something does go wrong, it's so long since I've had to be in the loop, I've long since forgotten the details and have to re-discover them. [21:44:12] this is the classic blunder of our Developer account system: it is backed by LDAP so we connected a number of LDAP capable things to it. [21:44:50] But I'm getting dangerously close to "old make shakes fist at sky and complaints about modern life" territory... [21:44:51] LDAP is not an SSO service; it's a data and authentication service [21:45:17] /me has been burned by pwmanager autofill too often and now just knows the KeePassXC shortcuts to copy username and password [21:45:27] roy649: there are >650 login items in my password manager :) [21:45:51] I use keepassx which let's me copy/paste but doesn't fully automate it. I think I like it this way. And I don't have to relogin THAT often. once a week for google seems the most frequent of all [21:46:24] yep, what lucas said [21:47:15] Bitwarden allows you to override the URL matching per-item, and that lets you match on an entire hostname. That sorts out most of the Wikimedia multiple account problems that I've had - I'm assuming somewhere Lastpass will have a similar setting [21:49:03] Lastpass lets you do that too. It's just that the default is "use the root part of the domain" and if you don't stay on top of that, things get ugly quickly. [21:49:22] roy649: if you actually create a phab feature request though that overrides all the downsides of "shakes fist at sky" though :)