[08:23:38] !admin set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`) [08:23:45] !log admin set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`) [08:23:46] xd [08:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:59:00] !log tools.speedpatrolling deployed edc0f1020 (update dependencies, Flask+Werkzeug 3) [10:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL [10:59:51] !log tools.wd-image-positions deployed 9d55c80b99 (update dependencies, Flask+Werkzeug 3) [actually ~10 mins ago but I forgot to log it] [10:59:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [11:12:52] !log tools.quickcategories deployed 23486f8772 (update dependencies, Flask+Werkzeug 3) [11:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [12:18:59] !log tools.quickcategories deployed d174a2b28c (support ProofreadPage index pages; should resolve background-runner crash loop) [12:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [12:53:47] I'm giving Tool Forge's new buildpack support a go but it seems to fail to select a buildpack for my small Python project although I have a requirements.txt in the root directory of my repository(https://codeberg.org/abbe98/wikidata-status-updates-atom-feed). [12:53:49] [12:53:50] Anyone who knows what's going wrong or have suggestions for how I can debug it further? [13:05:00] Abbe98 what error are you getting? [13:05:50] @Abbe98: the name of your requirements.txt file seems to actually be named 'requirements.txt ' with an extra space at the very end of it [13:06:23] Good eye xd https://codeberg.org/abbe98/wikidata-status-updates-atom-feed/src/branch/main/requirements.txt%20 (%20 should not be there) [13:07:16] That's an impressive find! Thanks! [13:08:21] !log cloudvirt-canary remove canary VMs from cloudvirt-wdqs hosts T346948 [13:08:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL [13:08:24] T346948: Move cloudvirt-wdqs hosts - https://phabricator.wikimedia.org/T346948 [13:29:10] !log  wikispeech wikispeech-producer.wmcloud.org: Updated MediaWiki to 1.39.5. Installed the following extra extensions: UniversalLanguageSelector, PluggableAuth, Wikispeech, WSOAuth. Applied workaround in https://phabricator.wikimedia.org/T337827#8938778 for WSOAuth. [13:58:29] !log tools.mabot Update mabot to pywikibot stable 8.3.3 [13:58:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mabot/SAL [14:23:56] I'm working on Catalyst, thinking about authentication, especially into an app that interacts with Cloud VPS/Openstack. Is there an OAuth/SUL-like workflow for authenticating a user against Cloud VPS and returning them to the app with their token? [14:26:02] !log tools.pagepile-visual-filter deployed d1c2f1c49f (update dependencies, Flask+Werkzeug 3) [14:26:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile-visual-filter/SAL [14:26:45] kindrobot: Openstack uses ldap directly for authentication, we have idp.wmcloud.org also, that we use for karma(alertmanager) [14:26:57] it uses ldap underneath too [14:27:14] what do you want to do "as the user"? [14:29:19] !log tools.ranker deployed 358730fbbf (update dependencies, Flask+Werkzeug 3) [14:29:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [14:32:06] !log tools.ranker rm -rf www/python/venv-3.9/ # unused [14:32:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [14:32:36] !log tools.translate-link deployed b907a429f9 (update dependencies, Flask+Werkzeug 3) [14:32:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [14:32:55] !log tools.translate-link rm -rf www/python/venv-3.9/ # unused [14:32:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [14:33:11] !log wikispeech wikispeech-producer.wmcloud.org: Updated MediaWiki to 1.39.5. Installed the following extra extensions: UniversalLanguageSelector, PluggableAuth, Wikispeech, WSOAuth. Applied workaround in https://phabricator.wikimedia.org/T337827#8938778 for WSOAuth. [14:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [14:39:19] dcaro: most of the time make kubectl commands to a cluster therein, so maybe initially get some kind of kubectl certificate...through magnum...? [14:40:53] I'm not familiar with magnum myself xd, so probably a Roo.k question, but I'd expect you to have one auth for your backend to openstack, to create the clusters, that will get you a cert somehow to access the k8s cluster as admin, and from there you might want to create more certificates for each user (or give them full admin) [14:41:12] maybe andrewbogott knows too [14:42:07] I don't know much, I agree that that's a rook question [14:42:57] Yeah, that sounds about what I would expect. If we're going the one cloud vps project per team approach, I'd probably want to confirm the user is part of the project (for some access control). Is that something I could do by e.g. getting a token through idp.wikimedia.org and getting a list of their projects? [14:43:36] ...through the OpenStack API [14:44:01] Is there general documentation for integrating with idp.wikimedia.org? [14:45:19] not really xd [14:45:26] Hehehe [14:45:45] it should be really really similar to https://wikitech.wikimedia.org/wiki/CAS-SSO [14:45:53] (it was before the development server for it) [14:46:52] Would that be a way to go though? ... (1) do a handshake with the IDP (2) get a token or LDAP equivalent (3) ??? Openstack (4) Profit [14:49:59] you want to act as that user on openstack then? [14:50:15] users don't have privileges to create projects [14:50:19] (just fyi) [14:51:27] Yeah, I assume they'll be a part of the project already. Really I'd just want to list their projects. [14:52:50] hmm, one step back please, what you want, is given a user (ldap/openstack) get the list of openstack projects they are member of? [14:56:29] kindrobot: so you have two options, mostly [14:57:59] (1) is to simply use the groups (or memberOf, don't remember the exact name) that idp.wmcloud.org gives you. each openstack project has a matching project-$NAME group. the downside is that it doesn't include inherited membership (= cloud vps admins who have access to all projects without being explicitely added to them), and you can't distinguish [14:57:59] between readers and members [14:59:31] (2) is to take the username idp.wmcloud.org gives you, and use a service account (either novaobserver or a catalyst-specific one) to ask the openstack identity api (keystone) which roles the user has in a specific project. this is more accurate, but needs more steps [15:00:40] regardless of the option you choose, you can't act _as_ the user to anywhere (openstack or ldap), you need to use a service account if you need to interact with a system that needs login. that's a general constraint of the environment that I'd really rather not change [15:06:55] OK, this is very helpful. Thank you both. [15:07:11] I'm sure I'll be back with more questions :> [15:33:23] If this becomes something that more folks want, adding an API endpoint equivalent of https://openstack-browser.toolforge.org/user/bd808 would be pretty easy [16:26:55] kindrobot: there's some docs in https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Magnum describing things like how to setup a template. And get a credential. [16:26:55] https://github.com/toolforge/paws/blob/main/terraform/123_7.tf is an example of doing all of that with terraform [18:04:23] !log tools.integraality Deploy da99820, 801ddeb, f2ffd22 (T312729) [18:04:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [19:28:34] !log tools.integraality Deploy 3ba5e84 (T312729) [19:28:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:06:08] Hello everyone. https://ws-export.wmcloud.org/ is down. [20:06:27] Is this related to https://wikitech.wikimedia.org/wiki/Incidents/2023-09-29_CloudVPS_vms_losing_network_connectivity ? [20:06:49] And BTW, https://wikitech.wikimedia.org/wiki/Incidents/2023-09-29_CloudVPS_vms_losing_network_connectivity contains code with hardcoded username and password ... [20:07:17] As a consequence no export is possible from any of the wikisource sites. [20:08:26] !help [20:08:26] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [20:11:23] looks like that's the wikisource project, https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource [20:17:51] https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=wikisource <-- instance down [20:32:02] Guest963: hopefully you can no longer see https://phabricator.wikimedia.org/P52761 [20:32:09] User10: even [20:32:21] (If that's not the password you talk about, please PM) [20:33:58] (fwiw that password is public https://gerrit.wikimedia.org/g/labs/private/+/797ab8c73f1158b270fab3751a5383fc5fec1dd0/modules/passwords/manifests/init.pp#49 ) [20:34:16] labs/private isn't [20:34:32] re:password, thanks, this was only quick observation. [20:35:06] I'd much rather have people reporting passwords when they see them, those passwords should probably all be prefixed with ThisIsPublic or something [20:35:19] AntiComposite: quite possible that they are faked. And yes prefixing this public is good. [20:35:20] ^^ [20:35:36] It's hard to tell otherwise if it's a fake password [20:38:03] AntiComposite: https://phabricator.wikimedia.org/T348067#9222540 [20:38:32] User10: 100% always better safe than sorry [20:38:43] AntiComposite I cannot see the password anymore :) [20:39:12] User10: thats because I protected the paste to acl*security for now [20:40:14] Ok everyone, thanks for repyling, I guess wikisource admins will be or have been alerted about instance down... [20:40:33] I'd file a task so they can reboot or whatever the ws-export thing [20:40:40] Okay. [20:41:57] I think something is going on now ..."Firing for 23s" ... unless this is something automatic, someone is trying to bring it up. [20:43:18] * RhinosF1 looks at TheresNoTime and wonders if they can put their CommTech hat on [20:57:33] have y'all considered filing a task or something about ws-export being down, instead of pinging random folks on a mostly unrelated channel? [20:58:15] Filed as T348068 while we speak [20:58:15] T348068: ws-export is down (2023-10-03) - https://phabricator.wikimedia.org/T348068 [21:12:37] RhinosF1: I don't have access to https://openstack-browser.toolforge.org/project/wikisource (: [21:13:24] RhinosF1: I see you also fell into the trap that is the novaobserver password :D (see also T326009 ) [21:13:25] T326009: Exposed novaobserver password on WMCS GitLab repo - https://phabricator.wikimedia.org/T326009 [21:14:04] stw: yes [21:28:01] !log tools.integraality Deploy 3ba5e84 (T312729) [21:28:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [21:59:24] !log wikisource Rebooted `wsexport-prod01`, volume did not mount, ran `sudo mount -a` T348068 [21:59:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisource/SAL [21:59:29] T348068: ws-export is down (2023-10-03) - https://phabricator.wikimedia.org/T348068