[07:25:58] Wurgl: you can try running `toolforge jobs quota`, that might give you some idea [07:27:09] It vanished magically [07:39:55] https://pastebin.com/KPYKNLxC <-- this is how it looked when I asked, I have never seen some similar output [07:40:24] Interesting, which tool is this? [07:40:32] persondata [07:41:23] at about 7:50 GMT the problem vanished. I did not change anything [07:45:21] It might be jobs triggering too close to each other looking [07:48:38] yep, sometimes shell sessions (`webservice shell`) don't close properly, and they linger taking up resources [07:48:57] https://www.irccloud.com/pastebin/v5znqBze/ [07:49:09] it says there's 12 running pods, when no jobs are actually running [07:49:24] https://www.irccloud.com/pastebin/gl9EUC1d/ [07:49:35] (sorry, 2 jobs are running) [07:50:49] the rest are shell sessions, there's a task to manage those better T315735 [07:50:50] T315735: Shell pods continue running after ssh session exits - https://phabricator.wikimedia.org/T315735 [07:53:16] Hmm … how to stop them? [07:53:39] I'll add a note there, for now you can run `kubectl delete pods -l 'app.kubernetes.io/component=webservice-interactive'`, until we get a better way [07:54:02] (you can also kill them one by one, with `kubectl delete pod `) [07:54:12] in case you don't want to kill them all [07:54:43] I do not need any of those [07:55:01] all are gone. Fine. Thanks [07:55:23] 👍 [07:56:20] you can subscribe to that specific task or https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog and you'll be notified when we get a nicer way to handle them (or ask around from time to time xd) [08:03:29] @dcaro … take a look at the last lines in /data/project/persondata/.profile … thats my solution [08:05:18] Wurgl: hehehe, that also works xd, might stop working if the labels change at some point though but it's good until there's a proper fix (ex. T311917 might change the label to something different than `webservice-interactive` as it would not be part of webservice anymore) [08:05:20] T311917: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917 [10:12:24] !log tools.openstack-browser restart webservice, page wasn't loading and logs contained lots of gunicorn errors [10:12:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [11:07:11] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [11:07:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [12:10:59] Is wmopbot supposed to be DCing and reconnecting everyday? [12:54:37] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed aa319bc391 (l10n updates: fr, he, qqq) [12:54:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:10:49] I wonder: Lighttpd has configured index file names (index.php, index.html, index.htm) and those work correctly. However, if I go to the exact same folder using the tools-static server, the index file is not used, and a directory listing is displayed instead. Is this a bug, feature, configuration problem...? Could/should I configure something to [14:10:49] improve that? (Cf. https://mormegil.toolforge.org/propertylinker/volby/ and https://tools-static.wmflabs.org/mormegil/propertylinker/volby/ with the index available at the explicit https://tools-static.wmflabs.org/mormegil/propertylinker/volby/index.htm instead.) [14:38:51] Mormegil: things like `index.php` will never run on tools-static as it will not run any code, but feel free to open a task to request the static files, we can take a look [14:43:01] Sure, I did not expect _PHP_ to work (so, yeah, probably the index-file.names configuration should exclude index.php from the list on the static server, even though when you put PHP files on a static server, you're asking for trouble, anyway). I am quite surprised the directory listing is enabled. I'll open a task, thanks. [15:18:44] Mormegil: the historic use case of tools-static is to provide access to things like bot log files which I think makes it serving up directory listings rather than forbidden responses for directories without an index very logical. I don't think it would be a problem to update the configuration to support the very old-school windows .htm extension your tool seems to be using for its index file. [15:19:59] Oh, yeah, that makes sense: on a static server, you don’t have alternatives to directory listing. [15:21:23] And… you mean the only reason the index file does not work is that the static server supports only index.html while the normal server supports both .html and .htm? That's... surprising and funny, I'd say :-) Just a bit of bad luck on my side, then, choosing exactly the wrong extension. [15:22:02] The "normal" server you refer to is pretty ancient too :) [15:23:00] the lighttpd config in the php base images is mostly from ~2014 and copied from older things [15:26:46] The tools-static nginx config does not have any explicit `index ...` stanzas in it, so it is using the default that is equivalent to `index index.html;` [15:38:46] Oh, it's just the default? That I could have known, I just don't use lighttpd at all. [15:41:00] tools-static is powered by nginx. it doesn't use lighttpd at all. But yes it is just default config for index files. [15:41:33] Even more complicated than I thought :-) But that explains it. [15:42:56] So it is just served directly by the frontend reverse proxy (which is nginx, I believe)? [15:45:49] !log lucaswerkmeister@tools-bastion-13 tools.lucaswerkmeister-test webservice stop && unlink www/static # stop old tests; the webservice was a copy of wd-shex-infer (Python) and static was a symlink to ../cdnjs-index/out/ from cdnjs testing [15:45:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-test/SAL [15:46:06] !log lucaswerkmeister@tools-bastion-13 tools.lucaswerkmeister-test create new simple www/static/ [15:46:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-test/SAL [15:46:26] https://tools-static.wmflabs.org/lucaswerkmeister-test/indextest/ seems to show the index.html indeed, FWIW [15:58:37] Do modules use Extension:SyntaxHighlight by default? The highlight feature does not work for some (large) module pages [15:59:38] Mormegil: tools-static is served by a dedicated set of nginx servers that are distinct from the Toolforge or Cloud VPS shared proxy servers. We don't seem to have ever written much about it on Wikitech other than its use of $HOME/www/static as a docroot. [16:15:49] OK, thanks. [18:19:38] !log urbanecm@tools-bastion-13 tools.sal webservice restart [18:19:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [18:32:30] Hey all, I have added a new SSH key (RSA 4096 bit) for WMCS in idm.wikimedia.org, but I cannot activate it although it confirms to me that "SSH key has been activated". What am I missing? [18:39:39] MisterSynergy: What happens when you try to activate it? Is there some kind of error? It's not like I know about the system, I'm just looking at it myself and maybe I should link my accounts there [18:40:40] Green success message "SSH key has been activated.", but I'm back on the list of SSH keys and therein it is listed as not active [18:40:47] so the upload works? [18:40:52] hmm, I see [18:41:44] Yes, upload works. No complaints from the system at all [18:43:59] This sounds like maybe we should create a Phabricator ticket for it. Just because it's more effective than finding people in realtime that live across timezones. [18:44:18] and we could then link to discussion when it might happen to others [18:45:45] a screenshot showing how it's invalid and what is pasted in the form could be good [18:46:10] Do you experience the same behavior? Then it does look like a bug and should be reported. Just wanted to make sure in the beginning that I am not just doing it incorrectly somehow... :-) [18:47:23] so when you paste the key, you paste the key comment in that separate form field for "comment" and the other part is just the key without the comment? [18:48:18] comment = anything at the end that follows a space [18:51:06] MisterSynergy: I made some tests, but my key is already uploaded. so when I try that again it notices and tells me "key already exists" and if I try to mess up the format it it notices that too and says "invalid key" [18:51:35] so I would have to create a completely new 4096 bit key first [18:51:43] but can do [18:52:35] Just tried it again. I can add the key with or without comment into the "SSH public key". The tool automatically recognizes the comment, and accepts the key with and without comment as valid key. But I cannot activate it [18:54:08] MisterSynergy: I created a new 4096 bit RSA key, uploaded it, sucess.. and there isn't even an Activate step, it's just there.. and I could click "Suspend" to deactivate it [18:54:43] deactivated it again, no issue with that either [18:55:29] I have to add though I already have a working key in there, that I never had to upload here, it came from the previous interface [18:55:35] My keys are not active by default (and I also cannot authenticate when I use them) [18:56:36] I also have an older, activated, and working key in there that was automatically imported from a previous system. [18:57:01] for me they are active immediately after upload and I can suspend and activate again.. and that "Activate" step also works [18:58:16] Turns out I can suspend old keys, but not activate them again as I just found out :-/ [18:58:31] MisterSynergy: I'd say there are about 3 routes from here: create ticket in phabricator, send mail to mailing list, just idle for a while [18:58:49] maybe upload screenshots somewhere [18:59:12] Yeah, phabricator [18:59:23] Will share a ticket later here [19:00:22] thanks, perfect [19:14:14] https://phabricator.wikimedia.org/T366525 [19:19:14] looks good, subscribed [19:19:36] sounds like things I reported in T360966 too [19:19:36] T360966: The workflow for managing an ssh key via Bitu is cumbersome - https://phabricator.wikimedia.org/T360966 [19:21:29] MisterSynergy: you can also try using https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ if you are blocked by the issues you have found with Bitu. [19:32:16] @bd808 thanks, toolsadmin.wikimedia.org works indeed, and the newly added key does appear in BITU as activated. And I can actually use it to authenticate. [19:32:45] Maybe [[wikitech:Help:SSH]] should not advertise BITU at this point. Seems new and not yet fit for the task [20:05:58] MisterSynergy: folks are pushing pretty hard for Bitu to replace all other methods of managing a Developer account. filing bugs reports like the one you created is what we all need to do at this point I guess [20:43:36] Any thoughts on this? (re @Yetkin: Do modules use Extension:SyntaxHighlight by default? The highlight feature does not work for some (large) module pages) [21:09:17] @Yetkin: That question does not seem to have anything to do with Wikimedia Cloud Services. You might try asking in the #wikimedia-tech or #mediawiki IRC channels. [21:58:21] Does anybody have any idea for solving this problem with PHP? https://phabricator.wikimedia.org/T366543