[06:36:04] <AmandaNP>	 Can someone kick whois-referral on toolforge again and ping me when it's back up? Trying to do a lot of lookups at once and it's down.
[07:04:24] <wm-bb>	 <Alexander> Hi guys, I hope someone can help me. I'm manager of an veeery old tool, originating on toolserver -> anno. Now it stopped working, I assume webservices are not active. But I'm unable to login...
[07:05:05] <arturo>	 AmandaNP: done ✅
[07:05:17] <wm-bb>	 <Alexander> I generated a new Key with puttygen, but the server don't want to accept my login, neither putty nor winscp
[07:06:26] <wm-bb>	 <Alexander> my user is AleXXw, the tool i need to access is https://toolsadmin.wikimedia.org/tools/id/anno
[07:18:47] <wm-bb>	 <tehreedy> Where did you upload it to?
[07:19:31] <wm-bb>	 <Alexander> I added the public key in my settings
[07:19:42] <wm-bb>	 <tehreedy> On toolsadmin?
[07:19:47] <wm-bb>	 <Alexander> yes
[07:21:33] <wm-bb>	 <tehreedy> I don’t think you should have to, but you might need to update it on wikitech too
[07:22:18] <wm-bb>	 <Alexander> tried that already, there came the error "key already there" or sth -> seems to be OK
[07:26:46] <wm-bb>	 <Alexander> the error message is always "No supprted authentification methods avialable"
[07:27:15] <wm-bb>	 <tehreedy> Have you changed your putty settings and/or loaded the new key into pageant?
[07:29:20] <wm-bb>	 <Alexander> I got a new pc recently, so I started all over again. I can login to several other (non-wmf)-servers with several authentification methods, including ssh. So PC, Firewall etc. should not be a problem
[07:29:56] <wm-bb>	 <tehreedy> ssh isn’t an auth method ;)
[07:32:02] <wm-bb>	 <Alexander> True :D
[07:38:39] <wm-bb>	 <Alexander> I strictly followed that: https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#How_to_set_up_WinSCP_for_direct_access_to_your_Toolforge_account
[07:39:30] <wm-bb>	 <Alexander> and also tried PuTTY, same error
[07:41:55] <taavi>	 what's your username? I can poke in the logs
[07:42:05] <wm-bb>	 <Alexander> AleXXw
[07:45:22] <taavi>	 you're then using the wrong username, you need to use the 'shell username' which for that account is `alexxw`
[07:46:32] <wm-bb>	 <Alexander> oh no, thx mate
[07:46:43] <wm-bb>	 <Alexander> that did the trick
[08:08:29] <wm-bot>	 !log tools.poty-stuff <legoktm> Updated from 5f614ea to a917697
[08:08:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL
[08:35:40] <wm-bot>	 !log tools.poty-stuff <legoktm> Updated from a917697 to 36c9249
[08:35:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL
[08:38:41] <wm-bot>	 !log tools.poty-stuff <legoktm> Updated from 36c9249 to 93201d6
[08:38:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL
[08:39:51] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 855271cbd3 (initial setup pre build packs)
[08:39:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[08:46:23] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 7f3f9839ee (/has-secret-key)
[08:46:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[08:46:31] <bd808>	 !log tools Building new perl532-sssd/{base,web} images (T323522, T320904)
[08:46:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[08:46:34] <stashbot>	 T320904: Add MIME::Lite to the Perl image - https://phabricator.wikimedia.org/T320904
[08:46:35] <stashbot>	 T323522: Toolforge Kubernetes perl profile: module PHP::Serialization needed - https://phabricator.wikimedia.org/T323522
[08:47:16] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> added secret key to config.yaml
[08:47:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[08:49:00] <chicocvenancio>	 Hey, anyone can help with access to a gitlab toolforge-repos,  alberobot ? albertoleoncio should have access
[08:52:17] <bd808>	 chicocvenancio: I see the access requests. Give me a minute to hit some buttons
[08:52:49] <bd808>	 chicocvenancio: {{done}} in theory
[08:52:59] <chicocvenancio>	 thanks!
[09:00:30] <wm-bb>	 <lucaswerkmeister> dammit the CNCF Buildpacks PPA is only available for LTS Ubuntu releases :(
[09:01:23] <taavi>	 the `pack` tool is also on the toolforge repo, although that's not intended for use outside our servers so use at your own risk
[09:03:16] <wm-bb>	 <lucaswerkmeister> I’m trying the Docker version now
[09:04:05] <dcaro>	 note that the `pack` tool has features not available on toolforge (or tekton), like project.toml parsing or explicit buildpack selection
[09:13:37] <wm-bb>	 <lucaswerkmeister> the docker version worked – should I add it to the wiki page? would that be useful?
[09:17:15] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 5468034c9e (still running as normal webservice for now)
[09:17:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[09:19:41] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 802a6f0eb7 (pulled without webservice restart; second build now running)
[09:19:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[09:20:16] <chicocvenancio>	 bd808: I think we need more access there
[09:20:23] <chicocvenancio>	 seems developers can't create branches?
[09:22:28] <bd808>	 chicocvenancio: I jsut made Albertoleoncio an owner there. 
[09:22:40] <bd808>	 They can choose to change your rights as they see fit
[09:22:51] <chicocvenancio>	 cool, thanks
[09:56:58] <isaacj>	 arturo: T337019
[09:56:59] <stashbot>	 T337019: Build dataset of Quarry queries - https://phabricator.wikimedia.org/T337019
[09:57:01] <arturo>	 cool
[10:16:04] <dcaro>	 isaacj: sweet
[10:17:31] <isaacj>	 dcaro: haha yeah, been doing some thinking about what sorts of datasets we could put together to encourage more outside researchers/developers to try to build tools that would be useful for Wikimedia. still thinking about the best way to do this but i was sitting next to arturo and realized I couldn't miss the opportunity to ask as this one is a really interesting one :)
[12:16:27] <Ponor>	 Hi all. Anyone here to debug PAWS "notebook runs but cannot be saved" bug as it is happening? (https://phabricator.wikimedia.org/T310622)
[12:16:28] <Ponor>	 Rook? chicocvenancio?
[12:19:22] <Rook>	 hmm...
[12:21:00] <Ponor>	 I'll keep it running, so no rush. Just though it might help to try debugging this way.
[12:23:25] <Rook>	 What's the username for the server you have running?
[12:23:37] <Ponor>	 PonoRoboT
[12:23:44] <Rook>	 👍
[12:24:26] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed aa1162b097 (buildservice webservice type – works but does not find config.yaml yet)
[12:24:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[12:26:43] <Ponor>	 Rook There are two notebooks now that cannot be saved
[12:37:23] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 7dcf4d195f (chdir to tool directory; doesn’t seem to be working yet)
[12:37:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[12:38:16] <Rook>	 Ponor: I see a lot of these errors, was that something you were working on? Looks like it was stopped via manual intervention https://www.irccloud.com/pastebin/39P5Fa2G/
[12:40:54] <Ponor>	 Yeah, something happened on my end and the two notebooks refreshed with 12+ hours old code.   I did c&p the all the code from one of the notebooks so nothing's lost. Had I not noticed... hours of work would be lost
[12:41:31] <Ponor>	 I don't know what those errors mean, tbh.
[12:42:40] <Ponor>	 Even though the notebook refreshed with the old code, the old kernel is still running
[12:42:47] <Rook>	 What were the notebooks that do not save?
[12:43:50] <Ponor>	 Py - Popisi stanovništva - enwiki.ipynb
[12:44:15] <Rook>	 Thank you
[12:44:59] <Ponor>	 wikidata editing.ipynb didn't save until the refresh, then it saved it with only old code
[12:46:13] <Rook>	 Yeah looks like the last save was about 12 hours ago `-rw-r--r--. 1 tools.paws tools.paws 813464 May 19 00:52 'Py - Popisi stanovništva - enwiki.ipynb'`
[12:47:50] <Ponor>	 all new notebooks can be saved as expected, only some enter this limbo
[12:48:38] <Rook>	 The logs don't say much around then, but there does appear to be a gap in them at that time... https://www.irccloud.com/pastebin/e8sjd1Ur/
[12:57:40] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed c82d5de3f1 (chdir to www/python/src subdir; still not working and idk why)
[12:57:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[12:58:33] <Rook>	 If you select "File" is "Save Notebook" Allowed or is it gray for that notebook?
[12:58:46] <Rook>	 Sorry I have to step away, I'm being called to breakfast
[13:01:16] <Ponor>	 Sorry for interrupting your breakfast :(   To answer your q: nothing's grayed out; I also tried Save As..., it asked me for a new name, but nothing was saved
[13:02:21] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 3899387238 (more debug code)
[13:02:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:03:50] <Ponor>	 I lets me rename the notebook, but it still cannot be saved
[13:05:26] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 075e8f7d04 (fix syntax + typo)
[13:05:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:09:55] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed f62a03a545 (log Flask base path)
[13:09:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:12:45] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 714a6e59d3 (import flask after chdir)
[13:12:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:17:20] <wm-bb>	 <lucaswerkmeister> taavi: apparently Flask’s config root path defaults to the directory the source code is being loaded from, not the current working directory as I thought
[13:26:55] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 21f3fd2f9d (pass root_path into Flask)
[13:26:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:27:22] <wm-bb>	 <lucaswerkmeister> https://gitlab.wikimedia.org/toolforge-repos/lucaswerkmeister-wmde-test/-/commit/21f3fd2f9df21eae82ea1ce03daeb21911df2c2a works
[13:31:00] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 4de30c0be7 (absolute config path instead of root path)
[13:31:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:31:12] <wm-bb>	 <lucaswerkmeister> https://gitlab.wikimedia.org/toolforge-repos/lucaswerkmeister-wmde-test/-/commit/4de30c0be7f7d4d46eb7fd93542e4c9f4f7c12fa also works and is probably better
[13:35:46] <wm-bb>	 <lucaswerkmeister> I updated the wikitech page
[13:37:39] <Rook>	 Ponor: it's strange. As though the tab itself has failed but not visibly
[13:38:25] <arturo>	 !log tools uncordon tools-k8s-worker-47/48/64/75
[13:38:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:39:14] <taavi>	 !log tools.openstack-browser restart, was timing out, logs had timeouts from openstack
[13:39:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL
[13:43:59] <wm-bot>	 !log tools.lucaswerkmeister-wmde-test <lucaswerkmeister-wmde> deployed 4d46d0cd73 (Python 3.11.3 :o )
[13:44:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lucaswerkmeister-wmde-test/SAL
[13:45:55] <Rook>	 Ponor: do you mind if I stop all the kernels running in your server?
[13:46:06] <wm-bb>	 <lucaswerkmeister> (btw, `kubectl rollout restart deployment TOOL_NAME` still works to restart a buildpack-based tool ^^)
[13:46:33] <Ponor>	 Rook a few more observations:
[13:46:34] <Ponor>	 1) I tried "Save as..." and gave the new file a -saveas suffix.  Nothing happened in my folder.   But when I right-click in the tab and choose "rename" it offers me the -saveas name as the current name. When I hover the tab, the baloon stil shows the old name... even though
[13:46:34] <Ponor>	 2) I renamed the file in the folder, and the terminal clearly shows the suffix that I added, -renamed
[13:46:46] <Ponor>	 Rook, I will stop all the kernels
[13:49:24] <Rook>	 Thanks!
[13:50:49] <Rook>	 Load is still ~2 without the kernels running. I wonder if we can find what is causing it
[13:52:35] <Rook>	 ooo and error that looks topical `FileNotFoundError: [Errno 2] No such file or directory: '/home/paws/HR/Py - Popisi stanovništva - enwiki - saveas.ipynb'`
[13:56:49] <Ponor>	 how about I kill my browser without logging out?
[13:59:49] <Rook>	 Before you do that. The file still doesn't save?
[14:00:57] <Ponor>	 It did get saved 7 minutes ago, as that -saveas
[14:01:28] <Ponor>	 saved with code loss, of course
[14:01:46] <Rook>	 Ah ok, so the new code in your browser does not save. Just the file itself?
[14:03:08] <Rook>	 And from your end does it look like `HR/Py - Popisi stanovništva - enwiki.ipynb` saved? It still looks like the last save was more than 12 hours ago in my view
[14:03:20] <Ponor>	 well the thing is the notebook did refresh its content at some poing this am (when I went to check Hub Control Panel and you noticed the new oauth)
[14:04:19] <Rook>	 What does refresh mean in this context?
[14:04:25] <Ponor>	 Oh yes, you're right. Didn't notice the old file was still there. So yeah, 13 hours ago for that one
[14:04:44] <Rook>	 Oh ok, I'm seeing errors about that one now
[14:05:01] <Rook>	 `[W 2023-05-19 14:01:08.332 SingleUserLabApp web:1869] 409 PATCH /user/PonoRoboT/api/contents/HR/Py%20-%20Popisi%20stanovni%C5%A1tva%20-%20enwiki%20-%20saveas.ipynb?1684504868286 (10.100.3.0): File already exists: HR/Py - Popisi stanovništva - enwiki.ipynb`
[14:05:03] <Rook>	 kinds of things
[14:05:23] <Rook>	 It looks like something is trying to save/create that file and is failing
[14:06:24] <Ponor>	 Refresh in this context: Last night I was adding some new code and running it. And that code was visible in the browser this morning and could be run, but saving was not possible. Then the notebook "refreshed" itself with the old code, not the additions
[14:07:28] <Ponor>	 the additions were saved in the " - new" file last night, because I c&p all the cells
[14:08:16] <Rook>	 Ah, thank you for the clarification.
[14:10:18] <Rook>	 So your browser has a notebook open, and the notebook runs and works as expected, except after a few days it fails to save, either automatically or manually. Save as does not work either, as the result is a copy of the last time the current notebook saved (thus nothing new that is visible and functioning in the browser)?
[14:10:58] <wm-bb>	 <lucaswerkmeister> I added it now (re @lucaswerkmeister: the docker version worked – should I add it to the wiki page? would that be useful?)
[14:12:05] <Ponor>	 That's right, Rook
[14:13:00] <Rook>	 That makes it seem like the "Save as" function is a save followed by a copy
[14:13:01] <Ponor>	 An apparently, that "save as" took some 1+ hours
[14:14:35] <Rook>	 And that the notebook loses track of it has a connection to the disk, but happily continues to live and run in memory
[14:14:40] <Ponor>	 I believe I tried "save as" before, and that alwas ended up being copy (of what's already saved) +rename
[14:15:30] <Rook>	 Go ahead and try restarting the browser and see if that gets it back to a working state
[14:15:46] <Ponor>	 OK. brb
[14:19:09] <Ponor>	 Crashed the broswer, went back to its JupyterLab tab, got this message: "File Load Error for Py - Popisi stanovništva - enwiki - saveas.ipynb"
[14:23:01] <Rook>	 Interesting
[14:23:36] <Rook>	 `-rw-r--r--. 1 tools.paws tools.paws 1616550 May 19 14:11 'HR/Py - Popisi stanovništva - enwiki.ipynb'`
[14:23:44] <Rook>	 Looks like it did manage to update, and double in size
[14:26:02] <Ponor>	 That happened just before I crashed the browser because I clicked on the file and it opened its duplicate (from the disk). Not sure if that'd normally happen or was it 'thinking' that the other JupyterLab (internal) tab was the -saveas file
[14:26:40] <Ponor>	 JupyterLab is now slowing down the broswer; click on any file does nothing
[14:28:35] <Rook>	 Let me see something
[14:34:16] <Rook>	 I'm not sure if https://github.com/jupyterlab/jupyterlab/issues/5327 matches well enough. it seems for at least some users it is triggered by running certain code. Though mostly it seems to parallel with firefox
[14:37:52] <Ponor>	 Looks similar, though I can save other files, old and new
[14:38:49] <Rook>	 I feel we should open an upstream ticket. Do you think we should add to that one or simply reference it in a new ticket?
[14:38:55] <Ponor>	 Whatever got asved in "'HR/Py - Popisi stanovništva - enwiki.ipynb" ~30 minutes ago is crashing JupyterLab
[14:40:07] <Rook>	 Interesting, on the terminal it looks like a notebook, though it may be malformed
[14:41:28] <Ponor>	 Looks similar enough to the problems they're reporting in old ticket
[14:46:03] <Rook>	 What version of firefox are you running?
[14:46:09] <Ponor>	 I used labs at PAWS months before you officially switched, but I don't remember this happening back then.
[14:46:10] <Ponor>	 Don't know how to describe the technologies behind all this, but I remember it used to ask me whether I wanted to respawn? the session (it does not ask anything now, everything is just there)
[14:47:01] <Ponor>	 112.0.2 (64-bit) for Fedora
[14:47:47] <Rook>	 It could be related to the change in default. I don't know much about how the backend of that really works, jupyterhub itself changed the default so it was an upgrade of jupterhub that moved to labs from notebooks. It's possible that an integration change at that point caused the problem, though I am only guessing
[14:50:58] <Rook>	 It's tempting to suggest that https://phabricator.wikimedia.org/T324002 is related. Notebooks don't load when we upgrade. Looks like jupyterlab 4.0.0 came out so perhaps it will work. I'll post to the upstream and go from there
[14:52:29] <Ponor>	 Thank you!!!  I think we're done with this experiment. The "original" file and zero-bytes -saveas file cannot be opened
[14:53:32] <Rook>	 I agree, I think we've learned what we can learn at this point. But let's keep at it :)
[14:53:58] <Ponor>	 See you around. Thanks again!
[14:56:20] <Rook>	 No problem, and likewise!
[16:58:23] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed b80c8ff9db (fix “logged in” indicator in several languages)
[16:58:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[17:04:36] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed b59c2f0aad (l10n updates: es, hi, zh-hant)
[17:04:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[18:31:20] <wm-bot>	 !log tools.ldap-beta <legoktm> Move to buildpack webservice image!
[18:31:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ldap-beta/SAL
[18:35:30] <wm-bb>	 <lucaswerkmeister> oooh, where’s the repository? (I tried but failed to find it ^^)
[18:41:05] <wm-bb>	 <lucaswerkmeister> ah, I see, it’s https://gitlab.wikimedia.org/toolforge-repos/ldap
[20:29:45] <wm-bot>	 !log tools.poty-stuff <legoktm> Updated from 93201d6 to c3516a4
[20:29:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL
[20:33:10] <wm-bot>	 !log tools.poty-stuff <legoktm> Updated from c3516a4 to 5b4a776
[20:33:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poty-stuff/SAL