[09:36:51] !log admin move maintain-dbusers to cloudcontrol1005 [09:36:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:07:47] !log admin merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360 [12:07:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:36:18] !log admin merging cloudgw puppet role refactor https://gerrit.wikimedia.org/r/c/operations/puppet/+/961363 (T347469) [17:04:48] Hey @rook. It's been a while... but, has PAWS been recently upgraded? Last weekend everything that was (apparently) saved for two days, was lost... And just now out of the blue, while my script was running the interface went into a loop asking me if I'm ok with my working directory not being there any more.     I don't wanna start another phab [17:04:48] report, just letting y'all know something strange IS happening again. Thnx. [17:05:19] heh, was upgraded about 30 seconds ago :p [17:07:29] Though there have been some problems with the underlying openstack cloud which made a mess of some other upgrades: [17:07:29] https://phabricator.wikimedia.org/T347150 [17:07:29] https://phabricator.wikimedia.org/T347111 [17:07:29] Mostly sum up what happened. I just tried a deploy a few minutes ago to see if there was any remaining trouble with the volume detach in openstack. It seems like it is working...Though I don't actually know if hub switched nodes. I need to check on that with another deploy. [17:07:42] * dcaro off [17:09:17] In short it has been a rough week for PAWS as some of what it relies on was not working as hoped. That being said I'm not immediately convinced that what you're observing is the same. It wouldn't surprise me if an hour or a little more data was lost, though mostly I would expect it would have gone unresponsive (this was on Thursday when I found the volume problem during an upgrade that brought down the cluster.) So I would say keep an [17:09:17] eye on what you're seeing and open a ticket if it persists [17:12:22] OH haha :O I'll restart everyting, just in case.    And I'll keep an eye on it.     About two days were lost, when I re-opened my notebooks I could only see some old output, and little bug fixes that I did were not there any more.   But then... I left it open for two days and thought it was my fault.   Thanks again. [17:27:26] @rook: every so often a modal window shows up telling me "Directory not found". Then Save error, invalid response: 405 Method Not Allowed... as we're speaking [17:29:34] Is that when you open a session when the last session was in something other than the home directory? [17:31:03] I restarted everything, went into a subdir, clicked on my notebook. It was running for a few minutes, then this started happening again. [17:42:55] !log tools.stewardbots Restarted SULWatcher bots. [17:42:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [17:53:00] Ponor: I feel like I've seen similar, though I have not been able to recreate it. Initially it seemed that I was just in a directory when I started, which seemed to go away for my session when I started in the root directory. I did not checked on if I was using an old direct link to that directory or the like, so it is possible that is what I was experiencing. [17:58:32] Rook I had another clean start. Maybe it was all because of the upgrade.  Notebook has been running fine for the last 15-20 minutes. Let's see what happens. [17:59:53] I think I'm done tinkering with it for today, so hopefully it stays steady [19:31:20] Hey all, on Toolforge I am using "toolforge jobs load file.yaml" to define cron jobs from a yaml file. Is there a way to define environment variables for the container, as in "kubectl --env"? Specifically, I want like to make shared pywikibot available inside the container by adding it to the PYTHONPATH. Thanks in advance! [20:19:15] https://toolsadmin.wikimedia.org/register/ is not up to date with how Toolforge or the Wikimedia Developer account system is now a SSO [20:19:31] I mean with https://wikitech.wikimedia.org/wiki/IDM [20:19:54] .. [21:23:48] Bsadowski1: still no luck with toolsadmin? [22:36:19] It looks like applying for toolforge access via toolsadmin is returning Error 500 today for all/some users