[09:06:55] Hello Cloud team :D Our Data Scientist at WMDE is having some trouble spawning an instance since last Thursday, the seems to have been stalled on "Scheduling". See: https://phabricator.wikimedia.org/T301380#7844097 [09:07:21] **the process [09:14:11] itamarWMDE: hello! could you post screenshots for all the vm creation setting screens on the task? I wonder if something's going wrong there since no other projects seem to be having the issue [09:16:31] didn’t mutante have a similar issue on Friday? that time it was due to reusing an instance name according to the IRC log [09:16:47] I can ask him to do so, since it seems like that stalled instance does not appear on the list of instances on my side [09:28:55] Hey, we have some 500's at sal.toolforge.org. Could someone restart the webservice? Thanks. [09:50:00] itamarWMDE: oh, looks like nova doesn't like that there is an instance with the exact same name in a different project. need to run now, but I'll file some tasks later to make it less confusing [09:53:06] My ItemSubjector jobs fail again with no errors. Hard to debug then :/ [12:01:01] taavi: Thank you! Yeah this seems likely, will follow up on it with him. [12:10:54] !log tools.sal restarting webservice per IRC reports of it showing some 500's [12:10:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [14:47:44] !log toolsbeta deploying custom version of the regitsry admission hook [14:47:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:59:16] !log admin created cloudinfra.wmcloud.org zone [15:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:17:49] !log toolhub Updated demo server to 723c3b [17:17:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL [17:41:34] taavi: could you please restart the SULWatcher bots? They seem to be down. Thanks. [17:43:38] hauskatze: sure! [17:44:16] ty [17:48:31] !log tools.diff-privacy-beam deleted kubeconfig file, somehow created as empty causing maintain-kubeusers to crash [17:48:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.diff-privacy-beam/SAL [18:25:46] !log gitlab-runners pausing runner-1011 in gitlab UI from accepting new jobs, then deleting instance in Horizon UI to replace it with another bullseye instance T297659 [18:25:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-runners/SAL [18:25:50] T297659: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 [19:06:38] !log gitlab-runners deleting instance runner-1011, creating instance runner-1022 to replace it [19:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-runners/SAL [20:20:22] !log gitlab-runners deleting instance runner-1012, creating instance runner-1023 to replace it [20:20:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-runners/SAL [20:29:00] !log gitlab-runners - create new instance runner-1023, wait for intial puppet run, then rm -rf /var/lib/puppet/ssl ; run puppet; sign new request on gitlab-runners-puppetmaster-01.gitlab-runners (normal procedure for fresh instance with local puppetmaster) [20:29:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-runners/SAL