[13:19:33] !log wmflabsdotorg drop legacy 128-25.155.80.208.in-addr.arpa. dns zone [13:19:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmflabsdotorg/SAL [14:13:05] !log superpes@tools-sgebastion-10 tools.stewardbots Restarted StewardBot not working on IRC [14:13:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [15:47:41] bunch of my tools just became inaccessible, something going on? [15:53:18] oop and they're back [15:56:05] there was a brief NFS problem, related to the underlying CEPH storage system, we are investigating [15:59:10] link-dispenser appears to have forgotten how to read from sockets :( [15:59:35] !log tools rebooting tools-k8s-worker-75.tools.eqiad1.wikimedia.cloud,tools-sgeweblight-10-[30,32].tools.eqiad1.wikimedia.cloud due to nfs failure [16:00:21] !log bd808@tools-sgebastion-11 tools.grid-deprecation Restart to try and get a different k8s host node with working nfs [16:03:17] !log lucaswerkmeister@tools-sgebastion-10 tools.stashbot restarted, had left #wikimedia-operations and not returned [16:03:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [17:21:53] andrewbogott: could you help me out with an edit in project Hiera of "redirects" project or alternatively temp add me to the project [17:22:28] yes, I can add you, one sec [17:22:44] thank you [17:24:49] mutante: what are you changing? [17:25:13] andrewbogott: adding a redirect from phorge.wmflabs to phabricator.wmcloud [17:25:23] cool [17:25:28] if you reload you should have access now [17:25:28] eh, phab.wmflabs i mean [17:25:33] thanks Andrew [17:33:09] andrewbogott: following the docs at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects but one thing stops me at the end. While there are existing "wmflabs.org" proxies in the redirects project it looks like I can't add any new proxies in the legacy domain. only wmcloud in the domain list. The reason to add it here would be to stop actually using the legacy domain though [17:34:22] seems like a catch 22 to get rid of "labs" [17:36:57] If you create a wmcloud proxy there's an automatic failover from the equivalent wmflabs domain [17:37:04] or should be [17:37:09] sorry, I should've mentioned that before :) [17:37:55] oh, so use 2 redirect steps then? let me try [17:39:08] so far I am getting to https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/Host_not_configured [17:39:41] runs puppet again on redirects [17:41:36] you should configure 'phab.wmcloud.org' in redirects hiera and proxies, then the redirect chain will be phab.wmflabs.org -> phab.wmcloud.org -> phabricator.wmcloud.org [17:43:14] thanks, just did that and it works. [17:43:21] except I should actually use .. phorge .. haha [17:45:57] !log lucaswerkmeister@tools-sgebastion-10 tools.lexeme-forms restarted webservice, log was full of various errors [17:46:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [17:46:04] * andrewbogott doing too many things at once now, sorry [17:48:04] andrewbogott: done from your end. dont worry. tx [17:49:21] !log lucaswerkmeister@tools-sgebastion-10 tools.lexeme-forms stopped webservice, restart wasn’t working so let’s try harder [17:49:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [17:52:19] hm, tool’s still not coming back up :S [17:52:36] I don’t see any new errors in kubectl logs or uwsgi.log but requests just seem to hang forever [17:53:01] toolforge's has any running issues, or it just happens to me. Steward election-related tools are not working; they are just loading in the browser. https://iw.toolforge.org/meta/accounteligibility/?user=MdsShakil&wiki=&event=70 https://stewardbots-legacy.toolforge.org/Elections/elections.php [17:54:10] taavi: debatable whether I should use "phorge.wmcloud" because that's the upstream name or "phabricator.wmcloud" because that matches production and which is less confusing. For now I redirected all the variants (phab / phorge | labs / cloud) to phabricator.wmcloud. thanks for pointing out the redirect project [17:59:01] !log lucaswerkmeister@tools-sgebastion-10 tools.lexeme-forms started webservice again (and patched the startup probe into it); took a while to come up but now it seems to be working [17:59:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [18:40:03] https://mix-n-match.toolforge.org/ is down [18:42:11] !log bd808@tools-sgebastion-11 tools.mix-n-match Restarted webservice after report of down/hang via irc [18:42:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mix-n-match/SAL [18:42:56] Mailaender: I restarted the web ui for it. the backing bot seems to have some other issues, but I don't know how to fix that part unfortunately. [18:43:52] thanks [19:34:39] !log superpes@tools-sgebastion-10 tools.stewardbots Restarted StewardBot not feeding on IRC [19:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [21:51:02] !log devtools rebooting gitlab-prod-1002 T356906 [21:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL