[09:18:32] !log tools reboot stuck tools-k8s-worker-84 [09:18:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:17:50] godog: ^ for the prometheus question above [10:20:43] ah yes that's my bad, I've upgraded Prometheus to latest upstream though the Debian packaging is minimal (and ours) as opposed to Debian https://phabricator.wikimedia.org/T354399 [10:20:48] taavi stw ^ [10:21:12] I'll restore Debian's default config path [16:35:45] stw: 2.48.1-0+wmf2 has the previous defaults (i.e. a smooth upgrade) cc taavi [16:35:58] 00 [16:35:59] ~ [16:36:11] that was me talking to ssh [16:36:22] <3 - thanks muchly. I'll give it a go later this evening [16:36:58] sure np, let me know how it goes [16:44:09] * experemental calls ExitProcess(0) [17:22:43] Hello, everyone, Happy New Year! I have a question: Does anyone know what happend to GLAMWiki Dashboard? It says https://glamwikidashboard.toolforge.org doesn't exist, and it is not showing up in the admin list of tools either. [17:27:40] I think it might be supposed to be https://glamwikidashboard.wmcloud.org/ these days [17:27:44] Not that it's loading with any sort of sped [17:27:47] *speed [17:28:48] https://openstack-browser.toolforge.org/project/glamwikidashboard [17:29:20] and eventually... [17:29:21] >This web service cannot be reached. Please contact a maintainer of this project. [17:34:51] It was changed at random without notice? 😢 (re @wmtelegram_bot: I think it might be supposed to be https://glamwikidashboard.wmcloud.org/ these days) [17:35:59] It would look like it may have been that way for a year or more at this point - https://phabricator.wikimedia.org/T324411 [17:36:34] Well, May 2023 - https://phabricator.wikimedia.org/T321702#8841945 [17:37:59] Did glamwikidashboard.toolforge.org actually ever work/exist? [17:41:11] First, it was at wmcloud, when it was under Wikimedia Israel server. Then WMF took over the support and transfered to Toolforge (it was working until mid december), then, from nowhere, the toolforge instance was deleted. [17:42:02] That is very, very damaging to our affiliate reports and our GLAMWiki partners. [17:42:47] I don't see any traces of that ever existing in Toolforge [17:42:51] googling doesn't seem to find (m)any results for glamwikidashboard.toolforge.org either [17:46:04] Humm, then I'm gaslighting myself. Removing the toolforge bit, the other part (support responsability) occured for sure [17:46:18] There is https://glamtools.toolforge.org but that's completely different [17:46:43] just because it moved to wmcloud doesn't mean the WMF took over support either [17:46:57] the request tasks make it look like it's probably still realted to WMIL [17:47:08] https://phabricator.wikimedia.org/T324411 [17:47:28] https://phabricator.wikimedia.org/T321702 suggests WMF support, but again, that's not the same as it taking it over [17:48:46] Taking over the support of the servers? That appears in the commons pages for glam tools [17:48:59] link? [17:49:34] https://w.wiki/8r8h [17:50:19] >We supported Wikimedia Israel to migrate the latest version of their dashboard to the Wikimedia cloud. This removed prohibitive data costs and allowed them to restore their service for existing partners. We were able to work with Wikimedia Israel on this because they had a developer who was knowledgeable about, and still actively working on, their service. [17:50:28] "we supported" is not the same as taking over [17:52:30] https://commons.wikimedia.org/wiki/Special:LinkSearch?target=https%3A%2F%2Fglamwikidashboard.toolforge.org also returns no results, fwiw [17:54:11] Yeah, you said it is in wmcloud, I checked that our links goes to there [17:54:30] https://pt.wikipedia.org/wiki/Especial:Pesquisar_links?target=https%3A%2F%2Fglamwikidashboard.wmcloud.org [17:54:54] That's good at least [17:55:57] Does that change the outcome of not working? Can you direct me to whom I need to alert of this error? [17:56:39] I would suggest the people listed as "members" on https://openstack-browser.toolforge.org/project/glamwikidashboard should be CC'd on the task you filed [17:57:02] they should potentially have a phab project created too if they're going to track tasks there, but that's a different issue [18:45:32] Mid December? Over 200 tools were shutdown around that time. Is this one affected to maybe? (re @ederporto: First, it was at wmcloud, when it was under Wikimedia Israel server. Then WMF took over the support and transfered to Toolforge ...) [18:50:33] Nevermind, that's a VPS project. If it depends on something else, that might be affected, see https://phabricator.wikimedia.org/project/board/6135/ for the list (re @MaartenDammers: Mid December? Over 200 tools were shutdown around that time. Is this one affected to maybe?) [18:53:28] godog: upgraded seamlessly, thanks a lot! [22:10:41] Hello, im a freelancer working on the glams dashboard, it appears that our DB has gone down spontaneously and a restart on horizon responds with an error. Would love for some help, at least with recovering the data as our last backup is a few months old [22:16:23] Probably worth posting (or pastebinning) the error ;) [22:20:23] your database seems to have ran out of disk space [22:20:28] The horizon error is just: "Unable to restart instance", I wasn't able to source more info about the error. When trying to connect to the DB the error is "EHOSTUNREACHABLE" which I assume means "unreachable host". [22:23:15] Is there any way to salvage the DB if it ran out of space? Not extremely critical as we can always process wiki's data again but would save a bunch of time and effort if possible. [22:25:55] All depends if/what corruption has occured [22:26:11] Any data that wasn't written to disk is obviously lost [22:27:41] yeah. either way the US holiday today means that it'll have to wait until the european morning tomorrow [22:28:52] I'll wait, thank you for your help [22:54:20] !log anticomposite@tools-sgebastion-10 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # Disconnected [22:54:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [23:00:15] !log anticomposite@tools-sgebastion-10 tools.stewardbots SULWatcher/manage.sh restart # SULWatcher2 unresponsive [23:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL