[10:03:56] !log jeanfred@tools-sgebastion-10 tools.integraality Recreated virtual environment using 'toolforge webservice python3.9 shell' / 'webservice-python-bootstrap --fresh' [10:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:52:14] !log tools reboot tools-legacy-redirector-2 (http probes were failing) [15:52:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:09:13] what’s the current status of third-party resource use on toolforge / cloud VPS? [18:09:27] all I can find is https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#5._Limited_use_of_third-party_resources_allowed which points to a page (https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Cross_Site_Policy) that has been a draft for over a year :/ [18:10:06] (specific motivation: I kinda want to reopen https://phabricator.wikimedia.org/T324103 with a comment to the effect of “complying with the terms of use is not optional”, except that the terms of use aren’t really clear AFAICT) [18:28:36] lucaswerkmeister: :sigh: there is no strict legal requirement at the moment. [18:29:09] alright :/ [18:29:10] thanks [18:29:51] I want to put an enforced CSP policy in place, but we also need a mechanism to alter that which has not yet been built [18:31:09] that being said, the TFSC can and should encourage people to do better [18:31:23] yeah, I was torn between bringing it up there and here [18:31:27] maybe I should still write to the mailing list [18:34:18] * bd808 nudges the task [18:35:30] thanks, you wrote that better than what I had above ^^ [19:03:26] hm, what happened to wm-bb [19:03:31] pods are both running… [19:04:01] ah, probably just an IRC hiccup [19:59:32] o_O why does https://gitlab.wikimedia.org/toolforge-repos/ranker not show up as a git repository at https://toolsadmin.wikimedia.org/tools/id/ranker ? [19:59:39] it was definitely created by Striker, you can see that at the bottom of https://gitlab.wikimedia.org/toolforge-repos/ranker/activity [20:15:45] !log lucaswerkmeister@tools-bastion-13 tools.bridgebot toolforge jobs restart bridgebot [20:42:58] lucaswerkmeister: that repo is tracked in Striker's backend. I wonder if something maybe got messed up in the display layer changes that taavi made recently? You can actually see it in the detail view -- https://toolsadmin.wikimedia.org/tools/id/ranker/repos/id/716 [20:43:45] I guess I’ll make a phab task then, because https://toolsadmin.wikimedia.org/tools/id/quickcategories seems to have the same issue [20:43:47] thanks for looking! [20:46:27] the bug is likely in https://gerrit.wikimedia.org/r/c/labs/striker/+/1108153/4/striker/templates/tools/tool.html. I think maybe line 130 is iterating a list that doesn't exist. [20:47:02] `{% for repo in repos %}` turned into `{% for repo in repositories %}` there [20:47:57] created T384068 [20:47:58] T384068: Striker (toolsadmin) doesn’t show GitLab repositories of (some?) tools - https://phabricator.wikimedia.org/T384068 [20:48:17] * lucaswerkmeister looks more closely at the gerrit change [20:48:41] yeah, there’s no associated Python(?) change that looks like it renamed the variable… [20:49:00] https://toolsadmin.wikimedia.org/tools/id/quickcategories/repos/id/574 is the quickcategories one. I think the tool view is just busted for all the tools. [20:49:36] left a comment on the gerrit change [20:51:28] * lucaswerkmeister was briefly confused by {% empty %} but apparently this is Django and not Flask and therefore probably also not Jinja2 ^^ [20:52:26] yeah, it is django syntax which is similar to jinja2 superficially [20:54:29] I’ve also heard they have a superficially similar thing to MarkupSafe [20:54:46] from my quick look into it it seemed like it would not be similar enough for great integration with toolforge-i18n though :S [20:55:04] but I suspect I’ll let that be Somebody Else’s Problem [21:22:13] looks like we've got problems again [21:22:49] ssh to login.toolforge.org hangs, though my tools are still up [21:22:55] whoop just got in [21:22:56] SSH is working for me [21:23:51] uptime shows a pretty high load avg but I don’t see anything in htop, might already be over [21:25:09] i did `less /data/project/geohack/error.log` immediately before [21:25:20] -rw-r--r-- 1 tools.geohack tools.geohack 91G Jan 17 21:22 error.log [21:25:21] yikes [21:26:15] ah [21:26:25] yeah that would do it [21:26:28] less -n would probably be better [21:27:43] that’s a lot of notices in that file [21:28:02] based on https://k8s-status.toolforge.org/namespaces/tool-geohack/ it's probably been broken for 3 days [21:35:18] the pod is in CrashLoopBackOff and has creationTimestamp: "2025-01-14T20:16:34Z" [21:35:42] whereas the deployment has creationTimestamp: "2023-11-08T10:31:05Z" [21:35:48] I’m not sure what to make of that [21:36:09] (replicaset is in between, creationTimestamp: "2024-08-26T15:02:29Z") [21:37:06] ok right now the pod is supposedly running again [21:37:16] though a request to it gave me 502 [21:37:23] yeah it’s back to CrashLoopBackOff [21:37:53] I’m guessing the liveness probe just fails repeatedly but I have no idea why or what to do about it [21:38:37] stop it, write a bug report in phab, and ping the maintainer [21:39:19] t.aavi had a crashloopbackoff hunting script at some point, but I bet he stopped running it [21:56:50] i had a 90% complete one and never bothered finishing it [21:58:04] given the log file has grown that much in a few days, it needs stopping sooner or later.. did you already do it or do I need to go get a laptop with the right keys? [21:58:26] s/sooner or later/sooner than later/ [21:59:09] I didn't touch it, but I can... let me peek [22:01:48] !log bd808@tools-bastion-12 tools.geohack `kubectl delete deployment.apps/geohack` -- pod in CrashLoopBackOff with 180 restarts [22:01:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geohack/SAL [22:04:46] !log bd808@tools-bastion-12 tools.geohack ~/error.log is 91G. Last 50 lines is all notice messages about "Only variables should be passed by reference" in various lines of ~/public_html/geohack.php [22:04:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geohack/SAL [22:05:42] lol. those notices in the error.log go back to the file's creation in mid-2022. [22:05:52] yup [22:08:30] https://toolsadmin.wikimedia.org/tools/id/geohack -- that maintainers list does not make me think that filing a phab task will get much of a response. [22:09:49] noooope [22:10:40] though Magnus edited today [22:11:18] AntiComposite: am I remembering correctly that this tool is linked to from article space in enwiki? [22:11:33] yes, every coordinate link on enwiki, commons, and most other wikis [22:12:16] :sigh: [22:14:01] some wikis replaced it with maplink links, but those still link to geohack because Kartographer doesn't link to nearly as many external servies [22:16:10] the wikivoyages have an extension that replaces it with Special:MapSources but it's not deployed elsewhere for reasons [22:19:03] !log bd808@tools-bastion-12 tools.geohack `truncate -s 0 error.log` [22:19:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geohack/SAL [22:20:53] !log bd808@tools-bastion-12 tools.geohack webservice stop; webservice php7.4 start [22:20:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geohack/SAL [22:25:01] It went right back to crashing when I turned it back on. "server.c.980) [note] sockets disabled, connection limit reached" -- I think it may be getting a large amount of traffic [22:34:42] T384092 if anyone has things to add to the bug report [22:34:43] T384092: geohack tool crashing repeately and quickly enough to trigger CrashLoopBackOff - https://phabricator.wikimedia.org/T384092 [22:35:36] !log bd808@tools-bastion-12 tools.geohack `webservice stop` (T384092) [22:35:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.geohack/SAL