[01:05:00] how should i? sorry im not sure, let alone their time (re @lucaswerkmeister: you could try contacting the Members or Viewers listed at https://openstack-browser.toolforge.org/project/math (Physikerwelt bei...) [01:34:33] andrewbogott: mathoid on math has been timing out since a good while, i dont know if months or something, 504 gateway timeout and this affects mathoid service on mediawiki installations with math installed [01:34:42] you're the only contact available [01:34:47] on chat that is [02:19:18] Juest: I can take a look in a few minutes but I don't know much about that project (I'm in a lot of admin lists because of having created initial projects.) Physikerwelt is the only active admin as far as I know. [02:24:03] andrewbogott: Where can i contact him or her? [02:29:03] hm, not sure, looks like they don't have a user page on wikitech [02:29:48] I've checked wikitech and meta without results [02:30:07] the other sre people either have outdated profiles or are not on chatrooms [02:34:55] Juest: want to pm me your email? I will forward it to physikerwelt and see if they want to follow up with you. [02:35:03] It may be that the've moved on and the project is abandoned though. [02:35:39] could you at least look at the server logs for an indicator of when it stopped responding? [02:36:19] everything else on that instance seems to be working except the mathoid service [02:38:36] oh, you have a login on the instance? You're a project member? [02:38:59] no, im seeing the other sites linked in proxies as working [02:39:11] i checked the live domains listed in projects [02:39:24] im not a project member, my wikitech user is juest [02:39:52] I was confused by you saying 'on that instance' [02:39:56] in https://openstack-browser.toolforge.org/project/math on the proxies section you can click on the domains to check them [02:43:49] sorry about that, just happens that i can access/see things with the public viewable information on the openstack browser [02:44:48] effectively, port 10042 is timing out [02:44:58] or i dont know whats going on [02:45:39] there are dozens of docker images on the host at that endpoint so it's not going to be trivial to figure out what should be responding and what's going wrong. [02:46:11] you could also open a phab task and assign it to them. But -- it's a volunteer-run project, and 'beta' is right there in the name so don't expect a high level of service :) [02:47:35] I see [02:48:04] sounds unfortunate that the mathoid service is coming to an end [02:48:58] I can open a phab task assigning them [02:49:39] andrew, what projects should i set the task to? [02:54:14] no idea, there might be a math or mathoid project [02:57:14] huh, it appears that the mathoid test page works but the web server is timing out [02:57:58] i got a bad request return on mathml [03:02:10] got it, it appears there been other similar tasks reporting mathoid being down but nothing like this gateway timeout specifically [03:07:26] andrewbogott thanks again, looks like i'll be posting on phabricator [03:07:45] ok! [03:10:27] I've found this https://www.mediawiki.org/w/index.php?title=User:Physikerwelt [03:10:34] and the information is outdated [08:47:02] !log toolsbeta switch project to new DNS recursor via horizon project hiera (T345240, T342621) [08:47:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:47:07] T345240: cloudservices1006: put into service - https://phabricator.wikimedia.org/T345240 [08:47:07] T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP - https://phabricator.wikimedia.org/T342621 [11:59:36] Still unable to load sudoers on Horizon. Any idea what's going on here? [12:12:45] Skynet: which project? [12:12:52] (horizon project), and which user? [13:49:27] dcaro: the sudo panel fails for me too [13:51:12] Skynet: is there a phab task for that? [14:13:36] andrewbogott: ack, I got it too, any project it seems [14:13:56] probably broke during the move to docker, I'm going to see if I can reproduce locally [14:19:23] I see the log in the container, there's not much more in it xd [14:19:37] is it trying to connect to ldap? do you know? [14:21:55] I don't think it's getting as far as ldap but I don't know much -- I see the same unhelpful logs as you [14:47:37] looks like mathoid-beta was unresponsive and got restarted, my mathoid problem is solved (i am on 1.35) [15:04:35] andrewbogott: not that I know of. I don't know if this is just some transient thing that can be fixed in 10 seconds or not. [15:38:32] bullseye down? "Server Error (500)" [15:38:52] https://bullseye.toolforge.org/ip/%E2%80%8E92.118.187.6 [15:41:34] GenNotability: ^ [15:50:34] andrewbogott: is this going to need a phab ticket> [15:52:25] only if you want to be helpful :) I can make one otherwise [16:01:32] andrewbogott: can I run interference instead? :p [16:33:02] MdsShakil: given the link you sent with the non-printable characters at the front, it looks like some odd characters snuck into your IP address of choice [16:35:15] GenNotability: could it catch it? [16:45:28] I could [22:40:04] there is an empty database vewikimedia_p is s3, the real vewikimedia_p is in s5, maybe someone want to drop it