[13:06:45] !log lucaswerkmeister@tools-bastion-13 tools.pagepile kubectl rollout restart deployment pagepile # T400995; just as in https://sal.toolforge.org/log/VweuxZYBffdvpiTrxEE7, `webservice restart` seemed confused about whether the webservice was running or not [13:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile/SAL [15:59:23] I need a cloud VPS network admin/expert [16:00:20] Especially one that can read tcpdumps [16:01:04] I suspect the issue is MTU related [16:43:42] !log lucaswerkmeister@tools-bastion-13 tools.pagepile webservice stop && webservice --health-check-path=/howto.html php7.3 start # try to add health-check-path to avoid recurrence of T400995; appears to have broken the tool šŸ˜¬ [16:43:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile/SAL [16:44:34] anybody got an idea why https://pagepile.toolforge.org/ is just giving 403 Forbidden now? should be standard lighttpd PHP toolā€¦ [16:46:38] oh, https://pagepile.toolforge.org/howto.html works [16:49:05] running `php public_html/index.php` from inside the kubernetes container appears to work just fine [16:49:29] you know what, lemme try php7.4 [16:50:00] !log lucaswerkmeister@tools-bastion-13 tools.pagepile webservice stop && webservice --health-check-path=/howto.html php7.4 start # try php7.4 instead of php7.3 [16:50:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile/SAL [16:50:13] that looks better [16:50:56] so, good to know, the php7.3 webservice type is apparently not just deprecated but outright broken? [16:51:05] though it still worked after the rolling restart earlier today [16:51:29] not sure what the difference between them is (would the rolling restart have kept the same, possibly-older version of the image?) [16:55:35] ok, I can reproduce the same behavior in another tool. https://lucaswerkmeister-test.toolforge.org/hello.html works, https://lucaswerkmeister-test.toolforge.org/hello.php is 403 Forbidden [16:56:16] Iā€™ll go file a phab task (though I wouldnā€™t be upset if it gets closed as WONTFIX ^^) [16:59:19] T401014 [16:59:20] T401014: php7.3 webservice type unable to run PHP - https://phabricator.wikimedia.org/T401014 [20:09:49] Sometimes it seems like I'm getting a 301 redirect from https://tools-static.wmflabs.org urls to redirect to https://tools-static.wmflabs.org:8000 (!) [20:10:12] But only intermittently. Not sure what it is going on, but once i get one they seem to get cached locally :( [20:11:25] e.g. for https://tools-static.wmflabs.org/bawolff/captcha/tim i seem to get it consistently [20:11:44] location: http://tools-static.wmflabs.org:8000/bawolff/captcha/tim/ - so it redirects both to http and port 8000 [20:20:18] something weird seems to happen when I try to fetch anything under https://tools-static.wmflabs.org/bawolff. [20:20:55] * bd808 doesn't remember any method for customizing that service [20:21:06] I don't think I've set anything custom or doing anything out of the ordinary [20:25:10] there's a symlink, but that's not weird [20:25:15] bawolff: I think this is a strange side effect of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1165841, but I'm not sure exactly how. [20:26:36] interesting, https://tools-static.wmflabs.org/bawolff/test.txt got two x-clacks-overhead headers [20:26:57] I think maybe its because nginx will send a redirect if you try to view a directory without including the trailing / [20:27:24] I think you are on to something there. [20:27:45] But I'm surprised it thinks its on port :8000, i would assume it would use the HTTP host header for that [20:28:27] https://tools-static.wmflabs.org/bawolff/ -- fast, works as expected. https://tools-static.wmflabs.org/bawolff - slow, ends up with a redirect to the port 8000 backend nginx [20:30:25] maybe you need a port_in_redirect config option http://nginx.org/en/docs/http/ngx_http_core_module.html#port_in_redirect [20:31:08] there's also an absolute_redirect and server_name_in_redirect options that look related [20:31:19] * bawolff not really sure, i haven't really used nginx [20:33:33] bawolff: I think ing you file a bug explaining that the `foo` -> `foo/` redirect in nginx is exposing the port 8000 backend URL to the client t.aavi will be able to figure out if haproxy or nginx needs a different config to make things work as expected. [20:33:43] *I think if [20:34:04] -šŸ‘ Thumbs Up Emoji | Meaning, Copy And Paste [20:34:05] * bd808 feels like his keyboard/irc client are inventing typos [20:34:08] well,t hat was a fail [20:34:42] anyways, i meant to paste only the thumbs up emoiji [20:34:56] :+1: ;) [20:37:08] All the discord kids are rubbing off on me with their emoijis [20:38:47] https://phabricator.wikimedia.org/T401024