[06:51:31] <wm-bb>	 <anafibnshahibul> Hi everyone, I'm having an issue with my Toolforge PHP webservice.
[06:51:31] <wm-bb>	 <anafibnshahibul> My tool is tools.wikiforms hosted on Toolforge Kubernetes. The webservice keeps returning 503 no healthy upstream even after restarting.
[06:51:33] <wm-bb>	 <anafibnshahibul> What I've checked:
[06:51:34] <wm-bb>	 <anafibnshahibul> webservice status shows it's running as php8.2
[06:51:36] <wm-bb>	 <anafibnshahibul> service.manifest has web: php8.2
[06:51:37] <wm-bb>	 <anafibnshahibul> PHP syntax check passes: No syntax errors detected in all files
[06:51:39] <wm-bb>	 <anafibnshahibul> When I run php8.2 ~/public_html/api.php inside webservice shell, it outputs valid HTML (Laravel is working)
[06:51:40] <wm-bb>	 <anafibnshahibul> But curl https://wikiforms.toolforge.org/ still returns 503
[06:51:42] <wm-bb>	 <anafibnshahibul> Commands tried:
[06:51:43] <wm-bb>	 <anafibnshahibul> webservice stop
[06:51:45] <wm-bb>	 <anafibnshahibul> webservice php8.2 start
[06:51:46] <wm-bb>	 <anafibnshahibul> webservice restart
[06:51:48] <wm-bb>	 <anafibnshahibul> All return 503 after starting.
[06:51:49] <wm-bb>	 <anafibnshahibul> Question: Is there a known Kubernetes pod issue right now, or is there something else I should check? Is there a way to see the actual pod logs?
[06:51:51] <wm-bb>	 <anafibnshahibul> Thanks!
[06:52:21] <wm-bb>	 <lucaswerkmeister> `kubectl logs deployment/wikiforms` might show something
[06:55:33] <wm-bb>	 <anafibnshahibul> Thank you so much! The issue is resolved now.
[06:55:34] <wm-bb>	 <anafibnshahibul> The problem was a duplicate entry in .lighttpd.conf:
[06:55:36] <wm-bb>	 <anafibnshahibul> "^/hall-of-fame\.php$" => "$0",
[06:55:37] <wm-bb>	 <anafibnshahibul> "^/hall-of-fame\.php$" => "$0",
[06:55:39] <wm-bb>	 <anafibnshahibul> Removing the duplicate line fixed the 503 error immediately. Really appreciate the quick help!
[08:38:16] <lucaswerkmeister>	 !log tools.admin (re?)add Reedy (should be covered by T249774; SSH key still present in puppet.git root-authorized-keys.erb AFAICT)
[08:38:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL
[08:38:21] <stashbot>	 T249774: Grant "Cloud admin" rights to Reedy - https://phabricator.wikimedia.org/T249774
[10:38:08] <wm-bot>	 !log jeanfred@tools-bastion-15 tools.integraality Deploy 6b79fa1 (Add noscript fallback for streaming update page) for T425477
[10:38:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[14:38:23] <godog>	 !log admin switch from SystemdUnitDown (eqiad only) to SystemdUnitFailed (codfw/eqiad) alerts -- there might be some codfw noise coming - T428873
[14:38:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:38:30] <stashbot>	 T428873: Ditch wmcs-specific alerts for systemdunitdown in favor of production alerts - https://phabricator.wikimedia.org/T428873
[14:53:58] <dcaro>	 ack
[14:54:26] <dcaro>	 godog: is this part of it? https://alerts.wikimedia.org/?q=team%3Dwmcs&q=alertname%3DSystemdUnitFailed
[14:54:30] <dcaro>	 (it's not codfw)
[14:55:34] <dcaro>	 I think it's a side effect of the other silence
[14:56:24] <godog>	 dcaro: heh indeed, the codfw one(s) I was expecting, the clouddb ones I wasn't
[14:56:29] <godog>	 I'm taking a look
[14:56:51] <dcaro>	 pinged cezmunsta in -sre
[14:57:51] <godog>	 ack thank you, will followup there
[17:03:42] <andrewbogott>	 jelto: I'm noticing that gitlab-1002 reports as 'down' and the console shows the network as unreachable. My question for you is: did it /ever/ work, or did the network setup fail on creation?
[17:14:19] <wm-bb>	 <nemoralis> bd808: thanks! filed https://phabricator.wikimedia.org/T432761
[18:26:54] <mutante>	 andrewbogott: that worked and it seems like you literally just fixed network?
[18:27:19] <andrewbogott>	 dhclient was not running and it didn't have an IP assigned. No guess how it got that way...
[18:27:26] <mutante>	 that would be the backend of https://gitlab.devtools.wmcloud.org/explore/projects/active and a minute ago I could not see it but now I can
[18:27:27] <andrewbogott>	 but starting dhclient seems to have cheered it up
[18:27:41] <andrewbogott>	 well you checked at exactly the right moment :)
[18:27:42] <mutante>	 so I wanted to answer that question with "yes, used to work"
[18:27:47] <mutante>	 and now it's back :)
[18:27:54] <mutante>	 ok, cool
[21:10:36] <grin>	 Hello, I rarely fiddle with job start/stop, but now I'm a bit puzzled. toolforge jobs list shows zero running jobs and one scheduled, while https://k8s-status.toolforge.org/namespaces/tool-commons-delinquent/ shows two running (apart from a shell I opened).
[21:10:49] <grin>	 Where are these listed? How can I stop / restart them?
[21:20:46] <grin>	 eh, asking the AI actually works, seems Gemini is pretty familar with toolsforge's internals. So the job was orphaned by flush, and needed to be found and killed by kubectl....
[21:20:49] <grin>	 THanks. :)