[00:34:42] <wm-bb>	 <abartov> looks like everything created in the buildservice shell evaporates, apparently including stuff created *via symlink* on the NFS drives!
[00:35:40] <wm-bb>	 <abartov> (I had to mv it away on NFS for *the files created from the buildservice shell on NFS* not to get wiped.)
[01:01:32] <bd808>	 @abartov: that sounds mysterious and worthy of a bug report in Phab so someone can try to reproduce your vanishing files problem and figure out what to do about it.
[01:07:25] <wm-bb>	 <abartov> bd808: done.
[01:07:36] <wm-bb>	 <abartov> https://phabricator.wikimedia.org/T411832
[01:08:25] <wm-bb>	 <abartov> It seems strange no one reported it sooner. Perhaps it's something wonky specifically with the (much less used) *Ruby* buildservice template, and that's why no one noticed so far?
[01:09:31] <wm-bb>	 <abartov> (side question: is the prompt inside the buildservice shell dramatically declaring "I have no name!" expected behavior?)
[01:09:31] <wm-bb>	 <abartov> I have no name!@shell-1764896572:/workspace$
[01:10:39] <bd808>	 yeah. the `I have no name!` bit just means that the runtime user is not present in the /etc/passwd inside the container.
[01:13:03] <bd808>	 @abartov: The /workspace directory is not an NFS mount. You need to `cd $TOOL_DATA_DIR` to get to the NFS space.
[04:13:20] <wm-bb>	 <abartov> what might cause this?
[04:13:21] <wm-bb>	 <abartov> Error from server (Forbidden): pods "shell-1764907933" is forbidden: exceeded quota: tool-gutensearch, requested: limits.memory=512Mi, used: limits.memory=7956527Ki, limited: limits.memory=8Gi
[04:13:22] <wm-bb>	 <abartov> I have no open shells running AFAIK. Is there a command to list any disconnected shells that may still be running on my behalf?
[04:17:58] <wm-bb>	 <abartov> ah, kubectl get pods. Found 'em. (re @abartov: what might cause this?
[04:17:58] <wm-bb>	 <abartov> Error from server (Forbidden): pods "shell-1764907933" is forbidden: exceeded quota: tool-gutensearch, re...)
[17:26:08] <wm-bb>	 <otckyc_2> If anyone need cloud service kyc or readymade account contact me 
[17:26:09] <wm-bb>	 <otckyc_2> Our service 24/7 we have a big team for work
[18:35:18] <mutante>	 !log codesearch re-enabled puppet on codesearch9 - service up; has double disk space vs before incident - T411047 T411728
[18:35:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[18:35:23] <stashbot>	 T411047: "error: No space left on device" for codesearch9:/srv - https://phabricator.wikimedia.org/T411047
[18:35:24] <stashbot>	 T411728: Codesearch down/unreachable (2025-12-03) - https://phabricator.wikimedia.org/T411728