[00:01:50] <bd808>	 AmandaNP: https://whois-referral.toolforge.org/ is alive again.
[00:02:26] <AmandaNP>	 thank you
[00:04:37] <bd808>	 !log tools Rebooting tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud (T335543)
[00:04:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:04:37] <stashbot>	 T335543: Pods getting stuck in "Terminating" status - https://phabricator.wikimedia.org/T335543
[00:07:08] <bd808>	 !log tools Hard reboot tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud via horizon (T335543)
[00:07:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:09:41] <bd808>	 !log tools `kubectl uncordon tools-k8s-worker-67` (T335543)
[00:09:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:09:44] <stashbot>	 T335543: Pods getting stuck in "Terminating" status - https://phabricator.wikimedia.org/T335543
[07:20:07] <dcaro>	 !log tools rebooting tools-sgegrid-shadow due to stale nfs mount
[07:20:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[08:27:06] <dcaro>	 !log tools rebooting tools-sgeweblight-10-28 (T335336)
[08:27:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[08:27:14] <stashbot>	 T335336: [toolschecker] jobs mtime check is flapping - https://phabricator.wikimedia.org/T335336
[09:44:38] <wm-bb>	 <Yetkin> /vi@wmtelegram_bot
[09:45:07] <wm-bb>	 <Yetkin> /vi@wmtelegram_bot
[10:37:17] <wm-bot>	 !log tools.wdmm <lucaswerkmeister-wmde> deployed ec0d05bdc3 (commonswiki vi)
[10:37:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL
[15:01:46] <arturo>	 !log tools force reboot tools-k8s-worker-79, unresponsive
[15:01:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:08:37] <legoktm>	 Some really weird issue in that my Rust bot process is sometimes running forever doing nothing now that I moved to toolforge-jobs
[16:08:41] <legoktm>	 it never had this issue on the grid
[16:08:52] <legoktm>	 But... | Status:     | Running for 2d4h7m                                               |
[16:09:55] <legoktm>	 -  command: /data/project/dbreps/src/database-reports/target/release/dbreps2
[16:09:55] <legoktm>	 +  command: /usr/bin/timeout 59m /data/project/dbreps/src/database-reports/target/release/dbreps2
[16:10:29] <arturo>	 it could be that kubernetes somehow lost track of the status of the process
[16:10:51] <arturo>	 I guess we could be a bit more cloud-native and introduce some kind of liveness checks into toolforge jobs
[16:14:35] <arturo>	 legoktm: T335592
[16:14:36] <stashbot>	 T335592: Toolforge jobs: consider having a way for jobs to report their liveness status to kubernetes - https://phabricator.wikimedia.org/T335592
[16:14:48] <legoktm>	 hmm
[16:15:00] <legoktm>	 this is the second time it's happened in less than a week, I would have expected more people to complain if it was that frequent of an issue?
[16:15:37] <legoktm>	 it's possible this is something related to this specific bot, my other Rust bot (significantly less complex) has been on toolforge-jobs for months now with no issue
[16:19:09] <taavi>	 could we attach a debugger or similar to figure out what it's stuck on?
[20:22:21] <wm-bot>	 !log tools.speedpatrolling <lucaswerkmeister> deployed 3cb8ab732d (update dependencies, Flask/Werkzeug 2.3)
[20:22:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL
[22:56:00] <wm-bb>	 <harej> Does Quarry work with the Toolforge user DBs?
[22:56:10] <wm-bb>	 <harej> Assuming the DB in question is a “_p”
[23:01:07] <wm-bb>	 <lucaswerkmeister> hm, I doubt it
[23:01:47] <wm-bb>	 <lucaswerkmeister> it’s not documented and the source code doesn’t look like it either (especially `quarry/web/replica.py`), but also that `git grep tools` doesn’t find much that looks relevant, and nothing for `tools.db`)