[15:48:09] <wikibugs>	 10Countervandalism-Network, 13Patch-For-Review: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#10864502 (10Krinkle)
[16:21:26] <Krinkle>	 2025-05-28T16:21:12.404112+00:00 cvn-app13 CVNBot.exe[4637]: ERROR [Thread Pool Worker] CVNBot.Program [CVNBot17] Caught unhandled exception in global catcher#012Mono.Data.Sqlite.SqliteException (0x80004005):
[16:21:34] <Krinkle>	 I can't seem to get CVNBot17 to keep running.
[16:21:42] <Krinkle>	 I'm not sure where the readonly state is being kept. 
[16:22:04] <Krinkle>	 Tehre's also a problem with lots of concurrent instances being spawned which makes it worse since that means they also have locks.
[16:22:09] <Krinkle>	 I haven't yet figured out which one happens first.
[16:22:16] <Krinkle>	 Maybe it's the root issue.
[18:26:45] <Krinkle>	 https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?var-project=cvn&orgId=1&from=now-6h&to=now&timezone=browser&var-instance=cvn-app13
[18:26:59] <Krinkle>	 can't even ssh in the thing anymore, so busy.
[18:27:00] <Krinkle>	 rebooting...
[21:08:43] <Krinkle>	 OK. The lock definitely happens before the process growth
[21:09:39] <Krinkle>	 killed all procs, moved a fresh copy of a working db file in-place, tailed syslog, then ran cron/stillalive to spawn the first proc, and that first PID encounters the lock before it decides to restart/spawn a new child.  
[21:09:53] <Krinkle>	 Which then seems to encounter a bug where the original proc doesn't die, but that's a separate issue
[22:48:53] <wikibugs>	 (03PS1) 10Krinkle: Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1151809 (https://phabricator.wikimedia.org/T395164)
[22:48:59] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1151809 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[22:49:24] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Move CVNBot17 (cvn-wikivoyage) from cvn-app12 to cvn-app13" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1151809 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[22:50:00] <wikibugs>	 10Countervandalism-Network, 13Patch-For-Review: Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#10866146 (10Krinkle) >>! In T395164#10866143, @gerritbot wrote: > Change #1151809 had a related patch set uploaded (by Krinkle; author:...