[00:52:59] <taavi>	 i'm looking at toolsdb
[00:53:57] <taavi>	 oom killed once again
[00:53:58] <taavi>	 [3211718.996904] Out of memory: Killed process 725680 (mysqld) total-vm:64552308kB, anon-rss:63683220kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:125676kB oom_score_adj:-600
[00:55:03] <taavi>	 last one was on nov 9th, it made it almost a month without crashing
[00:55:11] <andrewbogott>	 oops I just rebooted it out from under you sorry
[00:55:26] <andrewbogott>	 But I imagine you were about to do that anyway :)
[00:55:56] <andrewbogott>	 taavi: do you mind setting r/w after it comes up?  I have food on the stove that's about to burn :)
[00:56:00] <taavi>	 sure
[00:56:18] <taavi>	 yeah I was just going through lots of email from the past week and happened to spot that particular one just as it got in
[00:56:22] <andrewbogott>	 thank you! I'll check back in a few to make sure it was an easy resolution
[00:58:24] <taavi>	 mariadb is taking suspiciously long to even start
[00:58:54] <taavi>	 there we go
[07:14:07] * dcaro paged
[07:16:57] <dcaro>	 oom killed too
[07:16:57] <dcaro>	 Dec 09 07:03:41 tools-db-1 systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
[07:18:29] <dcaro>	 I'm starting it again
[07:19:32] <dcaro>	 made it writable again
[07:30:54] <dcaro>	 created T353093 to follow up
[07:30:55] <stashbot>	 T353093: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093
[07:34:17] <dcaro>	 everything seems stable, /me back to sleep
[07:34:42] <dcaro>	 I wonder if I'm getting charged by the oncall sms... it comes from a foreign phone number (+1 ...)
[11:33:45] <dcaro>	 Oh, paged again, but I'm not near a laptop
[11:34:13] <dcaro>	 I'll try to do something, but might take me a while
[11:50:25] <dcaro>	 Manually set the db as writable again, it was up and running :/
[16:32:35] <andrewbogott>	 I am in a car so not very available but I'm tryiing to restart the db yet again :/
[16:38:18] <andrewbogott>	 dcaro: are you awake/around by chance?
[16:38:37] <andrewbogott>	 and/or rook?
[16:38:57] <dcaro>	 Oh, ok. I'm around
[16:39:00] <dcaro>	 Let me look
[16:39:43] <andrewbogott>	 I'm on db-1 and mariadb won't start up
[16:39:47] <andrewbogott>	 am I on the wrong VM?
[16:39:56] <andrewbogott>	 (I rebooted it a few minutes ago)
[16:40:23] <dcaro>	 tools-db-1 was the one
[16:41:19] <andrewbogott>	 well /now/ it's running, did you do anything other than wait?
[16:41:26] <dcaro>	 just started it
[16:41:37] <andrewbogott>	 huh
[16:41:37] <andrewbogott>	 ok
[16:41:40] <dcaro>	 root@tools-db-1:~# systemctl start mariadb
[16:41:54] <andrewbogott>	 Is it ok if I leave you to it for a few? I have a realtor waiting on me.
[16:41:57] <dcaro>	 just set it as read-write
[16:42:00] <dcaro>	 ack
[16:42:00] <andrewbogott>	 well, maybe more than a few :/
[16:42:14] <andrewbogott>	 My mistake was maybe thinking that puppet would start it
[16:42:19] <andrewbogott>	 Thank you!
[16:44:26] <dcaro>	 yep, puppet does not touch it iirc (for safety reasons), and it also starts as ro
[16:57:09] <andrewbogott>	 I'm temporarily back! Are we all good?
[16:57:38] <dcaro>	 so far yes, I suspect that there's something making it crash (a user query someone sends)
[16:57:47] <dcaro>	 but not sure
[16:58:11] <dcaro>	 up and running for now, we can wait and see if it crashes again (it's 4 times today)
[17:01:01] <dcaro>	 only thing that comes to mind is enable swap, and instead of hard crashing, send an alert when the swap starts getting used so we can go and check what's going on
[17:01:21] <dcaro>	 (though the performance of the DB might get unusable, we might be able to find the cause for the crash)
[17:03:13] <andrewbogott>	 Ok! I'm going back to my day for now but we should talk about it more on Monday. Thanks for the rescue!
[17:03:32] <dcaro>	 👍
[17:03:35] * dcaro same