[19:16:26] <RhinosF1>	 taavi: quarry nfs failed again
[19:22:31] <taavi>	 again?
[19:23:16] <TheresNoTime>	 mhm
[19:23:29] <TheresNoTime>	 T302154
[19:23:30] <stashbot>	 T302154: quarry-nfs-1 went down; quarry is offline - https://phabricator.wikimedia.org/T302154
[19:23:35] <taavi>	 !log quarry hard rebooted quarry-nfs-1 again T302154
[19:23:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[19:24:53] <taavi>	 andrewbogott: I think at least one of your shiny new nfs servers has a kernel bug :(
[19:25:13] <andrewbogott>	 quarry broke again, huh?
[19:25:17] <taavi>	 yeah
[19:25:25] <andrewbogott>	 Let me try rebuilding it with Buster and we'll see if that goes better
[19:30:08] <andrewbogott>	 In theory there's a ready-made process for migrating to a new server.  Let's see if it works!
[19:31:51] <TheresNoTime>	 I love the phrase "in theory" :')
[19:32:08] <andrewbogott>	 I tested it quite a bit but not with a running service :)
[19:32:25] <andrewbogott>	 taavi: In case you ever wind up wanting to mess with this yourself, the runbook I'm following is https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Create_an_NFS_server#Create_a_replacement_server_for_an_existing_service
[19:36:25] <taavi>	 woo, proper docs!
[19:38:27] <andrewbogott>	 yeah! I started to write those a few days ago and discovered I had already written them
[19:44:15] <andrewbogott>	 taavi: anything we should do before I try the failover?
[19:49:50] <andrewbogott>	 !log quarry moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue
[19:49:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL
[19:49:54] <stashbot>	 T302154: quarry-nfs-1 went down; quarry is offline - https://phabricator.wikimedia.org/T302154
[20:04:53] <andrewbogott>	 ok, that was shaky but it looks like things are working now
[23:03:58] <wm-bot>	 !log tools.bridgebot <bd808> Bridge #wikimedia-ve to Telegram (T299326)
[23:04:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[23:53:03] <wm-bot>	 !log tools.toolinfo-scraper <bd808> Update to 922ce7c (T294142)
[23:53:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolinfo-scraper/SAL
[23:53:26] <zhuyifei1999_>	 andrewbogott taavi: can you give me a ping if this quarry issue happens again?
[23:54:22] <andrewbogott>	 I would also like to be pinged :)