[00:16:30] !log tools rebooted tools-sgeweblight-10-24, seems to be oom [00:16:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:22:59] !log tools rebooted tools-sgeweblight-10-30, oom [00:23:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:54:30] s5 & s7: ERROR 1044 (42000): Access denied for user 's51412'@'%' to database 'heartbeat_p' [07:55:45] https://replag.toolforge.org/ <-- shows 2^63-1 "seconds" for s5 & s7 which is ~292 billions (american billions) of years, a little bit older than our universe ;^) [08:41:21] Wurgl: it's known [08:41:29] ok [08:42:27] Wurgl: same root cause as the s1-3 lag [08:42:37] Just different stage of recovery [11:11:10] !log tools.network-tests delete pod by hand, it was stuck [11:11:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.network-tests/SAL [18:57:04] !log tools.copypatrol `webservice restart` for T337791, no effect [18:57:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.copypatrol/SAL [18:57:07] T337791: CopyPatrol error 500 - https://phabricator.wikimedia.org/T337791 [20:07:55] enwiki views on replicas down? [20:11:55] Yes, T337446 [20:11:55] T337446: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446