[06:34:24] \i/ [07:59:09] Nice, thanks for the update Amir [07:59:28] looking forward to see the impact [11:48:21] https://grafana.wikimedia.org/goto/bfvnegzhgpp8gf?orgId=default [12:31:44] https://phabricator.wikimedia.org/T366938#12229060 Cool to see the immediate impact [13:29:32] pretty! [13:29:54] a graph that looks good for once! :D [13:50:35] <_joe_> that change means that our redis servers are more critical than before; please document it where appropriate [14:57:58] _joe_: It's a separate group of redis servers [14:58:36] Amir1: mediawiki can handle one of three going down at a time right? [14:59:12] yes, it can handle it no problem [14:59:35] If two go down what happens? :P [14:59:58] one third of cases would be slow but there might be issues here and there [15:00:30] on the risk matrix of likelihood vs impact. It's acceptable risk IMHO [15:04:19] <_joe_> claime: yeah my point was we can't reboot two of them at the same time [15:04:22] <_joe_> for instance [15:04:33] <_joe_> and/or if we might want to do failover to the replica if that happens [15:05:05] There's no replica, they're standalone servers [15:18:04] yeah, as long as it's done one by one, anything is okay [15:20:04] they are basically temu version of redis HA cluster [15:20:19] <_joe_> so much disrespect to temu smh [15:20:37] <_joe_> but yeah, my point was "there is some care to apply when rebooting them" [15:22:10] about to deploy it everywhere (https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1327138) now https://usercontent.irccloud-cdn.com/file/bGMTJRo6/grafik.png [15:34:59] the memory usage has shot up... to 14MB... in total