[09:15:27] the number of backed up media files from is now close to 150 million [10:43:53] Happy monday [10:44:06] https://usercontent.irccloud-cdn.com/file/h8A1Pmbi/image.png [10:53:19] it's clear from that one that the important bit is to not look while you're doing it ;) [11:13:53] :D [15:35:01] jynus: thanks for the link you shared ( https://www.honeycomb.io/blog/testing-in-production ) [15:35:41] so I wanted to be clear that I wasn't proposing not to test before [15:35:54] but to test before & in production [15:36:11] by adding production checks to every task performed [15:36:23] IMO I find the title a bit clickbait but the contents are good. I'm a bit surprised the author did not describe this process as staged rollout [15:36:24] specially regarding state [15:36:37] no, because that is a method, not the goal [15:36:54] you test in production and do canaries to mitigate issues [15:37:07] but the goal is to test in production (too) [15:37:11] let me give you an example [15:37:56] this is production testing: https://phabricator.wikimedia.org/T207253 [15:38:17] and is necessary independently of the unit & integrations tests you do on your code and migrations [15:42:12] if we don't test in production we should remove alertmanager! [15:44:20] as well as k8s rollbacks! [17:28:57] o/ We're planning on doing the codfw->eqiad DC switchover on the 18th of March, just wanted to give ye an explicit heads-up about it given the circular replication and all https://phabricator.wikimedia.org/T385155 [17:30:05] should all be good, just be weary of white smoke [17:31:04] haha :D/D: [18:29:16] This is not what is meant by a smoke test, people :)