[08:12:52] thanks David, giving a heads up / FYI as you did is good as a general rule. [08:16:53] For the traffic ramp-up stages you want to do over the coming week, we were discussing with jhathaway to have a form of premortem in a doc or the Phab task, i.e.: 1. add there the link to the graphs/metrics you look at and how to know if something goes wrong [08:16:53] 2. for each stage prepare the rollback CR for oncallers and add it there just in case. [08:16:53] 3. give a heads up to oncallers before ramping up each stage with the link to that doc/task [14:27:40] slyngs jasmine_ bjensen hnowlan this is a potential addition to the Switchover process that came after codfw outage: https://phabricator.wikimedia.org/T440178 do you think this is necessary before Oct 26th switchover? [14:28:50] I would say yes, I think it's pretty cheap to add to the process. [14:29:09] and given The Previous Events it would be a waste not to [14:29:35] That's basically for free [14:30:25] Only issue is: What to do with a switch-over that is planned two months in advance if a maintenance thing comes in the week before? [14:31:51] I think that's up to the operator to evaluate - 99% of maintenances are still safe to switchover during, it's mostly a matter of checking with stakeholders about the level of risk [14:33:17] imo if we had seen the codfw maintenance communication before the switchover we still would have gone ahead, imo this is mostly for awareness and ability to recognise what's going wrong [14:35:30] more generally I don't think we should get super gunshy about things as a result of having been through the worst case scenario, just be sure we have better awareness [14:42:04] got it, thanks. I'll move it under the Oct switchover umbrella, and Jasmine and Simon you can discuss and add to runbook