[08:25:37] for some reason, first byte latency of 4XX on ms-fe2 increased a lot, but godog you are only doing maintenance on eqiad, right? That's weird [08:26:04] https://grafana.wikimedia.org/d/000000584/swift-4gs?viewPanel=27&orgId=1&var-DC=codfw&var-prometheus=thanos&from=1632720356238&to=1632731096238 [09:02:11] jynus: checking [09:02:40] (for context, I am not running any backup at the moment) [09:04:40] jynus: that was a mistake on my end, I accidentally pushed old rings to codfw, should be recovering soon [09:05:03] ah, good that I noticed, then [09:05:37] indeed [09:06:07] can you pm or write on data-persistence swift planned things, I would like to plan for when to do codfw backups? [09:07:07] that sounds badly, I mean I want to plan for backups and would like to know more about what you are doing [09:08:03] so I don't cause issues for you [09:08:52] this is the routine hw eqiad expansion, perhaps we can subscribe data persistence to T290546 and T288458 ? I'd like to avoid having to basically !log twice for every operation [09:08:52] T288458: Put ms-be20[62-65] in service - https://phabricator.wikimedia.org/T288458 [09:08:52] T290546: Put ms-be10[64-67] in service - https://phabricator.wikimedia.org/T290546 [09:09:41] yes, I saw that, not asking to log it twice [09:10:21] just want to know future timeline and future pooling [09:11:54] (there is not much of that on the description) [09:12:49] sure sounds good, we'll discuss pooling/depooling swift on -data-persistence when the time comes, I don't have a timeline atm [09:13:07] no, I don't need it there, I can read the ticket [09:13:41] it is more of: I know eqiad was the only active dc for upload recently [09:14:06] will that change to codfw at some point, be kept like that, pool both dcs? [09:14:15] (for reads) [09:14:42] a "I don't know" is good enough for me, too :-) [09:15:58] I don't know :) [09:16:04] fair enough :-)