[05:22:41] I've done a switchover on m2, which contains requestctl, everything seems to be working but if anyone notices something strange, let me know [06:07:36] <_joe_> marostegui: you mean within the same datacenter, right? [06:09:10] <_joe_> yeah things seem to be working, although I'm not sure the change was actually picked up or not [06:11:21] _joe_: yeah within the same [06:19:23] <_joe_> marostegui: what's the new db? just to verify we aren't writing to the replica :P [06:19:36] _joe_: you can't because it is RO and the old master is down :) [06:19:40] db1290 is the new master [06:20:11] _joe_: | 28321 | gus_gus | 10.64.16.7:57838 | requestctl | Sleep | 496 | | NULL | 0.000 | [06:20:15] that's on the new master [06:20:40] <_joe_> yeah and I just checked my last write is there as well [06:20:42] <_joe_> all good :D [06:20:48] thanks for checking :) [11:50:38] fyi, codfw rack A7 maintenance starting in ~15min [17:26:09] <_joe_> anyone knows why kafkacat is nowhere to be found on the kafka servers anymore? [17:54:42] <_joe_> cscott: we need to rollback your previous bump of parsoid to the new version [17:54:48] <_joe_> it's causing timeouts in prod [17:54:53] <_joe_> or, we suspect so [17:57:12] ok [17:57:29] <_joe_> let me recheck [17:57:33] do you want me to create the patches? [17:58:30] <_joe_> cscott: what was the patch for that bump, sorry? [17:58:57] it's a pair of patches, one in vendor one in core [17:59:09] [Bump wikimedia/parsoid to 0.24.0-a12 (1305720) · Gerrit Code Review](https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1305720) [17:59:09] <_joe_> we might not need to roll them back [17:59:20] <_joe_> hold a sec on this :) [17:59:27] [Bump wikimedia/parsoid to 0.24.0-a12 (1305719) · Gerrit Code Review](https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/1305719) is the other one [17:59:53] <_joe_> yeah we were off by one hour I think [18:00:01] <_joe_> it shouldn't be your patch that was problematic [18:00:15] if it helps, we can roll back anyway just to reduce the possible culprits [18:00:48] our new parsoid version doesn't fix the bugs its meant to solve without a config patch, and we have to rollback the config patch when the fires started and scap got stuck [18:57:18] <_joe_> sorry cscott didn't see your message; no I think we should be ok [21:18:57] cwhite OK if I puppet-merge you change? [21:20:28] inflatador: yes please! [21:20:34] cwhite ACK, done