[07:01:33] running `shell.php` on `mwscript-k8s` is taking ~2-3 minutes to launch, is this expected? [07:08:55] I am also seeing that deployments of patches are going very slowly [08:59:22] 06serviceops, 10Ceph, 10envoy, 06SRE, 10SRE-swift-storage: Data-persistence envoy upgrades to 1.26.8-1 - https://phabricator.wikimedia.org/T403374 (10MatthewVernon) 03NEW [09:05:06] 06serviceops, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#11135433 (10Ladsgroup) >>! In T324866#11134539, @daniel wrote: >>>! In T324866#11134511, @Kelson wrote: >> @daniel Hi... [09:24:23] 06serviceops, 10Ceph, 10envoy, 06SRE, 10SRE-swift-storage: Data-persistence envoy upgrades to 1.26.8-1 - https://phabricator.wikimedia.org/T403374#11135496 (10MatthewVernon) [09:44:13] very stupid request. Would be okay to flush old keys of redis? See https://phabricator.wikimedia.org/T386640#10873880 [10:13:46] 06serviceops, 10Ceph, 10envoy, 06SRE, 10SRE-swift-storage: Data-persistence envoy upgrades to 1.26.8-1 - https://phabricator.wikimedia.org/T403374#11135684 (10MatthewVernon) [10:20:08] kostajh: you [10:20:34] kostajh: did you manage to finish up ? [10:21:02] yes, deployments finished a while ago [10:21:07] afaik [10:21:40] Amir1: what if we do so after the code that adds TTLs is in place? [10:22:11] see if we can find a path to prioritise this [10:23:13] effie: I don't know an easy way to introduce that [10:23:47] it's not impossible but some amount of work [10:25:27] personally I find this OK, to at least bandaid this, but I may be naive of the side effects [10:45:22] 06serviceops, 10Ceph, 10envoy, 06SRE, 10SRE-swift-storage: Data-persistence envoy upgrades to 1.26.8-1 - https://phabricator.wikimedia.org/T403374#11135763 (10MatthewVernon) [10:45:29] 06serviceops, 10Ceph, 10envoy, 06SRE, 10SRE-swift-storage: Data-persistence envoy upgrades to 1.26.8-1 - https://phabricator.wikimedia.org/T403374#11135764 (10MatthewVernon) 05Open→03Resolved [16:18:14] 06serviceops, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#11136955 (10Benoit74) > Looking at the current numbers, things got worse: we are now seeing nearly 10,000 requests per... [17:33:44] 06serviceops, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#11137169 (10daniel) >>! In T324866#11136955, @Benoit74 wrote: > This has been advised by and discussed with @cscott in... [22:23:19] 06serviceops, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#11137499 (10Ladsgroup) >>! In T324866#11136955, @Benoit74 wrote: >> Looking at the current numbers, things got worse:...