[06:59:08] Krinkle: thanks I think I understand what you mean now, which containers are sharded and how many ways/levels is controlled by mediawiki/filebackend, not swift. specifically $wmfSwiftBigWikis and related [07:11:37] Krinkle: also to clarify, when you say "shard score into 16 segments" I take that to mean to have 16 different swift containers [08:40:07] be careful! I've been told Jaime is fishing for sanity check reviews for access requests: https://gerrit.wikimedia.org/r/c/operations/puppet/+/715204 😇 [13:14:17] hnowlan: o/ anything to be done for the maps servers? I see a lot of alerts in icinga :( [13:36:32] volans, thanks for reaching out on T287465. I left a comment noting I can't seem to run the add-wiki cookbook anymore. If there's anything I can do "live" with you to help debug, feel free to ping me. [13:36:32] T287465: Cookbooks repository: avoid stale code in master branch - https://phabricator.wikimedia.org/T287465 [13:48:06] elukey: agh, thanks for the heads-up. [13:48:54] hnowlan: we can help too, I didn't mean "you fix it", please lemme know if I can help! [13:49:58] Something has happened that caused everything to drop out of replication with the master. They still function fine for the most part (and aren't pooled) but I'll start the process of resyncing them now [13:50:15] <3 [14:18:28] Measure storage solutions. New post by Dan Luu. https://danluu.com/why-benchmark/ [16:01:32] hnowlan: we got a warning for maps1009, BW on its port maxed out past few hours - I assume that relates to you kicking off the resync though so nothing to worry about. [16:02:49] topranks: yep that would be it - anything I should do in advance of similar in future to avoid it alerting? [16:02:56] 1009 is the master [16:03:29] nah it's fine, its only a warning level I just happened to notice it and wanted to double check / let you know in case. [17:40:32] volans: shouldn't https://github.com/wikimedia/puppet/commit/722e33ab53c8ab2e71b0a98b44def9ce038bf9a0 be secure-cookbook not cookbooks [17:59:36] RhinosF1: yes ofc! Good catch, thanks. Today has been a rollercoaster day IRL for me, so I'm back online only now [17:59:46] balloons: ^^^ that should fix it, sending patch [17:59:56] and sorry for the late reply, same reason :) [18:00:22] volans, no worries at all. I didn't need to run the cookbook today, so it's not blocking me :-) [18:00:32] I appreciate you updating the docs as part of the change [18:03:07] https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/713931/ [18:03:24] ops, wrong link [18:03:28] https://gerrit.wikimedia.org/r/c/operations/puppet/+/715271 [18:06:22] balloons: patch merged and puppet run on the cumin hosts, if you're still up for testing feel free to retry ;) [18:07:04] volans, seems to be working now! [18:07:44] great, again, sorry for the delay and thanks to RhinosF1 for spotting it for me even before I started looking into it [18:07:54] volans: I'm nosey [18:08:05] I wondered what secure cookbooks was so went looking [18:08:11] :) [18:12:03] volans: I saw you cc'd me on patch, ty! I was in shower [18:13:20] yw, it was the least I could do :) [18:13:52] * RhinosF1 likes being nosey [18:13:55] Fun way to learn [18:15:19] if you look enough at something you'll surely find a rabbit hole to look into, and probably something to fix ;) [18:17:19] Hehe :) [18:17:24] True though [22:10:21] godog: thanks, I wasn't aware of that. Okay, so that is actually regexp based, and fully client-side. [22:11:27] I assume then that this kind of sharding isn't a built-in feature of swift but something we do on our own, meaning, any changes to it have to somehow be migrated or fallback or synced in some way that might even require read-only time?