[09:11:34] maybe not a big deal but I'm looking at the cache hit ratio of thumb.wm.o and it doesn't have anything with hit-local while upload.wm.o thumbs do. Maybe it's intentionally not picked, just worth mentioning. https://w.wiki/UQPj vs https://w.wiki/UQPq [09:25:25] That's a little weird. I mean I'd rather have hit-front, but no local seems odd. I would if we're not setting the header correctly on text [09:27:48] hey folks! I am working on adding lvs in front of pki servers, but I am not 100% sure how to proceed now. The main issue is that we already have an entry in service.yaml for pki, that just points to pki1002's and pki2002's IP addresses (no lvs etc..). [09:28:09] I realized the problem when working on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338936, since IIUC until the proper LVS entries in service.yaml are added it won't work [09:29:14] so in theory, I could merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338937 first, with puppet disabled on lvs servers. Then merge the other one, and run it on pki nodes (to get the loopback ips populated etc..) [09:29:25] and finally run puppet on lvs hosts as usual to set up the service [09:29:29] does it make sense? [09:29:51] the other bit is related to the fact that pki.discovery.wmnet is already working [09:30:13] That's a little more LVS than I'm comfortable with. I think fabfur is out today, so we might have to wait until bblack or sukhe is around later [09:30:46] no problem! I am writing stuff in here as brainbounce attempt, I wouldn't touch LVS on a friday :D [09:31:19] That would just be asking for trouble :-) [09:37:08] elukey: definitely not when I'm oncall [09:37:19] :P [09:42:36] Amir1: that would be a very nice incentive! :D [09:42:45] can you imagine pki down for the weekend? :D [09:43:00] 😭 [11:07:53] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Wikimedia-Performance-recommendation: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12311682 (10Ladsgroup) 05Stalled→03Resolved [11:21:45] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Wikimedia-Performance-recommendation: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12311735 (10Ladsgroup) {F102545189} vs. same time a month ago: {F102545216} And it will get much better after everything... [12:04:54] elukey: happy to take a look shortly [12:47:41] sukhe: <3 [13:00:13] 10netops, 06Traffic, 10Cloud-VPS, 10Data-Services, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12311975 (10Gehel) [13:55:17] elukey: I will comment on the CRs if that's fine/better vs here [13:55:20] let me know if that's OK [13:55:42] sure! [14:38:43] sukhe: thanks for the reply! What do you think if we do it on Monday? [14:43:54] elukey: yep! perfectly fine [14:44:30] we can start around 13:00 UTC if that works for you [14:45:19] yeah sure! Is it a good time for you? Even later is fine, like 15 UTC [14:46:57] now that I think more about it, maybe we could take it more cautiously and just move k8s and production to specific host endpoint. Like pki1002.eqiad.wmnet instead of pki.discovery.wmnet [14:47:15] once done, messing up a little with pki just be ok, even if it breaks [14:47:29] then we rollout again pki.discovery.wmnet when ready [14:47:45] It would take a bit more but maybe a little safer [14:49:14] that's also a good idea yes [14:49:23] it reduces the risk of downtime during this move [14:49:29] need to verify some bits in puppet, of course the config is not trivial afaics [14:49:56] all right let's try to do it.. we don't do it on monday, but when the other configs are ready [14:49:58] thanks! [14:50:07] I think it should be OK (last famous words) but yeah, being cautious is probably better. so whatever you decide. [14:50:13] ok! let us know if we can help [14:50:21] I don't want to win a tshirt before leaving :D [14:50:33] elukey: even worse, budget cuts mean you get a sock (one :) [14:50:34] :P [14:52:35] ROFL [15:02:44] bblack slyngs: I restarted swift proxy on ms-fe1014 and now the cache control is showing up for thumb.wikimedia.org too (by setting Host header) before that, it only was set if the host was set to upload.wikimedia.org [15:03:08] https://wikitech.wikimedia.org/wiki/Service_restarts#Swift how to restart the proxy service: [15:03:14] depool && sleep 3 && systemctl restart swift-proxy && sleep 3 && pool [15:07:07] now hit-local is showing up in thumb requests too \o/ [15:07:39] that's a nice Friday discovery \m/ [15:09:31] I will hold off for restarting the proxy everywhere but will do it first thing Monday morning [15:26:00] we also have a cookbook, which does all of that: sre.swift.roll-restart-reboot-swift-ms-proxies [16:00:44] moritzm: That doesn't work on Friday afternoons :-P [16:05:53] Amir1: https://w.wiki/URHd we have hit-local [16:10:02] but you can also pass a pattern to the cookbook, if you run [16:10:03] sudo cookbook sre.swift.roll-restart-reboot-swift-ms-proxies -r "refresh thumbs" --query 'P{ms-fe1014*}' restart_daemons [16:10:23] it will only restart Swift a single server and handle all the pooling etc. [16:10:26] plus downtime