[07:58:19] Krinkle: \o/ [16:57:21] sobanski: marostegui: https://phabricator.wikimedia.org/T282761#7170212 [16:57:24] summary of status quo [16:57:29] I'll submit a patch now to half the sleep [17:45:18] effie: based on very rudimentary testing against beta cluster, I did not see the expected effect yet. [17:45:46] I ran `watch -n0 'curl -H "X-Wikimedia-Debug: 1" -I https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page'` in a bunch of shell tabs concurrently, and it seems with every tab I added, the memc gets/s increased further [17:46:17] afaik Logstash is still broken there so hard to investigate currently, maybe there's some non-wan keys in there somewhere [17:46:53] I guess we could tcpdump or something direcltly on the main and the onhost memc host and see [17:56:31] I'm about to image the new thumbor100[56] hosts but old hosts are stretch [17:56:44] anyone know if i can bump new hosts to default? [17:57:24] wkandek: ^ ? [17:57:42] basically anytime a task asks for a really old distro, i feel the need to double check before imaging [17:58:02] (not trying to block, if I get zero response I'll still stage everything ready for the script firing) [17:58:37] every existing thumbor system is stretch [17:58:57] (but new hosts are also often used as the opportunity to try out new distros since it can expand cluster in short term) [17:59:24] _joe_: i saw you also commented on the ordering task for thumbor, but it may have simply been due to it being idetnical task for mw and thumbor orders =] [18:01:06] robh: Krinkle might be able to provide feedback. I fear that thumbor may be stuck on old distros pending rsvg things but would be delighted to find out that is an incorrect assumption. [18:01:23] bd808: thx!! i appreciate the feedback [18:01:44] yeah, when i saw the entire fleet is on it, i feared it was some package restriction [18:01:57] or versioning of something, seems like that is the case, hopefully its cleared indeed. [18:02:29] robh: I think thumbor has to be stretch still, see https://phabricator.wikimedia.org/T216815 [18:02:50] yeah, but often a new host is used to do this conversion right? [18:02:53] but indeed, thx for link [18:02:59] i'll tie it into the racking task references [18:03:01] I don't know anything about thumbor packaging other than that there are thumbor packages that are (mostly) maintained by us, and that thumbor connects with a lot of other packages (rsvg, etc.) that are extremely sensitive to specific versions, fonts, configurations etc for Commons output. Generally something we've historically upgraded very slowly and carefully wht a lot of roundtrip testing [18:03:16] cool [18:03:24] i'll just put T216815 in as a parent to the racking task [18:03:25] T216815: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 [18:03:40] these are essentially what used to be our mw imagescalers. [18:03:50] or maybe just a ref in body of task, yeah. [18:04:04] but yeah I hope we (as an org) manage to upgrade that to Buster soon-ish [18:04:41] thx for feedback folks, i'll stage and image as stretch for now [18:04:52] if it turns out one can be used to upgrade, reimage when its not in service is easy. [18:08:30] Krinkle: if you need beta logstash you can delete the massive /var/log/logstash/logstash-json* log files on deployment-logstash[4-6] and kill+restart logstash.service, it will work until the logstash logs fill up the disk again [18:34:21] robh: thumbor capex hosts need to be stretch still [18:34:42] these will only be switched to buster (or bullseye) when it moves to k8s [18:44:09] moritzm: thx! just imaged them with stretch heh [18:50:29] robh: sorry, just saw that - keeping on stretch is the right call for now. [18:55:20] np, ohters linked to docs that pretty much said that heh [18:55:32] and they're imaged and standing by for your teams use [19:54:17] sobanski: when can we apply the sleep change? [19:55:22] Tomorrow during the day time. Same time we planned for today? [19:57:17] Okay, I'll try to make it.