[08:24:56] FYI pybal package is in a weird state on pybal-test2003 that causes a change on every puppet run ( https://puppetboard.wikimedia.org/node/pybal-test2003.codfw.wmnet ) [08:25:58] I'll try to give a look, thanks volans ! [08:26:12] thx! :) [08:51:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) In terms of the LVS connections from rows C and D, when we move from old switches to new ones we need to land those on the Spines rather t... [10:30:07] ahoyhoy - I have another gateway routing change i'd like to make if it would suit to do this morning https://gerrit.wikimedia.org/r/c/operations/puppet/+/954890 [10:34:49] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10MatthewVernon) Thanks for this; I agree that we should probably (virtually) sit down and talk about this; I wanted to try and make sure we had most of the o... [11:36:17] hnowlan: both valentin and I are out today; I will be back in my afternoon (so evening for you) [11:36:33] happy to do it then or tomorrow :) [11:40:50] sukhe: ah cool, if you're out then I'm happy to do tomorrow :) [11:40:57] yep, bank holiday here [11:41:03] ping me tomorrow EU morning :) [11:46:28] hnowlan: BTW dunno how busy is that particular endpoint but the new version doesn't minify the JSON output [11:47:35] so for the same output the new endpoint returns a result that's ~66% bigger than the old one [11:47:57] https://www.irccloud.com/pastebin/kcd2nZHx/ [11:48:59] * vgutierrez back to his bank holiday [11:51:00] vgutierrez: sgtm! it's very quiet as an endpoint - I've asked the team if they'd be open to minifying in general, as I think that'd be pretty critical for the editor-analytics service, which is fairly heavily used. [11:54:01] https://grafana.wikimedia.org/d/2urzFwgIk/hnowlan-aqs-usage?orgId=1&viewPanel=2 see the editors endpoint here - and this is a subset of that endpoint [11:54:57] that's the non-cached hitrate, right? [11:55:08] aka you're measuring that behind the CDN [11:55:23] yeah [11:58:02] vgutierrez: your going back to your bank holiday is failing hard... go away ;) [11:58:21] hnowlan: turnilo agrees with you: https://w.wiki/7SEk [11:58:43] thanks volans! [11:58:51] 1.7MB in the last week [11:59:12] up to 2.8MB with the new endpoint [11:59:26] * vgutierrez goes away :_( [14:58:13] (VarnishHighThreadCount) firing: (2) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:03:13] (VarnishHighThreadCount) firing: (23) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:08:13] (VarnishHighThreadCount) firing: (25) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:13:13] (VarnishHighThreadCount) firing: (27) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:18:13] (VarnishHighThreadCount) firing: (28) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:23:13] (VarnishHighThreadCount) firing: (28) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:25:44] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [15:28:13] (VarnishHighThreadCount) firing: (26) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:33:13] (VarnishHighThreadCount) firing: (24) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:38:13] (VarnishHighThreadCount) firing: (21) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:43:13] (VarnishHighThreadCount) firing: (35) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:03:13] (VarnishHighThreadCount) resolved: (15) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount