[00:08:00] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster completed: - dns5004 (**PASS**)... [11:20:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10MoritzMuehlenhoff) ganeti5001 has been decommissioned and can be unracked. [11:28:06] 10Traffic, 10SRE, 10serviceops, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10Joe) >>! In T324200#8436279, @daniel wrote: > Note that we only need active purging if/when we emit cache control headers that tell th... [11:52:44] ok im going to log of now, puppet is running if you cant get on 15 feel free to ping me~.~. [13:58:13] 10Traffic, 10Infrastructure-Foundations: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10BBlack) [14:15:37] 10Traffic: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn' - https://phabricator.wikimedia.org/T324336 (10BBlack) [14:16:17] 10Traffic: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn' - https://phabricator.wikimedia.org/T324336 (10BBlack) [14:18:35] 10Traffic, 10Infrastructure-Foundations: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) To answer your points: 1) yes, it was chosen on purpose because is intrinsically RO and can't be made RW even if misconfigured, given that is exposed to the internet. That said... [14:19:09] 10Traffic: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn' - https://phabricator.wikimedia.org/T324336 (10BBlack) [14:22:55] 10Traffic, 10Infrastructure-Foundations: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10BBlack) Another thing I failed to mention above: for cases like this (automated git clones just for functional data) we could also potentially speed up the initial fetch by doing shall... [14:24:36] 10Traffic, 10Infrastructure-Foundations: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) That's surely an option, the external one is mainly needed for CI and those of us that have a local checkout to do some greps, nothing critical. [14:32:34] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10BBlack) Comparison point: operations/dns is quite a bit different: total byte size is ~1/4 the size (~6MB vs the ~22MB size of netbox-exports), but has ~... [14:38:36] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster [14:49:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1001 for host lvs5004.eqsin.wmnet with OS buster [14:53:54] 10Traffic, 10SRE, 10Patch-For-Review: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn' - https://phabricator.wikimedia.org/T324336 (10BBlack) [14:54:06] 10Traffic, 10SRE, 10Patch-For-Review: Replace edge cache conftool entries 'varnish-fe' and 'ats-tls' with singular 'cdn' - https://phabricator.wikimedia.org/T324336 (10BBlack) [15:03:18] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) Git a simple `git gc` we go from 22 MB to 4MB for a clone and 2.4MB for a bare clone. I'll run `git gc` everywhere right now. [15:03:35] bblack, sukhe: I'm about to run git gc on all the netbox dns checkouts, see ^^ [15:04:16] volans: ok, just as an FYI and I am sure you know but dns5004 reimaging is in in progress. not at the first Puppet run but soon [15:05:48] sukhe: great, so we can test it immediately :) [15:06:14] yeah we did set out the timeout to 10 minutes but I think we will still know! [15:06:37] I've just run it on the netbox exports (from where you checkout) [15:07:20] if that works and is enough we could add a timer to do it every now and then [15:07:41] I'll clean manually the other checkouts later, waiting to see if it makes a difference first [15:09:04] sukhe: if you could let me know how much it takes on dns5004 that would be great! [15:09:59] as triggered by puppet, on my home on dns5002 it takes 8s [15:10:42] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) On my home on `dns5002`, not triggered by Puppet: ` dns5002 0 15:09:41 ~ $ time git clone 'https://netbox-exports.wikimedia.org/dns.git' dns-te... [15:10:52] volans: ok! yeah 8seconds is definitely an improvement [15:11:23] I re-ran my test on dns5002 just to confirm with same methods as before: [15:11:27] real0m8.330s [15:11:36] so yeah, that was easy! :) [15:43:39] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns5004.wikimedia.org with OS buster completed: - dns5004 (**PASS**)... [15:48:00] 10Traffic, 10Infrastructure-Foundations, 10SRE: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10BBlack) 05Open→03Resolved a:03Volans Confirmed the same. That was a simple and elegant fix, so I doubt there's any reason to pursue more-complex options! Thank you! [15:55:14] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [16:28:29] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1001 for host lvs5004.eqsin.wmnet with OS buster completed: - lvs5004 (**WARN**)... [16:28:41] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1001 for host lvs5004.eqsin.wmnet with OS buster executed with errors: - lvs5004 (... [18:06:46] 10Traffic, 10Infrastructure-Foundations, 10SRE: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) And run `sudo cookbook -d sre.dns.netbox --force cbf1f50e2897f654f6f4f2ce639e7b9cf85e54cd -t T324334 "Test run after gc"` to ensure it all still works fine [18:45:52] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [18:52:19] 10Traffic, 10Infrastructure-Foundations, 10SRE: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10ssingh) While running the decommissioning cookbook for lvs5001, I ran into this: ` (13) authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4003-4004,5002,6001... [18:53:21] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `lvs5001.eqsin.wmnet` - lvs5001.eqsin.wmnet... [18:58:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [19:43:27] 10Traffic, 10Infrastructure-Foundations, 10SRE: netbox-exports git cloning perf issues - https://phabricator.wikimedia.org/T324334 (10Volans) Sorry for the trouble, that was me indeed, I've fixed the permissions and run the `sre.dns.netbox` cookbook successfully: ` sudo cookbook sre.dns.netbox --force 0b5a2...