[00:03:10] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10wiki_willy) Hi @Volans - just a heads up, it looks like some of those 5yr servers on the Accounting Spreadsheet are starting to pop up on the Netbox Error Report as accounti... [07:33:46] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10ayounsi) Following this [[ https://help.expandi.io/en/articles/5405660-making-a-webhook-with-google-sheets | doc ]] I was able to add data to a spreadsheet using a generic p... [07:53:06] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10Volans) @ayounsi we already have all that setup... [08:15:28] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10ayounsi) Thanks for the details! I'm always wary of adding configuration knobs and logic that could make troubleshooting more com... [08:44:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10cmooney) >>! In T332781#8744511, @ayounsi wrote: > I'm always wary of adding configuration knobs and logic that could make trouble... [08:53:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10cmooney) 05Open→03Resolved Patch merged, working as expected. Previous trace from bast1003 to a server in rack E1: ` cmooney@... [09:32:01] 10Puppet, 10netbox, 10Infrastructure-Foundations, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10ayounsi) Thanks! I think having the visualization of the data is a good start. Next step would to see how to do drop in replacement of some of t... [10:19:41] 10Puppet, 10netbox, 10Infrastructure-Foundations, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10cmooney) Late to the party here. >>! In T329669#8618111, @ayounsi wrote: > The other point related to above is that we don't have a strict/clea... [11:35:08] (DiskSpace) firing: Disk space urldownloader1002:9100:/ 5.957% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=urldownloader1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:42:22] slyngs: i think that your change has exploded the log file on urldownloader [11:42:39] Oh noes [11:42:43] we shuld probably revert for now and increase the disk capacity on those boxes [11:43:01] it only has 9G [11:43:09] Yep [11:43:51] I can just do that with revert in Gerrit right? [11:43:58] yes exactly [11:44:09] can look at the disk space next week [11:44:12] ill create a task [11:45:00] Thanks [11:45:01] https://gerrit.wikimedia.org/r/c/operations/puppet/+/904691 [11:47:09] +1 also added T333676 [11:47:10] T333676: Increase disk space for url_downloader servers - https://phabricator.wikimedia.org/T333676 [11:50:05] slyngs: let me know when its merged and ill clean up the logs [11:50:08] (DiskSpace) resolved: Disk space urldownloader1002:9100:/ 5.834% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=urldownloader1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [11:50:30] ^^ this may trigger again but will clean up after the merge [11:51:09] Merge, I'll just trigger the Puppet run [11:54:26] ack puppet ran every where along with logrotate --force /etc/logrotate.d/squid [11:54:31] shuld be good now [11:54:39] cheers [11:54:47] Thanks for the heads up [11:55:22] It doesn't look like the logs take up to much space compressed, could we just remove the "delaycompress" from logrotate? [11:56:10] slyngs: not sure when you merge the original change but the uncompressed log for today was allready at ~3G [11:56:25] i ran logroate a couple of times [11:56:38] That was Wednesday morning, and the .2.gz said 52MB I think [11:56:42] Oh okay [11:57:33] We need more space then :-) [11:58:56] slyngs: yes :) we could also change logrotate to hourly however i think its probably still worth adding some more space to deal with any bursts [11:59:25] I'll take a look at we can figure out the details next week [11:59:32] sgtm thanks [11:59:34] at / and [12:26:21] 10Puppet, 10netbox, 10Infrastructure-Foundations, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10jbond) > I take it you are basing this off the defined RFC1918 prefixes in the latest revision? yes w just uses pythons `ipaddress.ip_address(ad... [13:16:15] 10Puppet, 10netbox, 10Infrastructure-Foundations, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10cmooney) >>! In T329669#8745241, @jbond wrote: > yes i did wonder if this could be wrong for ipv6 I guess it comes down to whether we want the... [15:26:21] 10SRE-tools, 10netbox, 10DNS, 10Infrastructure-Foundations, 10Traffic: Enforce Netbox domain names without period termination - https://phabricator.wikimedia.org/T306809 (10BCornwall) [15:27:29] 10SRE-tools, 10netbox, 10DNS, 10Infrastructure-Foundations: Enforce Netbox domain names without period termination - https://phabricator.wikimedia.org/T306809 (10BCornwall) Updated the task description to accurately reflect the work that needs doing. I'm also going to remove the Traffic tag since it seems... [16:55:50] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10wiki_willy) Thanks, let us know if there's anything we can do in the meantime. Here's a list of the assets that are reporting as Netbox errors for accounting mismatch, whic... [23:01:17] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10Volans) >>! In T320955#8744401, @Volans wrote: > @ayounsi we already have all that setup... To add a bit more context, I was out on my mobile and just wanted to post a quic... [23:19:36] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10wiki_willy) Sounds good, thanks! > @wiki_willy I'll try to look at it next week, it should be easy to read from the spreadsheet you showed me and exclude those for now. The...