[06:33:06] 10netbox, 10DC-Ops: Various netbox alerts running for days - https://phabricator.wikimedia.org/T283483 (10ayounsi) [07:48:24] 10netops, 10SRE, 10Traffic, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10jbond) [08:44:29] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) [08:50:38] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) [08:54:15] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) [08:56:12] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) [09:08:17] 10netops, 10SRE, 10Traffic, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10fgiunchedi) In case it is useful: a lighter weight (but one that we have to maintain ourselves) solution for prometheus would be to use node-exporter's textfile collector... [09:20:05] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) [09:20:45] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10ayounsi) a:03ayounsi [09:29:43] 10netops, 10SRE, 10Traffic, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10cmooney) Thanks @ayounsi, That exaring project looks to be a fairly sensible approach alright, albeit fairly new. Might be worth testing out. I note our NS seem to use... [09:32:37] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10cmooney) [09:43:14] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10cmooney) [09:55:35] 10netops, 10SRE: Increase Google IX sessions prefix-limit - https://phabricator.wikimedia.org/T284447 (10cmooney) 05Open→03Resolved [12:31:40] 10netops, 10Traffic, 10VPS-project-Codesearch, 10serviceops: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10ema) [12:32:00] 10netops, 10Traffic, 10VPS-project-Codesearch, 10serviceops: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10ema) p:05Triage→03Low [13:36:44] 10netops, 10SRE, 10Traffic, 10VPS-project-Codesearch, 10serviceops: Consider using BindsTo instead of Requires to declare dependencies between systemd unit - https://phabricator.wikimedia.org/T284555 (10BBlack) When we looked into this for the Bird-based anycast stuff, we found that the combination you w... [16:01:17] XioNoX, topranks: do any of you plan to attend the network to code webinar they are organizing on network automation? [16:01:51] be aware that it might be a selling webinar though... [16:01:59] I wasn't aware of it volans... sounds interesting though [16:02:04] NTC always flogging something. [16:02:07] :) [16:02:14] let me forward the invite to you [16:04:34] nice one :) [16:05:25] "The Art of the Possible" [16:05:37] starts well [16:06:04] with Arista [16:09:22] https://netflixtechblog.com/how-netflix-uses-ebpf-flow-logs-at-scale-for-network-insight-e3ea997dca96 netflow with ebpf [16:21:04] XioNox: interesting post there [16:21:13] Less than 1% CPU you gotta love that :) [16:34:15] XioNoX: nice post, thanks for linking [16:35:39] wish they had released the exporter [16:45:06] 10netops, 10Data-Persistence-Backup, 10SRE, 10bacula: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) I've been able to find the source of the dropped traffic between eqiad and codfw. T... [16:53:28] 10netops, 10Data-Persistence-Backup, 10SRE, 10bacula: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) I also researched / played with TCP tunings. I don't believe the current CUBIC algo... [18:31:17] 10CAS-SSO, 10SRE, 10Patch-For-Review, 10User-jbond: Apereo CAS expose CASCookieSameSite via profile::idp::client::http - https://phabricator.wikimedia.org/T264605 (10jbond) 05Open→03Resolved a:05MoritzMuehlenhoff→03jbond This should be complete now please re-open if you still see issues, thanks [18:40:13] 10netops: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [18:56:46] 10netops: Create an alert for output discards on network devices - https://phabricator.wikimedia.org/T284593 (10cmooney) [19:02:47] 10netops: Create an alert for output discards on network devices - https://phabricator.wikimedia.org/T284593 (10cmooney) [22:03:08] 10netbox: Request to add Custom fields in Netbox - https://phabricator.wikimedia.org/T284614 (10Papaul)