[08:51:30] moritzm: have you seen the email spam from debmonitor's systemd timer? [08:51:45] is it not reading the default config file? [08:52:23] see -operations [08:52:59] ack [08:53:40] to fixup the legacy sysusers I'm removing the debmonitor user, then installing 0.3.0 and eventually I run debmonitor to have debmonitor catch up with the fact that it now has 0.3.0 [08:53:56] and I guess during the 667 codfw hosts it raced with the timer a few times [08:54:06] makes sense [10:44:52] topranks: https://librenms.wikimedia.org/device/device=89/tab=port/port=16787/ [10:45:07] something is about to saturate our codfw-ulsfo link [10:51:20] "something" ? [10:51:50] topranks: see -traffic, the CP servers are pulling ton of traffic from codfw [10:51:58] but doesn't seem related to external traffic [10:52:36] CP servers in ulsfo ? [10:52:42] yep [10:52:55] Does this seem normal to you? [10:52:56] eg https://librenms.wikimedia.org/device/device=91/tab=port/port=8039/ [10:53:37] hmm yeah. [10:53:37] usually it matches external traffic [10:53:42] but not here [10:54:03] Yeah, so like that's all application stuff, cache misses etc? [10:55:25] it's weird as it's ulsfo lowest traffic time [10:55:35] yeah very early pacific time [10:56:06] so what might it be? misconfiguration? some edge case triggered by genuine use? some edge case triggered maliciously? [10:56:07] any host in particular? [10:56:15] let me know if there is anything you think I can look at [10:56:22] someone crawling commons? [10:56:31] volans: cache uploads [10:56:36] text hosts or upload hosts? [10:56:44] volans: if it was that we would see matching outbound traffic to transit/peering [10:56:48] might be just a crawler on commons [10:56:57] I pinged people in -traffic [10:57:09] volans: but crawler would mean external traffic, no? [10:57:19] unless we have some direct line not seen there yes [10:57:20] :D [10:57:26] I should join that I'm guessing.... #wikimedia-sre-traffic ? [10:57:30] so yep doens't make sense [10:57:36] topranks: s/-sre// [10:57:45] volans: thanks :) [10:58:02] naming is hard, and totally inconsistent ;) [10:58:19] renaming is hard I guess is the problem :) [11:09:16] lol [13:35:39] 10netops, 10SRE, 10Traffic, 10User-jbond: varnish filtering: should we automatically update public_cloud_nets - https://phabricator.wikimedia.org/T270391 (10cmooney) I fear we could be quite disappointed about "corporate workstations" being on IPv6 if we went to look ;) Either way I assume we want this li... [13:45:40] XioNoX: Could you look at this when you've a moment: [13:45:41] https://gerrit.wikimedia.org/r/c/operations/homer/public/+/697970 [14:19:14] topranks: +1! [14:34:16] Nice one :) [14:54:18] 10netops, 10Analytics, 10SRE: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 (10Ottomata) Ok, for the kafka term, we no longer need any logstash hosts. kafka logging cluster used be colocated on a few logstash hosts, but no longer, they are all on kafka-loggingXXXX. This [[ ht... [14:58:00] 10SRE-tools: debmonitor: 500 on upgraded+downgraded package - https://phabricator.wikimedia.org/T282529 (10Volans) 05Open→03Resolved The new debmonitor client has been deployed fleet wide, resolving. [15:01:40] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Peerings to doh3001 and doh3002 added on cr1-esams and cr2-esams now. Anycast range is being announced and from here in Ireland I'm hitting doh3... [15:52:28] 10netops, 10SRE, 10Traffic, 10Patch-For-Review: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) There was an issue with peering to doh3002 due to a problem that occurred with Netbox automation, triggered by the VM creation running twice I be... [21:40:18] jbond: FYI django-cas-ng has released a new version that includes your bugfix that should unblock the work to add SSO to Netbox. [21:41:05] I don't recall if the duplicated users issue was still present with this or not though. [21:43:56] volans: yes this includes the release https://github.com/django-cas-ng/django-cas-ng/pull/285 ;) [21:44:37] this release includes the fix even [21:44:56] ) [21:45:05] :)