[09:00:46] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Write a cookbook to set a k8s cluster in maintenance mode - https://phabricator.wikimedia.org/T277677 (10Jelto) [09:01:15] 10SRE-tools, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Create a cookbook for depooling one or all services from one kubernetes cluster - https://phabricator.wikimedia.org/T260663 (10Jelto) [10:36:40] It seems to me that https://netbox.wikimedia.org/ipam/ip-addresses/7406/ is wrong and should be deleted, it's marked as VIP and has a /128, thoughts? [10:36:44] see the related IPs with https://netbox.wikimedia.org/search/?q=sretest1001.eqiad.wmnet [10:37:05] the equivalent /64 one exists already and is not a VIP and is assigned to the host [10:37:42] +1 [10:41:54] +1 on yesterday's discussion about collating entries for I/F into a single section in the meeting doc: in the end this is more flexible since the team covers more than just network, tooling and security, e.g. the update on Ganeti which ended up under Miscellaneous since there was no better place to add it [10:42:34] XioNoX: this one too seems leftover, the other 2 VRRP exists already [10:42:37] https://netbox.wikimedia.org/ipam/ip-addresses/2403/ [10:43:07] https://netbox.wikimedia.org/ipam/ip-addresses/2408/ too [10:43:25] https://netbox.wikimedia.org/ipam/ip-addresses/2413/ ditto :D [10:44:05] yeah looks like it's a netbox bug [10:45:06] more IDs coming: 2418, 2423, 2480, 2495, 2500, [10:45:26] should be all, all the other "duplicates" have 2 entries and same netmask so should be all the VRRPs [10:46:01] that should be a netbox report :) [10:46:18] but maybe the new Netbox vesion fixed that bug [10:46:54] XioNoX: can I delete all the reserved ones? [10:47:11] volans: I'm doing it now [10:47:26] ack, thx [11:33:58] the Lumen circuit between eqiad and esams is down again, fibercut near Eqiad.. [11:35:51] ack [11:54:58] 864 count fiber, let's hope we're on the first strand they splice :) [13:43:54] 10SRE-tools, 10Infrastructure-Foundations, 10User-jbond: Spicerack: improve Icinga module to support mgmt interfaces - https://phabricator.wikimedia.org/T226470 (10Volans) 05Open→03Resolved a:03Volans This has been actually fixed in [[ https://doc.wikimedia.org/spicerack/master/release.html#v0-0-52-202... [13:59:35] 10netops, 10Infrastructure-Foundations, 10SRE: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) Ok so this has been addressed for CR routers. You can view the NTP status as follows: ` cmooney@cr2-eqord> show ntp associations remote refid st t... [14:00:49] 10netops, 10Infrastructure-Foundations, 10SRE: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) 05Open→03Resolved Scrap that it does seem to be working, perhaps it only failed to query against itself after the initial change. ` cmooney@mr1-drmrs> show ntp assoc... [14:08:07] XioNoX, topranks: FYI in case you didn't get it elsewhere already: https://lists.fosdem.org/pipermail/fosdem/2021q4/003296.html [14:08:17] FOSDEM CfP network devroom [14:08:58] no had not seen it - thanks for passing it on! [14:09:07] indeed, thanks! [14:12:27] there has been so many conferences recently, dunno when I'll have time to look at them, RIPE, NANOG, SREcon (US + Europe), LISA, [14:13:43] it's super boring for me to attend/watch remote conferences, I find hard to find the talk that are worthed/interesting and then have the time to watch them (even at 1.x speed) [14:14:18] and often the title is not enough to figure out if it will be worth it or not [14:19:43] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10BTullis) Tagging #data-engineering because we will likely be managing the Gobblin and/or Druid ingestion parts of this pipeline. Should w... [14:43:42] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Decommission puppetboard[12]001 - https://phabricator.wikimedia.org/T296744 (10jbond) [14:43:57] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Decommission puppetboard[12]001 - https://phabricator.wikimedia.org/T296744 (10jbond) p:05Triage→03Medium [14:44:38] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) [14:58:31] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) [14:58:35] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: Decommission puppetboard[12]001 - https://phabricator.wikimedia.org/T296744 (10jbond) 05Open→03In progress [15:04:05] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: Decommission puppetboard[12]001 - https://phabricator.wikimedia.org/T296744 (10jbond) [18:03:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) I did not forget this task, but have been busy the last... [18:04:09] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) @BTullis thanks! Real-time, would be a nice plus, but a hard requirement (unlike netflow). @cmooney [[ https://gerrit.wikimedia... [19:28:23] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) [19:29:29] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) So both of the old mx80s are attached to mgmt, power, and scs. I've not bothered to document these connections in netbox, as they are un... [19:31:32] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) a:05RobH→03ayounsi I'm not sure if this should assign to @ayounsi or @cmooney, either can handle this! This is ready for software up...