[07:42:55] 10SRE-tools, 10Icinga, 10Infrastructure-Foundations, 10SRE, 10observability: Icinga paged for a host that should have been downtimed - https://phabricator.wikimedia.org/T309447 (10fgiunchedi) >>! In T309447#7966840, @Volans wrote: >>>! In T309447#7966236, @fgiunchedi wrote: >> Off the top of my head I ca... [07:44:27] 10SRE-tools, 10Icinga, 10Infrastructure-Foundations, 10SRE, 10observability: Icinga paged for a host that should have been downtimed - https://phabricator.wikimedia.org/T309447 (10Marostegui) >>! In T309447#7969207, @fgiunchedi wrote: >>>! In T309447#7966840, @Volans wrote: >>>>! In T309447#7966236, @fgi... [07:44:46] 10SRE-tools, 10Icinga, 10Infrastructure-Foundations, 10SRE, 10observability: Icinga paged for a host that should have been downtimed - https://phabricator.wikimedia.org/T309447 (10Volans) >>! In T309447#7969207, @fgiunchedi wrote: > Since this is hopefully rare, personally I think we should focus on movi... [08:04:23] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade Fastnetmon to 1.2.1 - https://phabricator.wikimedia.org/T271228 (10MoritzMuehlenhoff) This was the debconf diff for the puppetised fastnetmon.conf as presented by dpkg. We should check whether some new options should be covered in our puppetised config f... [08:34:58] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade Fastnetmon to 1.2.1 - https://phabricator.wikimedia.org/T271228 (10ayounsi) Great, there is nothing of immediate interest in the diff. IPv6 will probably be the next step here in a different task. [08:47:48] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade Fastnetmon to 1.2.1 - https://phabricator.wikimedia.org/T271228 (10ayounsi) left are eqiad/esams/eqsin. I'll take care of them later today or tomorrow. [08:51:18] 10SRE-tools, 10Icinga, 10Infrastructure-Foundations, 10SRE, 10observability: Icinga paged for a host that should have been downtimed - https://phabricator.wikimedia.org/T309447 (10fgiunchedi) >>! In T309447#7969225, @Volans wrote: >>>! In T309447#7969207, @fgiunchedi wrote: >> Since this is hopefully rar... [10:17:04] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade Netbox to 3.2 - https://phabricator.wikimedia.org/T296452 (10ayounsi) Had a meeting with John and Riccardo. Next steps are: # Create netboxdb1002/2002 on Bullseye (fully independent from the current production Netbox infra) # Setup Netbox 2.... [10:42:44] moritzm et al: I'm getting bounces (that most of the times goes into spam, but not always) for emails addressed to razzi and mukunda [10:43:15] the former is from an analytics-alerts alias, the latter from icinga [10:43:28] (rabuissa@ & mmodell@) [10:44:02] I can make an attempt at fixing both, but also wondering if these were skipped from offboarding, whether there are more [10:44:36] so maybe someone more clueful than me should have a look? :) [10:54:07] I'll have a look in a bit, maybe the analytics-alert alias was missed [10:59:08] for Mukunda's Phabricator alerts I need to check first whether to retire these or whether he even wants to those to a different non wmf email address [11:05:12] 10netops, 10Infrastructure-Foundations, 10SRE: DHCPd: update config to log more info - https://phabricator.wikimedia.org/T309524 (10cmooney) I agree @jbond it would be useful to have more granular detail. When we don't have a "match" on the dhcp snippet then we end up with a log like this: ` DHCPDISCOVER fr... [11:06:46] ack, thanks :) [11:56:29] 10netops, 10Infrastructure-Foundations, 10SRE: DHCPd: update config to log more info - https://phabricator.wikimedia.org/T309524 (10jbond) Thanks for looking at this @Volans @cmooney > Because that's a valid hostname in our DNS it would have just used that IP. So not sure how to "prevent" this. Doh! > It... [12:20:31] 10netops, 10Infrastructure-Foundations, 10SRE: Cannot verify NTP status asw1-b12-drmrs - https://phabricator.wikimedia.org/T305840 (10cmooney) 05Open→03Resolved a:03cmooney After a bit of back-and-forth with Juniper they eventually suggests just killing the ntpd process from a root shell. Which has do... [14:53:01] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade Netbox to 3.2 - https://phabricator.wikimedia.org/T296452 (10jcrespo) backups monitoring is complaining that netboxdb1002 has no backups: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=backup1001&service=Backup+freshness... [16:26:20] XioNoX: topranks: volans: FYI the netbox[12]002 serveres are no avliable via the cache using the discovery address. just need to add an appropriate entry to you hosts file e.g. [16:26:24] # 185.15.58.224 text-lb.drmrs.wikimedia.org. [16:26:26] 185.15.58.224 netbox.wikimedia.org [16:26:48] icinag is still complaining about some of the reports, ill check them out tomorrow [16:28:00] jbond: great work thanks! I’ll take a look now [16:31:31] jbond: ack thx [16:32:47] np [16:41:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, 10cloud-services-team (Kanban): Replace labstore100[67] with clouddumps100[12] - https://phabricator.wikimedia.org/T309346 (10Cmjohnson) This task does not require DC-OPs tag, once you have moved the data, please decommission labstores and crea... [16:53:18] awesome! [17:47:01] 10netops, 10Infrastructure-Foundations, 10SRE: codfw: Provision a server script can not run without a cable ID" - https://phabricator.wikimedia.org/T308768 (10Papaul) 05Open→03Resolved I tested this on backup2009 all is working with no issues. Thanks [21:15:58] 10netbox, 10Infrastructure-Foundations: netbox cannot import name 'cas_configuration' - https://phabricator.wikimedia.org/T309610 (10Peachey88)