[03:45:44] <jinxer-wm>	 FIRING: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[03:55:44] <jinxer-wm>	 RESOLVED: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[14:15:44] <jinxer-wm>	 FIRING: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[14:25:44] <jinxer-wm>	 RESOLVED: [2x] NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/extras/scripts/12/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[15:44:27] <arnaudb>	 how do we feel about running third party OS on our hardware? I've been asked by Dell to run this: https://www.dell.com/support/kbdoc/en-us/000198650/running-stress-tests-in-sli-support-live-image?lwp=rt (T374215)
[15:48:53] <_joe_>	 if the image is provided by our OEM provider, it doesn't increase the attack surface IMHO
[15:49:22] <arnaudb>	 ack, thanks!
[15:49:24] <_joe_>	 they could just bake whatever malicious backdoor in the firmware
[15:49:49] <cdanis>	 +1
[15:49:50] <_joe_>	 arnaudb: that's just my opinion, maybe moritzm or vgutierrez have other takes :)
[15:50:21] <arnaudb>	 keeping my eye out for a consensus :)
[15:50:42] <vgutierrez>	 do we have sensitive material on the impacted servers?
[15:50:59] <vgutierrez>	 I'd wipe TLS keys and that kind of stuff before running that IMHO
[15:51:08] <arnaudb>	 hm
[15:51:14] <cdanis>	 I imagine it's a DB replica heh
[15:51:25] <moritzm>	 or we simply reimage to the insetup role before running the test?
[15:51:35] <vgutierrez>	 yeah.. that could also work :)
[15:52:32] <arnaudb>	 ah good idea, I'll try and do that!
[15:52:41] <arnaudb>	 and yes, indeed, it is a db replica :D
[15:54:15] <inflatador>	 kinda sucks that they are making us run a bunch of HW tests before they give us a new chassis, but maybe I have unrealistic expectations
[15:54:51] <arnaudb>	 I guess another provider would have other caveats :D but yeah this is a tedious process in that  specific case
[15:56:44] <inflatador>	 We had an arrangement like that with Dell at a previous job, but I'm sure it was mega expensive
[15:57:18] <inflatador>	 and we could pass that expense directly to our customers...not really an option here ;)
[16:01:56] <arnaudb>	 we could try a new banner message "help us pay dell more so we are faster to fix things"
[16:02:56] <arnaudb>	 I've prepared the puppet patch here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1093372 if someone has time for a sanity check
[16:03:57] <inflatador>	 LOL. "Imagine a world where all service tags have access to free replacements"
[16:07:55] <moritzm>	 arnaudb: looks good, +1D
[19:56:17] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10341718 (10Reedy)
[20:50:32] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10341932 (10akosiaris) Adding a couple of people that are better situated than yours truly to look into this, as well as a bit more information.  Looking into vrts1003 logs I see...