[07:19:59] I'll have a look at the patch in a bit, was off yesterday [08:30:43] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10347326 (10LSobanski) p:05Triage→03High [09:33:59] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, 10Znuny: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10347389 (10Krd) I have the impression that it has changed something, but I still see a lot of RCVD_IN_VALIDITY_SAFE hits, example: https://ticket.wiki... [09:57:30] 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10347431 (10cmooney) This port bounced again overnight: ` cmooney@cloudsw1-d5-eqiad> show log messages.... [09:58:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10347436 (10akosiaris) Cool thanks, I 'll take over this one. [16:03:01] elukey: I think keeping two config-J supermicro boxes arournd for testing is probably enough [16:03:16] I'm a little pessimistic that supermicro will turn around with a fix [16:04:05] they seem eager on the frontend, but then I worry it gets sucked into some endless backlog, with one lonely engineer working the queue [16:04:48] jhathaway: :( [16:05:03] yes 2 should be good, one for each dc to keep the balance [16:06:12] sounds good, hopefully that is just some early morning pessimism, perhaps I need more black think juice :D [16:10:09] I share the same pessimism but it may be Friday's evening :D [16:18:28] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10349021 (10JMeybohm) This did bite us again and we had to {T380473} in a hurry. Quick fix to free up IPAM blocks without deco... [16:26:51] 07Puppet, 07IPv6, 13Patch-For-Review: Keepalived Puppet module: Support IPv6 - https://phabricator.wikimedia.org/T380057#10349037 (10taavi) 05Open→03Resolved [16:29:34] 10netops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 06serviceops: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10349049 (10JMeybohm) >>! In T375845#10246786, @akosiaris wrote: > Now, for the actual change. The biggest issue is coordinati... [17:23:23] 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349255 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6b283bec-74b8-4f8c-9a46-f9... [17:33:54] 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349290 (10VRiley-WMF) Replaced the transciever in cloudsw1-e4-eqiad et-0/0/54. Will test to see if th... [18:00:01] 10netops, 10Cloud-Services, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54 - https://phabricator.wikimedia.org/T380503#10349375 (10cmooney) Thanks @VRiley-WMF! Seems ok so far but we can make a call Monday based on if we... [19:09:54] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, and 2 others: spamassassin broken for VRTS - https://phabricator.wikimedia.org/T380396#10349592 (10eoghan) I see what's going on. There are three checks provided by VALIDITY configured by default. `RCVD_IN_VALIDITY_CERTIFIED`, `RCVD_IN...