[08:06:12] XioNoX, topranks: could either of you please rearm the keyholder on cumin1001 for homer? [08:06:26] mrotizm: let me have a look [08:06:31] moritzm: even [08:07:20] thx [08:08:27] ok think that should be it [08:08:55] thx [08:09:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=3b336fa4-f522-4b10-abdb-d6be83f6a04a) set by ayounsi@cumin2002 for 2:00:00 on 3 host(s) and th... [08:45:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e0d9eb2b-5520-4f80-912e-3627c94e9982) set by ayounsi@cumin2002 for 2:00:00 on 3 host(s) and th... [09:31:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ff1db65d-a6ee-4e20-ae07-837bbe264b2f) set by ayounsi@cumin2002 for 2:00:00 on 2 host(s) and th... [09:53:34] 10netops, 10Infrastructure-Foundations, 10SRE: Overlay VRF / VXLAN traffic failure between lsw1-f2-eqiad and lsw1-f3-eqiad - https://phabricator.wikimedia.org/T315038 (10cmooney) 05Open→03Resolved So after quite a bit of back-and-forth with Juniper and pulling logs etc. they say they can't see anything i... [10:05:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) cr2-esams and cr3-knams got upgraded as expected. cr3-esams failed as it requires a firmware upgrade, and only JTAC can provide us the firmware. We wi... [10:07:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10ayounsi) [10:50:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10cmooney) [10:52:40] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10cmooney) The firmware provided by Juniper seems to be accepted by cr3-esams: ` cmooney@re0.cr3-esams> show system firmware | match "^Part|version|i40" Part... [10:52:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade core routers to Junos 21+ - https://phabricator.wikimedia.org/T295690 (10cmooney) [11:53:05] moritzm, jbond: as I'm not following closely the Perc H750 issues, just wanted to check if the icinga raid handler related NRPE scripts have been (or needs to be) adapted [11:54:58] I had a look over them and there's a few smaller things which need updating, but for now the impact is esentially just support for the new controllers [11:55:24] the old "raid" fact is still around and I'll send a mail to ops@ (and finishing cleanups) before we remove it [12:03:48] yeah the problem shouldn't be the fact but the tool to use to check and report to the task the raid status [12:03:59] feel free to ping me if you need a hand with those [15:00:58] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) I was having the issue below upgrading mr1 to version 21 ` Validating against /config/rescue.conf.gz /config/rescue.conf.gz:61:(21) syntax error at 'rfc-co... [15:01:28] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) [15:01:52] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) ` papaul@mr1-codfw> show version Hostname: mr1-codfw Model: srx300 Junos: 21.2R3-S2.9 JUNOS Software Release [21.2R3-S2.9] ` [15:48:48] 10netbox, 10Infrastructure-Foundations: Netbox: use Custom Model Validation - https://phabricator.wikimedia.org/T310590 (10ayounsi) a:03Volans [15:49:17] 10netbox, 10Infrastructure-Foundations: Netbox: use Custom Model Validation - https://phabricator.wikimedia.org/T310590 (10ayounsi) a:05Volans→03None [17:40:17] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10Andrew) ` root@cloudcontrol2005-dev:~# dig +noall +answer SOA 16-29.57.15.185.in-addr.arpa. 16-29.57.15.185.in-addr.arpa. 120 IN SOA ns0.openstack.codfw1dev.wikime... [22:12:36] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE, 10User-herron, 10User-jbond: Prevent puppet catalog compiler workers from running out of disk space - https://phabricator.wikimedia.org/T222075 (10Dzahn) This happened again the other day and made me mail the SRE list. Then I added docs how to clea...