[07:16:23] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) From JTAC: > This message “Read-only file system” suggest file system issues. I found one case with same behavior and the upgrade had to do it with... [07:17:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) 05Open→03Resolved All good, thanks a lot! [10:39:28] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) fpc0 went back up fine, but fpc1 not so much... It's not fully booting and stuck at a busybox like shell. Root password works so that means the con... [11:23:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) We tried to boot on the Recovery Junos (both 14 and 20) but the same error happened. Next step is onsite "format install" https://supportportal.ju... [15:41:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) > Next step is onsite "format install" https://supportportal.juniper.net/s/article/EX-QFX-Procedure-to-format-install-QFX5K-device-using-a-USB?lang... [16:07:00] topranks: XioNoX: you can ignore the hijack alerts [16:08:47] jbond: thanks (sounds scary all the same!) [16:08:49] cool :) [16:09:38] n/goXi [16:53:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqsin, 10Wikimedia-Incident: asw1-eqsin: VC mastership change - https://phabricator.wikimedia.org/T323094 (10ayounsi) 05Stalled→03Resolved That's all done. [16:54:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [16:54:29] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [18:59:42] XioNoX: topranks: i have enabled the bgpalerter on production now, there where some errors regarding prefixes without ROA's so you may see something come to your email, but take it with a pintch of salt as the config is still a work in progress [19:14:55] jbond: thanks! don’t see anything in my inbox other than your test one from earlier [19:17:27] I think all of our prefixes should have valid ROAs, do you know which ones it errored on? [19:21:43] topranks: looking again all the alerts where for AS2914 so must be a config error, there is a comment about some upstream issue in the puppet code about 2914 so it may have been a side product of some other testing and allready be fixed in prod [19:24:09] ah ok cool. great work getting it going will take a closer look when I get a minute [19:24:55] no probs it will still need a bit of tweaking but hopefully not too noisey [19:42:44] 10Puppet, 10Data-Services, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): clouddumps1002: ferm is being started on every puppet run - https://phabricator.wikimedia.org/T323324 (10Andrew) via elimination I've convinced myself that the issue here is 10_dumps_rsyncd : ` # Autogener... [20:19:03] 10Puppet, 10Data-Services, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): clouddumps1002: ferm is being started on every puppet run - https://phabricator.wikimedia.org/T323324 (10Andrew) The troublesome entries are: ` ftp.acc.umu.se mirror.accum.se ftp.acc.umu.se mirror.accum.se `... [20:27:44] 10Puppet, 10Data-Services, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): clouddumps1002: ferm is being started on every puppet run - https://phabricator.wikimedia.org/T323324 (10Andrew) I don't see any real problem with those hosts other than that they're duplicates of each other.... [20:50:40] 10Puppet, 10Data-Services, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): clouddumps1002: ferm is being started on every puppet run - https://phabricator.wikimedia.org/T323324 (10Dzahn) @Andrew Is it not maybe 65.19.157.35 ? Because that is the only IP in there and it fails to reso...