[09:18:01] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9920631 (10ABran-WMF) [09:18:23] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9920630 (10Marostegui) >>! In T365995#9883497, @jcrespo wrote: > backup1009 is the main backup node for bacula on eq... [09:19:42] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9920643 (10Marostegui) [09:20:25] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9920645 (10ABran-WMF) [09:21:19] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9920659 (10ABran-WMF) [09:23:11] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9920670 (10ABran-WMF) [09:32:31] moritzm: easy +1 if you're around https://gerrit.wikimedia.org/r/c/operations/puppet/+/1049482 [09:34:54] +1d [09:36:00] We've switched idp-test to CAS 7, please report any weirdness you may encounter [09:55:24] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322#9920871 (10cmooney) >>! In T326322#9650260, @cmooney wrote: >>>! In T326322#9130092, @ayounsi wrote: >> @cmooney I came across https://w... [09:59:09] slyngs: so far so good! [10:00:46] FYI, netbox-extra is fully Netbox 4 compatible in its dev branch, reviews welcome, as usual [10:48:41] 10SRE-tools, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9921167 (10MoritzMuehlenhoff) [10:50:39] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9921174 (10jcrespo) > Is there a procedure for that so we know how to do so? Sadly, there is not. The code changes... [10:55:32] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:56:09] 10Packaging, 06Infrastructure-Foundations, 06SRE Observability: upgrade prometheus-ipmi-exporter to 1.8.0 - https://phabricator.wikimedia.org/T368088#9921189 (10MoritzMuehlenhoff) The dependency is added because some feature in the compiled Go code uses syscalls which were only wired up in 2.34 (maybe openat... [10:56:13] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9921190 (10Marostegui) I will try - but just in case @ABran-WMF please take some notes! [11:54:14] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:47:57] 10Packaging, 06Infrastructure-Foundations, 06SRE Observability: upgrade prometheus-ipmi-exporter to 1.8.0 - https://phabricator.wikimedia.org/T368088#9922011 (10herron) Nice, thanks for the pointer! It looks like `export CGO_ENABLED=0` does the right thing. At least, with this set the package builds and in... [14:55:41] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, and 3 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9922024 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7a21c2a6-e267-4150-8111-b348788c4a9b) set by cmoo... [14:58:35] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, and 3 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9922051 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=01b84d43-d6d0-4f45-bc2e-375ff79e21f8) set by cmoo... [14:59:03] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, and 3 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9922053 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=65c438b1-9725-4de3-9a45-8318edea15f1) set by cmoo... [16:36:38] 10netops, 06Data-Persistence, 06Data-Platform-SRE, 06DBA, and 3 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9922750 (10cmooney) 05Open→03Resolved [18:50:46] 10Packaging, 06Infrastructure-Foundations, 06SRE Observability: upgrade prometheus-ipmi-exporter to 1.8.0 - https://phabricator.wikimedia.org/T368088#9923370 (10herron) [18:53:37] 10Packaging, 06Infrastructure-Foundations, 06SRE Observability: upgrade prometheus-ipmi-exporter to 1.8.0 - https://phabricator.wikimedia.org/T368088#9923402 (10herron) Buster is looking fine with this deb as well. So I've gone ahead and uploaded 1.8.0 to bookworm-wikimedia, bullseye-wikimedia, and buster-w...