[07:20:02] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9836573 (10ABran-WMF) [07:20:15] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9836574 (10ABran-WMF) [07:24:38] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9836585 (10ABran-WMF) [07:27:52] XioNoX: volans WOW [07:28:00] thank you ! [10:08:57] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:18:41] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9837047 (10MatthewVernon) @cmooney can you give me a time estimate for when you're going to be doing these, please? I'd like to put notes in my calendar. [10:21:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837068 (10MatthewVernon) From a swift POV, should just be "check cluster is happy afterwards" [10:22:28] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837074 (10ABran-WMF) [10:22:38] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837077 (10ABran-WMF) [10:22:50] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837072 (10MatthewVernon) [10:23:06] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837079 (10ABran-WMF) [10:23:16] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837080 (10ABran-WMF) [10:23:26] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837081 (10MatthewVernon) Again, from a swift POV, this should just be a case of checking the cluster is happy after... [10:23:38] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9837083 (10ABran-WMF) [10:23:48] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837085 (10ABran-WMF) [10:23:58] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9837084 (10MatthewVernon) [10:24:14] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837086 (10ABran-WMF) [10:24:32] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837089 (10ABran-WMF) [10:24:42] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837091 (10ABran-WMF) [10:24:57] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837092 (10ABran-WMF) [10:25:07] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837093 (10ABran-WMF) [10:25:25] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837099 (10ABran-WMF) [10:25:48] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837103 (10MatthewVernon) [swift-wise, just need to check cluster OK afterwards] [10:26:12] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9837104 (10MatthewVernon) [10:27:03] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837106 (10MatthewVernon) [10:27:17] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9837111 (10MatthewVernon) [swift-wise, cluster health just needs checking afterwards] [10:28:30] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837115 (10MatthewVernon) [10:29:40] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9837117 (10MatthewVernon) [from a swift POV, just need to check cluster OK afterwards] [10:30:59] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837121 (10MatthewVernon) [10:31:57] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9837137 (10MatthewVernon) ms-fe1014 will need depooling before this work is done (and then repooling afterwards). T... [10:33:11] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837166 (10MatthewVernon) [10:33:22] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9837167 (10MatthewVernon) [swift-wise, just need to check cluster OK afterwards] [10:35:03] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837189 (10MatthewVernon) [10:36:28] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9837210 (10MatthewVernon) swift-wise, just need to check the cluster's happy afterwards. moss-be1003 is part of the... [10:37:55] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9837218 (10MatthewVernon) [10:40:57] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837237 (10MatthewVernon) [10:42:00] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9837239 (10MatthewVernon) I'm on annual leave this day, so someone else will have to handle the ms frontend, which n... [10:42:32] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837241 (10MatthewVernon) [10:44:39] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9837254 (10MatthewVernon) swift-wise, this should just be a case of checking the cluster is healthy afterwards. I'm... [10:44:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837257 (10MatthewVernon) [10:45:44] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9837261 (10MatthewVernon) From the swift POV, this is just checking the cluster is happy afterwards. I'm on annual l... [14:11:48] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:33:58] volans, topranks, rename + move in a single re-image works ! [15:35:19] woot! [15:35:25] \o/ [15:35:27] nice work very well done [15:46:40] and small Netbox progress https://github.com/netbox-community/netbox/pull/16319 [16:06:48] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:45:53] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07SRE-Unowned: Provide an utility script to replace a failed device in raid 0 array - https://phabricator.wikimedia.org/T350492#9838782 (10Dzahn) I was only asking because I was clinic duty and trying to triage things tagged "Unowned". But it can stay this... [17:17:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9838998 (10Papaul) [17:52:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9839193 (10Papaul) [17:53:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9839201 (10Papaul) [18:27:23] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny, 13Patch-For-Review: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9839342 (10Arnoldokoth) Last run: ` May 28 18:18:41 mx1001 systemd[1]: Starting Generate VRTS aliases file for Exim... M... [18:36:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102 (10RobH) 03NEW p:05Triage→03Medium [18:36:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102#9839398 (10RobH) [18:37:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Patch circiut CRT-008647 - https://phabricator.wikimedia.org/T366102#9839422 (10RobH) [19:25:44] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE: Update SPF records as needed - https://phabricator.wikimedia.org/T366113 (10jhathaway) 03NEW [19:26:00] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE: Update SPF records as needed - https://phabricator.wikimedia.org/T366113#9839708 (10jhathaway) p:05Triage→03Medium [19:32:45] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny, 13Patch-For-Review: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9839737 (10Dzahn) Do we still have the 2 remaining addresses here because line 83 in the script doesn't end with a comma... [19:57:36] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny, 13Patch-For-Review: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9839783 (10Arnoldokoth) Run with comma added manually. Thanks @Dzahn ` May 28 19:44:49 mx1001 systemd[1]: Starting Gene...