[08:35:15] 10CAS-SSO, 06Infrastructure-Foundations: Upgrade CAS to 6.6.15 - https://phabricator.wikimedia.org/T366140#9849572 (10SLyngshede-WMF) 05Open→03Resolved [10:45:50] 10CFSSL-PKI, 06Infrastructure-Foundations: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9849761 (10cmooney) >>! In T355750#9825850, @CDanis wrote: > Hi Arzhel, for when I do have time to look at this, do you have a recommended way of reproducing without bre... [10:49:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9849768 (10cmooney) So I tested pushing with Homer to the devices in row D and it was pretty much successful :) NOTE: As the devices need some additio... [10:53:53] 10CFSSL-PKI, 06Infrastructure-Foundations: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9849777 (10cmooney) Actually something tangentially related, //lsw1-a1-codfw// had a cert generated previously. This switch has now been moved and re-purposed with a ne... [11:13:42] 10netops, 06Infrastructure-Foundations, 06SRE: Include vlans with defined IRB int in device vlans even if no port present - https://phabricator.wikimedia.org/T366348 (10cmooney) 03NEW p:05Triage→03Low [11:17:59] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Include vlans with defined IRB int in device vlans even if no port present - https://phabricator.wikimedia.org/T366348#9849823 (10cmooney) Diff with this patch applied on one of the new codfw switches: ` cmooney@wikilap:~$ homer lsw1-d2-cod... [11:18:43] FIRING: SystemdUnitFailed: netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:33:05] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Include vlans with defined IRB int in device vlans even if no port present - https://phabricator.wikimedia.org/T366348#9849839 (10cmooney) [11:45:44] RESOLVED: SystemdUnitFailed: netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:35:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9850037 (10cmooney) @papaul @Jhancock.wm I noticed that the leaf in rack d8 is reporting one of it's power supplys down: ` cmooney@lsw1-d8-codfw> show... [13:53:14] 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 10Spicerack, 06SRE: Migrate puppet merges to a cookbook - https://phabricator.wikimedia.org/T366355 (10MoritzMuehlenhoff) 03NEW [14:19:46] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9850142 (10cmooney) @ABran-WMF thanks for creating all the tasks! Really appreciated, I did not expect to come back and see that :) >>! In T348977#9837047, @MatthewVe... [14:24:43] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9850163 (10cmooney) [14:25:56] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9850164 (10cmooney) [14:27:39] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9850169 (10cmooney) p:05Triage→03Medium a:05MatthewVernon→03cmooney [14:27:53] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9850174 (10cmooney) @ABran-WMF thanks for creating all the tasks! Really appreciated, I did not expect to come back and see that :) >>! In T348977#9837047, @MatthewVe... [14:28:06] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9850165 (10cmooney) p:05Triage→03Medium a:05MatthewVernon→03cmooney [14:39:52] 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 10Spicerack, 06SRE: Migrate puppet merges to a cookbook - https://phabricator.wikimedia.org/T366355#9850216 (10elukey) [15:31:23] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade ssw1-e1-eqiad to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361 (10cmooney) 03NEW p:05Triage→03Medium [15:50:44] FIRING: SystemdUnitFailed: update-tails-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:56:35] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade ssw1-e1-eqiad to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850549 (10cmooney) [15:58:51] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade ssw1-e1-eqiad to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850564 (10cmooney) [15:58:55] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9850565 (10cmooney) [15:59:50] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9850567 (10cmooney) [16:25:36] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850699 (10cmooney) [16:25:58] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850702 (10cmooney) [16:27:34] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850704 (10cmooney) [16:30:02] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9850726 (10cmooney) [16:31:10] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9850733 (10cmooney) [17:45:44] FIRING: [2x] SystemdUnitFailed: update-tails-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:50:44] RESOLVED: [2x] SystemdUnitFailed: update-tails-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:46:40] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Postfix outbound rollout sequence, mx-out - https://phabricator.wikimedia.org/T365395#9851280 (10jhathaway) [19:20:44] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:18:43] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed