[04:18:56] FIRING: SystemdUnitFailed: netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:46:46] RESOLVED: SystemdUnitFailed: netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:49:44] 10SRE-tools, 10Spicerack: Redfish _get_dummy_response() should return empty json - https://phabricator.wikimedia.org/T365680 (10ayounsi) 03NEW p:05Triage→03Low [09:28:44] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9824816 (10cmooney) We also now have the issue from T365204 that we can resolve with an upgrade of JunOS. Not essential in eqiad but still I think we need to stop proc... [10:24:47] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Redfish _get_dummy_response() should return empty json - https://phabricator.wikimedia.org/T365680#9825003 (10Volans) I guess we could use something like: `lang=python >>> a = requests.Response() >>> a.status_code = 200 >>> a.raw = BytesIO(b'{}') >>> a... [11:56:31] 10netbox, 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Cookbooks: move Netbox IP allocation to spicerack module - https://phabricator.wikimedia.org/T365694 (10ayounsi) 03NEW p:05Triage→03Low [12:12:28] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697 (10ayounsi) 03NEW [12:19:12] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9825281 (10ayounsi) [14:21:08] 10CFSSL-PKI, 06Infrastructure-Foundations: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9825850 (10CDanis) Hi Arzhel, for when I do have time to look at this, do you have a recommended way of reproducing without breaking anything or potentially actually aff... [14:24:41] 10CFSSL-PKI, 06Infrastructure-Foundations: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9825896 (10ayounsi) `sudo cookbook sre.network.tls --system lsw1-f8-eqiad` e8 and f8 are still experimental switches. The whole cookbook might not run, but it should go... [15:07:38] 07Puppet, 10Wikidata, 06Wikidata Dev Team, 10wmde-wikidata-tech, and 2 others: Remove the WDCM clone (stats1007) - https://phabricator.wikimedia.org/T351072#9826085 (10Lucas_Werkmeister_WMDE) >>! In T351072#9817102, @AndrewTavis_WMDE wrote: > So basically removing the wdcm.pp related file on GitHub and its... [15:20:23] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9826145 (10cmooney) [16:13:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9826511 (10Papaul) [16:26:46] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9826583 (10cmooney) [20:22:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cuminunpriv1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:30:19] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - - https://phabricator.wikimedia.org/T365762 (10Dzahn) 03NEW [20:30:29] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - https://phabricator.wikimedia.org/T365762#9827554 (10Dzahn) [20:32:46] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9827579 (10Volans) [20:32:55] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - https://phabricator.wikimedia.org/T365762#9827577 (10Volans) →14Duplicate dup:03T365697 [21:20:14] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07SRE-Unowned: Provide an utility script to replace a failed device in raid 0 array - https://phabricator.wikimedia.org/T350492#9827705 (10Dzahn) [21:20:36] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07SRE-Unowned: Provide an utility script to replace a failed device in raid 0 array - https://phabricator.wikimedia.org/T350492#9827708 (10Dzahn) Is this SRE-tools? or datacenter-ops? or really unowned?