[08:22:08] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Netbox to 4.x - https://phabricator.wikimedia.org/T336275#9975924 (10ayounsi) The initial puppet run on netboxdb1003 failed with: > Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Er... [09:04:49] Hey folks, I have upgraded httpd on netbox for https://phabricator.wikimedia.org/T369885 [09:04:53] if you see anything weird lemme know [09:05:12] thx [09:24:28] slyngs: o/ as FYI I updated httpd on the idm nodes, lemme know if you see anything weird [09:24:55] I'll check, but I doubt it, it's all very basic [09:25:01] super thanks [09:26:22] Seems very much unbroken :-) [09:33:13] thankkksss [09:33:50] rolledout completed, touched a lot of nodes (including phab/gerrit/etc..) but nothing to report [11:38:03] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9976274 (10cmooney) 05Open→03Resolved [12:50:50] 10netbox, 06Infrastructure-Foundations: Netbox : sync src/ submodule - https://phabricator.wikimedia.org/T369690#9976413 (10elukey) In my opinion the easiest way would be to use https://gerrit.wikimedia.org/g/operations/software/netbox, setting the submodule to pull from gerrit. Even if we don't maintain any p... [13:07:59] 10CFSSL-PKI, 06Infrastructure-Foundations: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9976497 (10elukey) Completely different use case: `traffic-cache-upload-bullseye.traffic.eqiad1.wikimedia.cloud` In there purged needs to get a cfssl discovery cert, bu... [15:37:01] 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: CFSSL gencert "remote error: tls: certificate require" - https://phabricator.wikimedia.org/T355750#9977022 (10elukey) I tried to validate `client_auth_CA.pem` on pkiXXXX and it looks good (allowing Puppet5/7/PKI client certs), so it must be somet... [15:56:00] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Create the python-release repository - https://phabricator.wikimedia.org/T367410#9977109 (10elukey) [15:56:55] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Create the python-release repository - https://phabricator.wikimedia.org/T367410#9977114 (10elukey) [15:57:31] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Create the python-release repository - https://phabricator.wikimedia.org/T367410#9977118 (10elukey) Next steps: * Wait for https://gitlab.wikimedia.org/repos/sre/python-release/-/merge_requests/1 to be reviewed and merged. * Test p... [21:02:41] 10Mail, 06Infrastructure-Foundations: Investigate using BATV to thwart backscatter - https://phabricator.wikimedia.org/T369952 (10jhathaway) 03NEW [21:02:54] 10Mail, 06Infrastructure-Foundations: Investigate using BATV to thwart backscatter - https://phabricator.wikimedia.org/T369952#9978298 (10jhathaway) p:05Triage→03Low [21:03:40] 10Mail, 06Infrastructure-Foundations: Investigate using BATV to thwart backscatter - https://phabricator.wikimedia.org/T369952#9978299 (10jhathaway) Postfix does not have built in BATV support. There is a milter, https://github.com/AGWA/batv-tools/tree/master, but it is unmaintained. [21:14:18] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:17:13] 10Mail, 06Infrastructure-Foundations: Investigate using BATV to thwart backscatter - https://phabricator.wikimedia.org/T369952#9978314 (10jhathaway) I asked folks on mailop for advice, https://www.mail-archive.com/mailop@mailop.org/msg22495.html [22:09:18] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed