[03:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [03:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [06:49:33] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#10810282 (10fgiunchedi) >>! In T371375#10808026, @cmooney wrote: >>>! In T371375#10807881, @cmooney wrote: >> Let me double check and report back. > > So i... [07:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [07:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [09:46:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10810721 (10cmooney) [10:11:50] 10netbox, 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox: librenms report errors - https://phabricator.wikimedia.org/T379907#10810797 (10ayounsi) 05Open→03Resolved a:03Volans Fixed by @Volans in https://gerrit.wikimedia.org/r/c/operations/software/netbox-extras/+/1135381 [11:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [11:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [12:14:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10811131 (10cmooney) [12:30:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10811243 (10cmooney) [12:32:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10811259 (10cmooney) [12:35:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10811287 (10cmooney) [13:18:30] volans, topranks, any reason to NOT delete all the cables in https://netbox.wikimedia.org/dcim/cables/?unterminated=True ? :) [13:18:39] asking before I do it [13:19:29] No I think it's ok to proceed [13:19:52] like we said interesting to know what happened, they all seem to be server-facing, so probably just errors/fix-ups left incomplete [13:20:21] but I think what we can do is delete the existing ones, and keep an eye out if more appear, then ask who was working on it what happened just in case there is some gap in our workflow [14:03:09] topranks: gaps in our workflow fixed in https://gerrit.wikimedia.org/r/c/operations/software/netbox-extras/+/1144556 (Especially the offline device script) [15:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [15:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:57:54] 07Puppet, 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812549 (10bd808) @bking Could you make some time to look into these failures? https://opensta... [16:00:49] 07Puppet, 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812563 (10bking) [16:01:18] 07Puppet, 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812564 (10bking) Thanks for the ping, @bd808 . I'm not sure when we'll... [16:32:12] 07Puppet, 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812752 (10taavi) The likely fix for this will be migrating that file d... [17:03:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Move connections on ssw1-f1-codfw to match normal pattern - https://phabricator.wikimedia.org/T393936 (10cmooney) 03NEW p:05Triage→03Medium [17:03:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Move connections on ssw1-f1-codfw to match normal pattern - https://phabricator.wikimedia.org/T393936#10812996 (10cmooney) [17:38:05] 10Packaging, 06Infrastructure-Foundations, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): upgrade prometheus-ipmi-exporter to 1.8.0 - https://phabricator.wikimedia.org/T368088#10813210 (10herron) 05Open→03Resolved a:03herron [19:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [19:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [19:52:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Move connections on ssw1-f1-codfw to match normal pattern - https://phabricator.wikimedia.org/T393936#10813758 (10cmooney) [20:00:16] 10Mail, 10fundraising-tech-ops: DMarc Email Address for Wikimedia.org - https://phabricator.wikimedia.org/T316899#10813789 (10Jgreen) 05Open→03Declined Requesting user is no longer with WMF [20:26:05] Hey, check_user has been broken for (I'm assuming) a long time now, since j.bond left (I'm guessing the google API key was from that account). Since there are other methods of checking accounts should check_user just get axed? [21:05:02] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: https://netbox-exports.wikimedia.org/dns.git takes age to clone - https://phabricator.wikimedia.org/T387575#10813984 (10hashar) 05Open→03Resolved a:03hashar [23:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting