[03:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [03:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:06:37] brett: I'd suggest opening a task [07:09:40] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr2-drmrs) - https://phabricator.wikimedia.org/T393991 (10LSobanski) 03NEW [07:11:41] 10netops, 06Infrastructure-Foundations, 06SRE: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#10814904 (10ayounsi) a:05ayounsi→03Papaul Re-assigning it to Papaul to do the change on `ulsfo` and `eqsin`. It is a good training opportunity, and would remove moving p... [07:36:52] 10Mail, 06Infrastructure-Foundations, 07Upstream: lists.wikimedia.org - adhere to RFC8048 (one-click unsubscribe) dkim guidelines - https://phabricator.wikimedia.org/T355802#10814990 (10Aklapper) [07:49:30] 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996 (10ayounsi) 03NEW [07:51:56] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:51:56] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [07:58:35] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox: unterminated cables - https://phabricator.wikimedia.org/T393188#10815065 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [08:23:41] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr2-drmrs) - https://phabricator.wikimedia.org/T393991#10815152 (10ayounsi) a:03ayounsi sent an email to PCH [08:41:56] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [08:41:56] RESOLVED: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:42:35] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [08:42:35] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:46:40] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#10815286 (10ayounsi) 05Open→03Stalled Current status: * `fasw`: done * `pfw`: {T393996} should be the last step * `mr`: waiting on JTAC case 2025-0506-688713 [09:14:58] ERROR:homer.transports.junos:Commit check error on lsw1-e1-codfw.mgmt.codfw.wmnet: Referenced filter 'loopback4' is not defined In [edit interfaces lo0 unit 0 family inet] (filter) [09:15:02] expected? [09:15:16] on e1/e3/f1/f3 [09:15:30] volans: they're being setup so yeah [09:15:33] k [09:15:39] (cc topranks) [09:16:45] yep I am looking at it [09:17:47] think it's due to missing entry in devices.yaml [09:18:52] actually let me make them planned in netbox dc ops probably shouldn't have set them to active [09:58:28] 07Puppet, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268#10815486 (10taavi) 05Resolved→03Open a:05jbond→03None Re-opening as the `purge_sudoers_d` flag was never actually enabled. [12:13:37] 10netops, 06Infrastructure-Foundations, 06SRE: Stage and configure new Juniper switches in codfw rows E/F - https://phabricator.wikimedia.org/T394021 (10cmooney) 03NEW p:05Triage→03Medium [12:14:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10816007 (10cmooney) [12:14:07] 10netops, 06Infrastructure-Foundations, 06SRE: Stage and configure new Juniper switches in codfw rows E/F - https://phabricator.wikimedia.org/T394021#10816008 (10cmooney) [12:21:15] 07Puppet, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268#10816016 (10taavi) {P76015} [12:36:13] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10816116 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a82fd52b-494d-4956-9f75-7cd844fe0007) set by ayounsi@cumin1002 for 2:00:00 on 1 host(s) and their servic... [12:42:35] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [12:42:35] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [13:40:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10816412 (10ayounsi) [14:53:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Move connections on ssw1-f1-codfw to match normal pattern - https://phabricator.wikimedia.org/T393936#10816875 (10Jhancock.wm) @cmooney got them swapped for you [15:12:35] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [15:12:35] RESOLVED: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:14:01] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [15:14:01] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [16:36:52] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Upgrade management switches to Junos 21.4 - https://phabricator.wikimedia.org/T390814#10817487 (10Papaul) We don't' have the updated version pushed to msw1-codfw (JUNOS 21.4R3.15 built 2022-09-03 07:18:28 UTC) we supposed to use (JUNOS 21.4R3-S10.9 built 2025... [16:46:12] XioNoX: Filed, thanks: https://phabricator.wikimedia.org/T394072 [17:19:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Upgrade management switches to Junos 21.4 - https://phabricator.wikimedia.org/T390814#10817789 (10Papaul) 05Open→03Resolved This is complete [17:52:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Move connections on ssw1-f1-codfw to match normal pattern - https://phabricator.wikimedia.org/T393936#10817962 (10cmooney) 05Open→03Resolved a:03cmooney Super @Jhancock.wm that all looks good now and links are working :) ` c... [18:02:25] FIRING: SystemdUnitFailed: isc-dhcp-server.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:27:25] RESOLVED: SystemdUnitFailed: isc-dhcp-server.service on install2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:17:35] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [19:17:35] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:17:35] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [23:17:35] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts