[00:19:05] (SystemdUnitFailed) resolved: netbox_report_accounting_run.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [05:24:05] (SystemdUnitFailed) firing: (5) librenms-alerts.service Failed on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:29:06] (SystemdUnitFailed) resolved: (5) librenms-alerts.service Failed on netmon1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [09:19:59] FYI, I'll switch the netbox DB hosts to nftables in a bit [09:20:51] ack [09:58:09] done a while ago, BTW. let me know if you run into any issues [10:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [11:46:36] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:48:54] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [13:54:50] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [14:41:44] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [15:03:21] 10Puppet: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870 (10TheresNoTime) [15:10:33] 10Puppet, 10SRE, 10Patch-For-Review: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870 (10TheresNoTime) [15:30:09] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [15:44:20] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [16:20:57] 10Puppet, 10Instrument-ClientError, 10Patch-For-Review: Google Translate and other translate services triggering client error alert - https://phabricator.wikimedia.org/T351738 (10Jdlrobson) @colewhite could you please help me with this? [16:33:20] 10Puppet, 10Wikidata, 10wmde-wikidata-tech, 10Technical-Debt, 10Wikidata Analytics (Kanban): Remove the WDCM clone (stats1007) - https://phabricator.wikimedia.org/T351072 (10Manuel) [16:37:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on a kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) [16:38:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) [16:41:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) p:05Triage→03Medium a:03Clement_Goubert [17:34:03] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Netbox: get rid of WMF Production Patches - https://phabricator.wikimedia.org/T310717 (10ayounsi) @volans I'd be interested by a review of https://github.com/XioNoX/netbox/commit/e4b5b27ecc24eb069da10b999a7057e648a32c89 as well to eventually send it... [18:23:44] 10Puppet, 10SRE, 10Patch-For-Review: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870 (10ClydeFranklin) I love fopkses! [18:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [19:46:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-codfw: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [19:50:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [19:54:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [20:19:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-codfw: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) [20:30:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) [20:31:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [21:44:48] 10Puppet, 10SRE, 10Patch-For-Review: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870 (10Peachey88) Is there any particular reason we desire to enable additional tech-debt by having to maintain this for years to come? [21:45:34] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-codfw: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [22:05:40] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [22:10:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10ops-codfw: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [22:18:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [22:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced