[00:05:26] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:53:03] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:33:30] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [06:12:45] 10netops, 06Infrastructure-Foundations, 06SRE: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937#10945822 (10Morale99) For anyone dealing with fiber connections or testing networks, using a good quality [[ https://www.firefold.com/collections/fibe... [06:53:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [08:08:12] There is some activity around UEFI for Ganeti, even though I don't understand all the words : https://github.com/ganeti/ganeti/issues/1374#issuecomment-3001826897 [10:53:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [12:11:03] 10SRE-tools, 10Spicerack: Flaky icinga unit tests - https://phabricator.wikimedia.org/T397833 (10fgiunchedi) 03NEW [12:11:33] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Flaky spicerack icinga unit tests - https://phabricator.wikimedia.org/T397833#10946842 (10fgiunchedi) [13:45:07] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Alert when anycast-healthchecker withdraws BGP route - https://phabricator.wikimedia.org/T374619#10947176 (10ssingh) I am going to tackle this for the DNS hosts at least and then we can revisit a generic solution. [14:53:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [15:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [18:14:21] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865 (10Jgreen) 03NEW [18:21:09] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948245 (10Jgreen) [18:23:49] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948248 (10Jgreen) [18:53:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [19:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [19:51:44] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948519 (10Jgreen) [19:51:50] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948521 (10Jgreen) [20:04:30] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948573 (10Jgreen) [20:05:25] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948575 (10Jgreen) a:05Dwisehaupt→03None [20:05:38] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948578 (10Jgreen) [20:06:10] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948582 (10Jgreen) [20:30:19] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948644 (10Jgreen) [20:37:31] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948654 (10Jgreen) [20:46:01] 07Puppet, 10Beta-Cluster-Infrastructure: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877 (10bd808) 03NEW [21:06:57] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948779 (10Jgreen) [21:21:33] 07Puppet, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#10948815 (10bd808) `lang=shell-session bd808@deployment-puppetserver-1:~$ sudo -i puppet agent -t... [21:26:59] 07Puppet, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: /usr/local/bin/puppetserver-deploy-code emits scary looking error messages during a `git rebase` operation - https://phabricator.wikimedia.org/T397877#10948820 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [22:53:04] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [23:33:31] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts