[06:47:12] 10netops, 06Infrastructure-Foundations, 06SRE: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606#11510983 (10ayounsi) Even more odd is that it flaps, even when no change is being done on the device. One mail it will remove it, another mail it will re-add it. Nothing... [08:42:44] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11511124 (10ayounsi) > Is sretest2003 the only one that shows this behavior, or do we have others? I am particularly interested in if you were able to set the... [09:00:14] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Remove pfw configuration related to former pybal/LVS service - https://phabricator.wikimedia.org/T414015#11511149 (10ayounsi) 05Open→03Resolved All done! [10:23:37] 10netops, 06Infrastructure-Foundations, 06SRE: Offline script - adjust to work with fundraising - https://phabricator.wikimedia.org/T414321 (10cmooney) 03NEW p:05Triage→03Medium [12:11:51] 10netops, 06Infrastructure-Foundations, 06SRE: Offline script - adjust to work with fundraising - https://phabricator.wikimedia.org/T414321#11511995 (10cmooney) [12:42:32] 10SRE-tools, 06DBA, 06Infrastructure-Foundations: Improve database master switchover script - https://phabricator.wikimedia.org/T200306#11512075 (10Marostegui) →14Duplicate dup:03T409926 [13:36:25] FIRING: SystemdUnitFailed: dnsmasq.service on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:37:22] should recover soon ^ [13:46:25] RESOLVED: SystemdUnitFailed: dnsmasq.service on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:46:55] FIRING: MaxConntrack: Max conntrack at 82.09% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [13:51:55] FIRING: MaxConntrack: Max conntrack at 100% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [14:00:25] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:01:55] RESOLVED: [2x] MaxConntrack: Max conntrack at 92.03% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [14:05:25] RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:17:10] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:21:55] RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:33:55] FIRING: MaxConntrack: Max conntrack at 83.63% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:38:55] FIRING: MaxConntrack: Max conntrack at 97.67% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:48:55] RESOLVED: [2x] MaxConntrack: Max conntrack at 99.99% on ganeti3005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack