[00:05:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:45:48] 10SRE-tools, 06Infrastructure-Foundations, 06serviceops-radar: Add --min-uptime to cookbooks - https://phabricator.wikimedia.org/T419967#11721774 (10JMeybohm) >>! In T419967#11720994, @Ajuanca wrote: > What's task `T419960` about? I don't enough privilegies to access it. Yes, I think a parameter with explici... [08:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:37:38] slyngs: is that one for you ? ^ [08:37:59] I can take it :-) [08:38:53] <3 [09:00:56] FIRING: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:08:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11721899 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b34081b8-989e-49aa-91c7-56b4548775e2) set by slyngshede@cumin1003 for 4:... [09:21:51] I'm upgrading Postgresql on the netbox DB nodes, there might be very brief glitches in using Netbox [09:30:56] RESOLVED: [2x] ProbeDown: Service mirror1001:443 has failed probes (http_mirrors_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mirror1001:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:32:27] jayme: congrats on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1242289 ! I hope it goes well [09:33:08] XioNoX: just touched it ...it does not feel hot yet and there is no smoke [09:33:13] I'm dissapointed [09:34:04] boring is sometime a good thing [09:55:27] always a good thing in this game :) [10:07:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, and 2 others: ULSFO: Update ULSFO LVS service IP's - https://phabricator.wikimedia.org/T418971#11722088 (10SLyngshede-WMF) 05Open→03Resolved @Papaul Done :-) [12:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:11:21] Why is that only broken when run as a service ... [12:14:27] users, permissions? [12:15:33] It runs as root, so should be fine [12:16:18] I see a timeout when talking to stat.ripe.net on the last run [12:17:24] I think those are just warnings, but that's a good point [12:18:01] maybe the proxies? [12:18:12] set for your shell session but not for the deamon? [12:18:14] It can generate the file [12:19:30] no, that's an ERROR and the script will exit 1 if at least one network fails to update like that [12:19:48] (lines 363-365 are what's responsible for handling that error specifically) [12:19:56] Found it GeekyWorks [12:20:10] Thanks :-) [12:30:23] Oh weird, all their prefixes just went away [13:03:06] o/ still looking for reviews for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1212097 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1211650 [13:48:08] We have a CFSSL profile change waiting review from IF if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1251117 [15:30:25] FIRING: [2x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:35:25] FIRING: [2x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:35:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed