[10:20:05] jynus: regarding https://gerrit.wikimedia.org/r/c/operations/puppet/+/927119 we are not at a good point to research why tat fails and fix it [10:20:15] we are now** [10:20:36] it is just maybe firewalling? [10:20:49] that's ok, I assume -dev are not important, so not in a rush [10:20:52] these servers have changed IP address / network setup [10:21:00] technically I have not disabled backups, only its monitoring [10:21:17] yeah, but we need to know what's going on, because eventually we will do the same in non -dev hosts :-) [10:23:02] you can compare to another host you handle that has backups for the port, service, etc. [10:23:16] let me see what the logs say [10:25:06] arturo: https://phabricator.wikimedia.org/P48707 [10:29:48] ok, so likely routing or firewalling [10:29:59] is there a phab ticket already open to track this? [10:30:56] I don't belive so [10:31:58] but if you blame modules/profile/files/backup/job_monitoring_ignorelist you can find all occurences so far [10:32:13] blame as in, git-blame, not literally :-D [10:33:25] I am going to focus on another incident but ping me if you need something specific from me (usually it is some service or network issue) [10:33:44] thanks, will open a phab task and let you know [10:33:55] yeah, CC me no issue [10:38:12] T338132 [10:38:12] T338132: cloudcontrol: review connectivity with backup system - https://phabricator.wikimedia.org/T338132 [10:41:20] arturo: I would start by some sanity checks- confirming it is not just the puppet profile being absent, or the systemd daemon failing, before going to network config [10:46:05] ack, maybe add that to the ticket? [10:47:04] doing [10:59:59] thanks