[09:51:13] o/ investigating connection issues to cloudelastic which is a service behind lvs, wondering if there are logs somewhere that track what hosts get pooled/depooled? [10:02:49] found the pybal grafana dashboard which I think answers my question [10:48:10] I have a lingering deleted node (relforge1006.eqiad.wmnet) that's causing PCC runs to fail for the deployment server, anyone knows how I can make sure it's purged from PCC? [10:50:05] you can force a manual sync: https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes [10:50:28] I did that yesterday, didn't fix it unfortunately [10:51:18] curl http://localhost:8080/pdb/query/v4/nodes/relforge1006.eqiad.wmnet returns the node object, but it's not in /var/lib/catalog-differ/puppet/ anywhere [15:54:17] moritzm: yeah so I ended up deactivating the node through a curl to the pdb cmd api on pcc-db1002 [15:55:10] Because nothing I did fixed it (also the whole purging nodes section seems to be outdated, that script doesn't exist anymore afaict) [16:35:19] <_joe_> claime: there is a cronjob rebuilding the catalog from scratch every morning, so probably you just got there at the wrong times [16:35:43] _joe_: two days in a row then [17:42:49] on graphite1005: Failed to open TCP connection to puppetserver1003.eqiad.wmnet:8140 (getaddrinfo: Temporary failure in name resolution) [17:43:28] graphite2004 (and alert/contint) dont not have the issue