[10:45:38] klausman: would you be so nice to deploy ores-legacy and recommendation-api-ng following https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1041072 ? [10:53:38] sure, can do [10:54:06] Also, thanks a ton for taking care of those NP changes <3 [10:59:45] cool, thanks. Although these where not network-policy related changes as you probably saw ;) [11:24:49] yeah, belatedly :) [11:25:16] I have way too many plates spinning in the air rn [11:31:15] jayme: all done! [11:35:06] ack, thanks [13:40:46] brouberol: I see PCC failures on deployment_server because Error while evaluating a Function Call, DNS lookup failed for elastic1062.eqiad.wmnet Resolv::DNS::Resource::IN::A (file: /srv/jenkins/puppet-compiler/2886/change/src/modules/profile/functions/kubernetes/deployment_server/elasticsearch_external_services_config.pp, line: 22, column: 12) [13:40:51] https://puppet-compiler.wmflabs.org/output/1041644/2886/deploy1002.eqiad.wmnet/change.deploy1002.eqiad.wmnet.err [13:40:56] is that a temporary thing? [13:42:40] that's really weird [13:44:02] have you pulled down the latest facts to the PCC servers? Nothing should've changed with that host, but I might try that next [13:45:07] ref https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Updating_nodes [13:46:15] the data is gatheres from a puppetdb query, I don't think updating facts will don anything there [13:46:26] is elastic1062.eqiad.wmnet supposed to be gone? [13:48:49] nope. Host is still up and I don't recall any decom tickets for it...it's not reached its rotation date [13:49:34] have to go AFK but might ask DC Ops if they've done anything w/it lately? [13:49:43] otherwise I'll check in ~20 [14:06:14] jayme: I wonder if that has to do with the fact that it might have gotten rebooted today. I know there was a round of host maintenance, some scheduled for today [14:06:17] is it still failing? [14:07:16] ah no. Second try actually worked [14:07:29] but why should a dns lookup fail when the host got rebootet? [14:08:44] oh nevermind, I somehow had the mental model à la k8s, where the pod IP isn't resolved when not ready