[08:43:21] oh, I missunderstood puppet's trouble it seems. I was under the impression that it would only reject deleting the directory where it's non-empty. So I only cleaned up where is was. [08:43:30] I'll clean up the rest [13:25:03] This is my first time interacting with the service mesh. I'm trying to replace an api call to https://meta.wikimedia.org/w/api.php?action=streamconfigs&constraints=canary_events_enabled=1 to some internal calls to the mesh. I'm trying to call mw-api-int via http://localhost:6501/w/api.php? but I keep getting a 404. Is that API not exposed via [13:25:03] mw-api-int? [13:27:18] brouberol: I think that you need to set the hostname as you call it. [13:28:28] Here is an example, I think: https://wikitech.wikimedia.org/wiki/Data_Platform/Internal_API_requests#Using_requests [13:31:12] ooh, thanks, let me try that [13:32:40] >>> requests.get("http://localhost:6501/w/api.php?action=streamconfigs&constraints=canary_events_enabled=1", headers={"Host": "meta.wikimedia.org"}) [13:32:40] [13:33:06] Cool. Here is an example of where we do it in nodejs: https://gitlab.wikimedia.org/repos/data-engineering/mpic/-/blob/ad95257f1bdc17c7b33ab043eae1043bebbb97f0/util/actionApi.js#L6-19 [13:35:35] And here is a python library for eventstreamconfig https://gerrit.wikimedia.org/g/analytics/refinery/+/master/python/refinery/eventstreamconfig.py [13:36:13] ok so that solves half of our issues. Now, to figure out how to talk to noc.wikimedia.org via the mesh [13:39:07] as mw-misc isn;t exposed in the mesh AFAICS, we could go through the discovery.wmnet endpoint [13:40:31] you can always add a new service mesh listener if you need one: https://wikitech.wikimedia.org/wiki/Envoy#Add_a_new_service_(listener) [13:41:27] oh, simple as that [13:41:31] nice. let me try! [13:48:46] jayme: I've submitted https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114383 [13:56:57] brouberol: btw the Host: header requirement isn't a quite property of the service mesh, it's enforced by mediawiki on the other end / the receiver (and the service mesh not modifying the requests) [13:57:22] yep, I thought as much, thanks for confirming! [13:57:49] sorry if I was overexplaining :) [14:05:33] no no, on the contrary! [14:06:10] I think this is the first time that "service mesh" becomes more than an abstract concept for me, so, I happily accept all information [14:06:17] we should document it better [14:18:23] jayme: once the puppet/envoy patch is merged, should I redeploy a specific service, or will the new config eventually propagate via xDS ? [14:23:58] brouberol: the latter, make sure that puppet ran on deploy2002 because the new config should have been created in there [14:24:36] then you can pick up the new localhost proxy port adding mw-misc to the "discovery" settings of your helfile config [14:24:52] and it should trigger a change for the envoy sidecar's yaml config [14:29:12] thanks! [14:43:53] curl -H "Host:noc.wikimedia.org" https://mw-misc.discovery.wmnet:30443/conf/dblists/open.dblist (from the deployment host) works, but requests.get("http://localhost:6508/conf/dblists/open.dblist", headers={"Host": "noc.wikimedia.org"}) from within a pod fails with an HTTP 404 [14:44:07] I'm thinking I might have messed up the service config, in some way [14:45:18] as mw-misc.discovery.wmnet points to a wikikube ingress DNS record, I've set `sets_sni: true` as suggested by https://wikitech.wikimedia.org/wiki/Envoy#Add_a_new_service_(listener) [14:46:41] ah, turns out that the underlying service (in service.yaml) is in `service_setup` state [14:49:56] according to T341859: we might need to add ingress support to the mediawiki chart, as with the amount of traffic I'd expect here, an LVS endpoint would be overkill [14:49:56] T341859: Move noc.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T341859 [14:50:12] so that might explain the service state [17:09:06] brouberol: o/ [17:09:15] was https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1052701 ever deployed to dse-k8s-eqiad? [17:09:32] I am seeing some violations in https://logstash.wikimedia.org/app/discover#/view/7f276c90-f8a0-11ee-be54-8fd74c07934f?_g=h@b8b0449&_a=h@c3b7e3a for istio [17:10:02] if not let's do it tomorrow [17:10:07] it should be a istioctl upgrade [20:38:26] I think it was yes. At least IIRC [20:41:03] and if I'm wrong, then, for sure!