[08:42:27] <aiko>	 morning!
[08:48:17] <klausman>	 Morning, Aiko!
[09:24:23] <wikibugs>	 06Machine-Learning-Team: Prepare docker image for hosting the logo-detection model-server on LiftWing - https://phabricator.wikimedia.org/T362598#9763238 (10kevinbazira) 05Open→03Resolved
[10:24:28] * klausman lunch
[11:39:14] <elukey>	 hello folks
[11:43:58] <elukey>	 I just depooled inference-codfw as prep-step for the upgrade
[11:54:21] <elukey>	 I've just rolled out the coredns change to force ipv4 resolutions (needed for the new stuff to work, due to an istio bug)
[11:55:44] <elukey>	 rolling out the istio changes
[11:57:14] <elukey>	 and now I am going to proceed with revscoring damaging
[11:58:07] <klausman>	 ack, thank you!
[12:10:48] <elukey>	 httpbb works for all the damaging pods in codfw, proceeding with the others
[12:19:25] <elukey>	 article-description works
[12:19:43] <elukey>	 side note - once deployed, the sidecar takes a while before adjusting with the new cofnig
[12:19:46] <elukey>	 *config
[12:20:15] <klausman>	 5-10 Minutes? Or much longer
[12:20:17] <klausman>	 ?
[12:20:25] <elukey>	 some requests, then it works
[12:20:32] <elukey>	 timeouts etc..
[12:20:46] <elukey>	 for article-descr, it took ~30 sec to get the first response
[12:20:50] <klausman>	 You think it may be DNS caching (or lack thereof)?
[12:20:52] <elukey>	 then normalized to the usual
[12:21:31] <elukey>	 nothing really changed on the DNS caching front, except that we force A records.. I think it may be envoy getting the config upgrades from istiod
[12:21:52] <klausman>	 ah, ack.
[12:22:11] <klausman>	 Wonder if this is a one-off or something we'll se with re-deploys/restarts from now on
[12:23:01] <elukey>	 nono I think it is a one-off
[12:23:23] <klausman>	 oh, good
[12:23:37] <elukey>	 IIRC it happened in the pat
[12:26:40] <elukey>	 same happened with readability, I think that our version of istio may have some extra latency when pushing the updates to the sidecar
[12:43:00] <elukey>	 all deployed, now I am trying to make httpb to work :)
[12:43:03] <elukey>	 *httpbb 
[12:51:04] <klausman>	 what's the issue with httpbb?
[12:51:38] <elukey>	 the timeouts that I was talking above :)
[12:51:43] <klausman>	 Oh, right :)
[12:51:53] <klausman>	 My brain has too many tabs open
[12:51:54] <elukey>	 hopefully they should be fixed now
[12:57:45] <elukey>	 ok mostly done! I am fixing the wikidata headers that I missed
[12:57:51] <elukey>	 like https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1026558
[12:58:25] <elukey>	 before we used to hit the MW API with "wikidata.wikipedia.org", that doesn't exist, but the httpd's vhost config + probably the API format made it work
[12:58:52] <elukey>	 (we explicitly fix the host header also for wikibooks etc..)
[13:02:36] <klausman>	 Ah, sneaky
[13:03:03] <elukey>	 now if we don't fix it we'll get the https:// Location header redirect :(
[13:03:56] <elukey>	 Sending to inference.svc.codfw.wmnet...
[13:03:57] <elukey>	 PASS: 113 requests sent to inference.svc.codfw.wmnet. All assertions passed.
[13:04:01] <elukey>	 finally :)
[13:07:26] <elukey>	 the only one that still not works seems to be article-descriptions, afaics we don't have anything in httpbb
[13:07:50] <wikibugs>	 06Machine-Learning-Team, 06Structured-Data-Backlog: Pass image objects to the logo detection service - https://phabricator.wikimedia.org/T363506#9763776 (10mfossati) >>! In T363506#9757394, @isarantopoulos wrote: > We would need the upload wizard to send a resized image (224x224) instead of the whole file. Is...
[13:07:58] <elukey>	 now it works, ok :)
[13:12:51] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1026559 to add the test :)
[13:58:48] <wikibugs>	 06Machine-Learning-Team, 10Add-Link, 06Growth-Team, 07User-notice: Deploy "add a link" to 18th round of wikis (en.wp and de.wp) - https://phabricator.wikimedia.org/T308144#9763975 (10Trizek-WMF)
[14:26:41] <elukey>	 ok folks from my point of view codfw is ready to be pooled again
[14:26:51] <elukey>	 anybody that wants to double-check/spot-check if anything is weird?
[14:39:14] <aiko>	 looks good to me
[14:39:30] <elukey>	 thanks!
[14:39:33] <elukey>	 will do it in a bit
[14:51:29] <chrisalbon>	 Morning all
[15:00:34] <elukey>	 o/
[15:03:29] <elukey>	 codfw repooled!
[15:05:37] <elukey>	 I am checking the following
[15:05:38] <elukey>	 https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&from=now-30m&to=now&var-cluster=codfw%20prometheus%2Fk8s-mlserve&var-namespace=All&var-backend=mw-api-int-ro.discovery.wmnet&var-response_code=All&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99
[15:06:32] <elukey>	 so far latency and response codes are ok
[15:24:18] <elukey>	 please ping me if you see/spot anything weird, so far I don't see anything
[16:15:30] <chrisalbon>	 YES
[16:15:35] <chrisalbon>	 Thanks elukey
[16:15:35] <elukey>	 folks I am going afk for the evening, will check later just in case
[16:15:37] <elukey>	 <3
[16:15:42] <elukey>	 have a good rest of the day!
[16:15:45] <chrisalbon>	 Night elukey! Thanks for this!
[16:15:48] <elukey>	 I also left a note on slack