[09:53:31] Morning! [10:02:12] morning :) [10:11:29] klausman: one qs about the api-gw code change - would it make sense to have an upperbound of #charts in the (\w+) regex? [10:12:01] I am wondering if people could exploit it adding loooong things in there, in some way that I am now not able to picture [10:12:34] I have considered that (as well as enumerating the supported wikis) [10:13:18] The RE engine in Go (and thus in the APIGW) is constant-memory and constant-CPU, so that is not an attack vector. [10:14:18] On top of that DNS names have an upper limit in length as well (I forget what it is, but it's not super high maybe 128 bytes?) so that would limit anything that can be pointed at LW as well [10:15:00] Ah, found it, max DNS name length is 240 characters. [10:15:05] 250* [10:15:47] the longest wiki name so far seems to be eswikiquote, 11 chars, maybe using 15 as upper limit would be good [10:16:04] 250 is a big tweet, you can store a lot of things in there [10:16:35] Sure, let's go with 15 chars max for the \w RE [10:16:41] super [10:16:45] do we also want a 2-char minimym? [10:16:49] minimum* [10:16:55] i.e. \w{2,15} [10:16:57] yeah makes sense [10:17:06] Ok, will update the change accordingly. [10:17:10] <3 [10:17:57] Just saw this on the Golang RE page: [10:18:03] > Implementation restriction: The counting forms x{n,m}, x{n,}, and x{n} reject forms that create a minimum or maximum repetition count above 1000. Unlimited repetitions are not subject to this restriction. [10:18:23] I think we're safe :D [10:19:58] Also side thought: since DNS names must be ascii (as opposed to Unicode), the Go char class \w is safe (it's ASCII-only), but if we ever get Punycode names in there, the 15c limit might need tweaking. [10:21:21] I hope not :D [10:37:13] Same, but better to consider consequences that are unlikely. Murphy is always listening :D [11:15:48] * klausman lunch [11:39:34] * elukey lunch [12:03:45] * isaranto lunch [13:21:45] looks like there is going to be a Wikimedia hackathon in Athens :D https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023 [13:45:24] morning! [13:47:25] /o [13:48:20] elukey: how do 8 connections and 4 threads sound to you for load testing on a single pod (+just 1 connection and 1 thread)? [13:58:49] isaranto: should be a good target! Let's see how it goes [14:22:09] klausman: o/ is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/860925 ok to merge? [14:22:31] looking [14:22:47] LGTM [14:23:04] danke :) [15:00:25] I'll be 2-3m late for the meeting [15:05:29] 10Machine-Learning-Team: Add new syntax directive to blubber.yaml files to enable users to directly use docker build with blubber.yaml. - https://phabricator.wikimedia.org/T322006 (10calbon) 05Open→03Resolved [15:22:53] isaranto: a while ago I created https://logstash.wikimedia.org/app/dashboards#/view/fa21f5e0-42ef-11ed-ae81-bb78ac0690d3 to get the logs from the kserve container [15:23:08] it is currently empyt (I hope) only because no traffic is hitting the pods [15:23:24] but if you want a quick way to filter/select kserve logs you can use it [15:23:44] (not sure if it answers the question that you raised in the meeting about where to check things) [15:44:14] klausman: I saw https://gerrit.wikimedia.org/r/c/operations/puppet/+/862276 passing in the chan, and it makes a good point - we could think about leveraging our CDN to cache some api-gw results [15:44:37] but it may be tricky to set, we don't have anything to purge automatically if needed [15:45:21] Yeah, good point. I'd also have to examin what the caching strategy/key is there. [15:46:07] I think that it is ATS caching in common use cases (like if we set cache response headers etc..) [15:46:28] one thing that I don't know is if there still be a default caching TTL or not [15:46:44] mmmm lemme write it in the code review just in case [15:50:20] ack [15:51:02] It's definitely something we should look at. If we can cache at that level, our own caching can become a lot more simple (internal), though of course the question of caching for internal queries then pops up [15:51:55] elukey: I don't have access to logstash, but yeah seems like a more proper way to get the logs. All I did was dump the pod logs in files [15:52:19] isaranto: how come you don't have access? You should in theory, with the wikitech account [15:52:29] ah no wait, you need to use the shell name probably [15:52:33] have you tried? [15:52:45] shellname + wikitech password [15:53:31] no luck [15:53:50] elukey: objection to me depooling ores2009 as requested by alex/papaul? [15:54:24] klausman: ah ok already doing it :) [15:54:27] ah, you're ahead of me :) [15:55:07] klausman: I have downtimed and depooled (on the node), please go ahead with stopping daemons + shutdown :)) [15:55:17] ack [16:04:06] isaranto: lemme check because it is very strange [16:04:14] can you access turnilo.wikimedia.org? [16:15:41] elukey: I can access logstash now (turnilo as well) [16:16:18] ah okok [16:20:44] elukey: I'm done with the wrk tests for now for non MP and I want to try MP for all models. Could you patch the pods some time tomorrow? [16:22:51] isaranto: all revscoring models in staging right? [16:23:33] exactly! I'm logging off now so anytime tomorrow would be fine. I will work on the other ticket (dockerfiles) in the meantime [16:30:03] isaranto: sure let's do this - tomorrow morning we create a patch for staging only, and we deploy it [16:30:14] so it is a good way for you to check templates etc.. [16:30:31] (and we see if the templates support the extra bits for resource/limits) [16:30:36] let's sync tomorrow! [16:33:47] ack! [17:09:49] elukey: of course the \w{2,15} RE triggers a bug in Envoy, so we'll have to make do with \w+ for now [17:18:34] sigh [17:19:20] going afk folks! Have a nice rest of the day :) [17:19:58] \o