[01:09:04] <wikibugs>	 10serviceops, 10MW-1.40-notes (1.40.0-wmf.7; 2022-10-24), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) >>! In T292552#8354775,...
[01:12:51] <wikibugs>	 10serviceops, 10MW-1.40-notes (1.40.0-wmf.7; 2022-10-24), 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10IKhitron) Well, It's too late, the problem never was fixe...
[10:44:56] <elukey>	 hi folks, I have a high level question about rate limits in api-gateway
[10:45:26] <elukey>	 IIUC at the moment a backend like lift wing can set two rate limits - anon users and authed ones
[10:45:58] <elukey>	 not sure what is the granularity for the latter (like: specific rate limits for certain client-ids authenticated via bearer token)
[10:46:23] <elukey>	 but I was wondering what's best when a bot/spammy-user/etc.. will hit Lift Wing and we'll have to rate limit it or block it
[10:46:55] <elukey>	 requestctl is surely the first line of defense, but if so how should we ratelimit apps on the api-gateway?
[10:47:12] <elukey>	 I mean what best practices should we use etc..
[10:47:52] <elukey>	 ideally I'd love to have a client-id in some http header for auth-ed requests landing on Lift Wing, so that I can easily identify and rate limit some client if needed
[10:48:08] <elukey>	 (either on the api-gateway or on lift wing, both of them have enovy etc..)
[11:13:38] <wikibugs>	 10serviceops, 10MediaWiki-REST-API, 10Parsoid, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): HTTP 412 Errors when editing Officewiki - https://phabricator.wikimedia.org/T331629 (10daniel) Analysis: * on officewiki, VE is configured to use REST endpoints exposed by the Parsoid extension directly. It doesn't...
[13:40:10] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy - https://phabricator.wikimedia.org/T275035 (10akosiaris) >>! In T275035#8653555, @JMeybohm wrote: > I took the lazy approach to just convert the comm...
[13:44:06] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Refactor common_templates/0.2/default-network-policy-conf.yaml into a GlobalNetworkPolicy - https://phabricator.wikimedia.org/T275035 (10JMeybohm) Thanks for the insights. Let's add the GlobalNetworkPolicy then as proposed. I'll open a sepa...
[13:45:22] <jayme>	 akosiaris: if you're okay with the general approach I'll rebase https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/893019/ - just did not want to do it before knowing your opinion 
[13:47:43] <akosiaris>	 btw, 1 thing that is interesting in all of this
[13:48:20] <akosiaris>	 if the destination of a request originating in the cluster is an LVS IP that is also in the cluster (e.g. think almost all the services)
[13:48:34] <akosiaris>	 the allow pod-to-pod thing allows it
[13:49:37] <akosiaris>	 e.g. curl https://blubberoid.svc.eqiad.wmnet:<port> will just work
[13:49:48] <akosiaris>	 despite that hostname resolving to 10.2.2.X 
[13:50:01] <akosiaris>	 that might be confusing to a newcomer
[13:50:53] <jayme>	 hm? allow-pod-to-pod should only allow egress to the pod network(s), no?
[13:51:16] <akosiaris>	 yes. But DNAT ;-)
[13:51:19] <jayme>	 why should it allow LVS IPs?
[13:51:49] <akosiaris>	 cause the LVS IP will get DNATed to a pod IP on the node, apparently before the networkpolicy is consulted
[13:52:22] <jayme>	 aaah 💡 because the nodes do have the LVS IP themselves
[13:53:33] <akosiaris>	 yes. That means that after the cleanup, we probably wanna start looking a bit closer to our allow-pod-to-pod global rule
[13:53:50] <akosiaris>	 we are hitting enough critical mass now that it makes sense to revisit
[13:54:01] <jayme>	 yep
[13:56:22] <jayme>	 Will do the rebase as soon as I find a not so freezing place in the train station of Osnabrück
[13:57:43] <akosiaris>	 I thought it was your day off ?
[13:57:57] <akosiaris>	 why are you on your laptop ? Enjoy the cold instead!
[14:02:07] <jayme>	 yeah, it is but I'm in trains for a couple of hours so I can catch up on some things
[14:10:28] <moritzm>	 if you're willing to leave the station, train there's a nice Syrian place with baklava, maybe 200 metres down Möserstr (after the bus station outside)
[14:12:21] <jayme>	 moritzm: Albasha?
[14:14:54] <moritzm>	 no idea about the name, but I doubt there's more than one Syrian restaurant :-) it's a few houses after the Burger King 
[14:18:42] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Decide on new Pod and Sevice IPv4 ranges for wikikube clusters - https://phabricator.wikimedia.org/T326617 (10Marostegui)
[14:27:47] <jayme>	 ack. That's Albasha then ;-)
[14:36:36] <moritzm>	 ok :-)
[14:51:31] <taavi>	 just spotted deploy1002 in T308339, wondering if we should take the opportunity to do that now given 2002 is the active one
[14:57:46] <jayme>	 akosiaris: the rebased patch creates a strange diff in CI claiming that some deployments could not be rendered in HEAD. Not sure why that is...
[14:57:55] <jayme>	 I won't merge before Monday anyways because trains
[15:01:57] <akosiaris>	 ok
[15:51:20] <wikibugs>	 10serviceops, 10MediaWiki-REST-API, 10Parsoid, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): HTTP 412 Errors when editing Officewiki - https://phabricator.wikimedia.org/T331629 (10daniel) Config used to reproduce the issue locally: `lang=php wfLoadExtension( 'VisualEditor' ); $wgVisualEditorDefaultParsoidCl...