[08:37:30] FYI: looks like the short outage of the SUP producer @ codfw this morning (5:30-6:00 UTC) was caused by name resolution issues. The flink app was unable to reach zookeeper. [09:04:39] pfischer: there was a broader incident in k8s this morning, related to too many calico network policies being created IIUC [10:12:51] lunch [13:04:15] inflatador: about T297995, this is really a product decision, and it is difficult to find who owns that. I've been struggling with it for the last > 1 year. I'll try to raise it again and find resolution either way. [13:04:16] T297995: Remove authentication from Wikimedia Commons Query Services (WCQS) - https://phabricator.wikimedia.org/T297995 [13:13:08] o/ [14:24:11] \o [15:18:40] o/ [17:17:24] dinner [17:20:38] Trey314159: forgot to mention, looking for review on https://phabricator.wikimedia.org/T376161 (which is poking at the notebook results on stat1008) [17:40:08] ebernhardson: I will try to look today [17:48:47] tx [18:05:07] * ebernhardson ponders if opensearch-cross-cluster-replication plugin would be a good fit for cloudelastic. There is operational benefit to eqiad/codfw having separate consumers, but CCR for eqiad->cloudelastic might be nice (but private wikis might complicate things) [18:06:57] lunch, back in ~40 [22:11:22] meh, might have to remove blubber from this...opensearch docker image ships as a rpm based system, to decompress the deb packages requires installing stuff, but only builtin bits can use root via blubber [22:14:47] it's also randomly curious the .deb i build has .zst inside it, rather than the .xz it's always had in the past. Apparently .deb supports both, but our script would have to do some detection