[01:20:06] 10serviceops, 10Data-Engineering, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) [01:20:19] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream: k8s deployment-charts mesh module should allow use of mesh without public_port Service - https://phabricator.wikimedia.org/T326252 (10Ottomata) 05Open→03Resolved Ya dunno why we didn't! [08:23:04] 10serviceops, 10MediaWiki-Internationalization, 10MediaWiki-extensions-General, 10WMF-General-or-Unknown, 10Wikimedia Foundation Governance Wiki (foundation.wikimedia.org): Update footer links to direct to proper locations on Foundation Governance Wiki - https://phabricator.wikimedia.org/T331680 (10Varnen... [08:35:03] 10serviceops, 10MW-on-K8s: Backport mediawiki chart mesh changes - https://phabricator.wikimedia.org/T338210 (10Clement_Goubert) [08:35:25] 10serviceops, 10MW-on-K8s: Backport mediawiki chart mesh changes - https://phabricator.wikimedia.org/T338210 (10Clement_Goubert) p:05Triage→03Medium a:03Clement_Goubert [09:16:03] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: httpbb fails requesting mw-on-k8s during deployments - https://phabricator.wikimedia.org/T331609 (10Clement_Goubert) [09:16:17] 10serviceops, 10MW-on-K8s: Allow setting draining strategy and time in envoy image - https://phabricator.wikimedia.org/T338014 (10Clement_Goubert) 05In progress→03Resolved Setting the following environment variables now controls envoy's draining behaviour: `DRAIN_TIME_S` `DRAIN_STRATEGY` In case a non-def... [09:44:09] 10serviceops, 10MediaWiki-Internationalization, 10MediaWiki-extensions-General, 10WMF-General-or-Unknown, and 2 others: Update footer links to direct to proper locations on Foundation Governance Wiki - https://phabricator.wikimedia.org/T331680 (10Varnent) I think https://gerrit.wikimedia.org/r/927601 shoul... [09:49:10] 10serviceops, 10Sustainability (Incident Followup): Page only if videoscalers are unavailable for longer than the default time - https://phabricator.wikimedia.org/T338220 (10fgiunchedi) [10:25:15] mszabo: I've been playing with this: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/926455/1/images/envoy/drain-envoy.sh [10:28:19] Currently it doesn´t really do what I want (even with 2 replicas, my requests don't switch to the other available pod) [10:32:37] I think there's a timing issue, when doing a rolling restart I get a bunch of connection refused, I think we're not waiting long enough for new pods to be up maybe [11:16:01] 10serviceops, 10GitLab, 10Release-Engineering-Team: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10kostajh) [11:24:26] 10serviceops, 10GitLab, 10Release-Engineering-Team: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10Joe) Specifically, what we want is that image tags are obviously sortable in chronological o... [11:32:55] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Yann) 429 responses sky rocketed this morning: https://grafana.wikimedia.org/d/FMKakEyVz/hnowlan-thumbor-k8s?orgId=1&viewPanel=61&... [11:32:59] 10serviceops, 10GitLab, 10Release-Engineering-Team, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) kharlan opened https://gitlab.wikimedia.org/repos/media... [11:33:09] 10serviceops, 10GitLab, 10Release-Engineering-Team, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) [11:46:56] 10serviceops, 10GitLab, 10Release-Engineering-Team, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10kostajh) I think we could do something like `export CI_IMAGE_PUBLISH_D... [12:40:25] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Xover) >>! In T337649#8905727, @Yann wrote: > 429 responses sky rocketed this morning: More 429 responses than 200 responses in... [13:41:02] <_joe_> effie: ^^ [13:41:13] k [14:12:43] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10akosiaris) >>! In T333318#8900126, @AntiCompositeNumber wrote: > is there a specific reason the tile viewer at https://maps.wikimedia.org/ was bro... [14:15:42] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10TheDJ) >>! In T333318#8900129, @GeneralNotability wrote: > Hey y'all, I was pointed to this task by @stjn. These changes unexpectedly broke my too... [15:54:02] 10serviceops, 10GitLab, 10Patch-For-Review, 10Release-Engineering-Team (They Live 🕶️🧟): Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10thcipriani) [16:06:21] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Yann) One more failed thumbnail https://commons.wikimedia.org/wiki/File:The_Encyclopaedic_Dictionary_(cu3192... [17:17:03] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10jijiki) We bumped the number of concurrent renderings, but so far I wouldnt say we are doing great, we will pick this up later or... [17:24:28] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10jijiki) >>! In T333318#8900129, @GeneralNotability wrote: > Hey y'all, I was pointed to this task by @stjn. These changes unexpectedly broke my to... [17:26:49] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10stjn) >>! In T333318#8906960, @jijiki wrote: > This is on #serviceops and we are terribly sorry for that. We have been trying to come up with ways... [17:37:29] 10serviceops, 10MediaWiki-Internationalization, 10MediaWiki-extensions-General, 10WMF-General-or-Unknown, and 2 others: Update footer links to direct to proper locations on Foundation Governance Wiki - https://phabricator.wikimedia.org/T331680 (10Jdforrester-WMF) >>! In T331680#8905484, @Varnent wrote: > I... [18:12:16] 10serviceops, 10Abstract Wikipedia team (Phase θ – Throttling), 10Patch-For-Review: Kubernetes Wikifunctions security and control measures - https://phabricator.wikimedia.org/T326785 (10Jdforrester-WMF) a:03cmassaro [18:36:26] 10serviceops, 10MediaWiki-Internationalization, 10MediaWiki-extensions-General, 10WMF-General-or-Unknown, and 2 others: Update footer links to direct to proper locations on Foundation Governance Wiki - https://phabricator.wikimedia.org/T331680 (10Varnent) >>! In T331680#8906992, @Jdforrester-WMF wrote: >>>... [18:37:12] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Ladsgroup) I wonder if it's possible to setup a dedicated poolcounter instance for thumbor (temporary) to be able to properly look... [18:41:14] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Xover) >>! In T337649#8906929, @jijiki wrote: > We bumped the number of concurrent renderings, but so far I wouldnt say we are doi... [18:57:08] 10serviceops, 10All-and-every-Wikisource, 10Thumbor: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Ladsgroup) Throttling haven't improved (in the past six hours): {F37095713} Mostly guessing/thinking out loud. All of expensive t... [19:08:35] Amir1: re https://phabricator.wikimedia.org/T337649 , I think you might be onto something. I'm dead now, but I'll try to help tomorrow. Thank you for looking into it! [19:08:54] oh no worries [19:08:59] Go get rest! [19:09:31] yeah :-) just wanted to say I really appreciate your help [19:09:46] <3 I wish I could do more [19:12:16] I think we all do when it comes to thumbor :D your idea is really helpful, with your pointer I can dig into it, so it's great <3 [20:52:26] I found this lol https://www.mediawiki.org/w/index.php?title=PoolCounter&diff=prev&oldid=2863307 [21:11:17] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10jijiki) Looking at a tcpdump, it appears that we are able to acquire a lock, ` 20:27:38.234550 IP 10.67.1... [21:35:27] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10jijiki) >>! In T333318#8906966, @stjn wrote: >>>! In T333318#8906960, @jijiki wrote: >> This is on #serviceops and we are terribly sorry for that....