[01:20:17] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Krinkle) I was asked by @jijiki to take a look. >>! In T337649#8901152, @Soda wrote: >>>! In T337649#890078... [01:34:54] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Soda) >>! In T337649#8908246, @Krinkle wrote: > I was asked by @jijiki to take a look. > >>>! In T337649#89... [04:40:43] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Joe) >>! In T337649#8907263, @Ladsgroup wrote: > Throttling haven't improved (in the past six hours): > {F37... [04:41:58] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Joe) As a side note - I would urge all community members commenting here to try to stick to their experience... [05:39:50] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Joe) I should add - we do have a correctly written python poolcounter client, but its code uses blocking i/o... [08:26:13] 10serviceops, 10Thumbor: Revisit thumbor's poolcounter integration - https://phabricator.wikimedia.org/T338297 (10Joe) [10:31:44] 10serviceops, 10Maps, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Improve tile storage for maps.wikimedia.org - https://phabricator.wikimedia.org/T333318 (10Pikne) >>! In T333318#8906256, @akosiaris wrote: >>>! In T333318#8900126, @AntiCompositeNumber wrote: >> is there a specific reason the tile viewe... [10:49:57] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Gracefully handle pod termination in mw-on-k8s - https://phabricator.wikimedia.org/T331609 (10Clement_Goubert) a:03Clement_Goubert [10:51:40] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Gracefully handle pod termination in mw-on-k8s - https://phabricator.wikimedia.org/T331609 (10Clement_Goubert) Currently testing the following timings: - `envoy`: sleep 2s then drain with a 5s timeout, wait for incoming connections to reach 0 then SIGTERM - `php... [10:52:09] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10jijiki) For the time being we have set `release_timeout: 59`, which maximum time a lock on poolcounter will... [10:54:02] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Gracefully handle pod termination in mw-on-k8s - https://phabricator.wikimedia.org/T331609 (10BanMeAlreadyLMAO) BanMeAlreadyLMAO. I'm aka KEMONO_PANTSU.BanMeAlreadyLMAO. I'm aka KEMONO_PANTSU.BanMeAlreadyLMAO. I'm aka KEMONO_PANTSU. [11:02:34] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Ladsgroup) >>! In T337649#8908246, @Krinkle wrote: >>>! In T337649#8907263, @Ladsgroup wrote: >> All of expe... [12:25:23] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Krinkle) >>! In T337649#8909019, @Ladsgroup wrote: >>>! In T337649#8908246, @Krinkle wrote: >> […] [PoolCoun... [12:29:41] 10serviceops, 10All-and-every-Wikisource, 10Thumbor, 10Patch-For-Review: Thumbor fails to render thumbnails of djvu/tiff/pdf files quite often in eqiad - https://phabricator.wikimedia.org/T337649 (10Ladsgroup) Hmm, maybe because of envoy or some other networking magic in k8s, the connections are getting re... [14:40:41] 10serviceops, 10Wikimedia Enterprise, 10Performance-Team (Radar), 10Wikimedia-Hackathon-2023, 10affects-Kiwix-and-openZIM: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866 (10TheDJ) [15:37:08] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10Clement_Goubert) Following this deployment and backlog times growing, @Ladsgroup added a specific lane for `parsoidCachePrewarm` in changeprop-jobqueue https://... [15:40:21] eoghan: jelto: so yeah usually I am quite calm but sometime I get upset for irrational reasons :-] [15:40:44] anyway the operations for contint* hosts get clarified which is great [15:41:50] for the releases hosts switch over, they run Jenkins which config is entirely in https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/ and is scap deployed. We might need to change a hostname there. jnuche (he is in Europe) can help on that front :]