[00:11:00] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10thcipriani) noting {T296098} as possibly related [04:32:19] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10RLazarus) Recap from today: - The OOMs originally flagged in this ticket were all on wtp1025, but that turned out to b... [04:33:11] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10RLazarus) [19:03:21] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) The memory dump of APCu is in my home directory in mw1414, I'm planning to take a look in Monday, t... [19:04:46] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Ladsgroup) Another note: Most of OOMs in appservers seems to be coming from rest.php (is rest.php being served... [19:49:05] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10ssastry) Note that for Parsoid itself, errors ilke this `Allowed memory size of 1468006400 bytes exhausted (tr... [22:55:29] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10daniel) >>! In T297517#7564757, @ssastry wrote: > Are rest.php requests on the app servers to the Parsoid REST... [22:56:51] 10serviceops, 10SRE, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10daniel) By the way, @tstarling has recently been working on optimizing Parsoid. If it's Parsoid related, he ma...