[17:46:50] Reedy: a log! [17:46:51] https://www.irccloud.com/pastebin/SCKmNWMl/ [17:58:58] https://www.irccloud.com/pastebin/VrbaDEPM/ [18:02:06] what is php's memory_limit? [18:02:14] how much memory does the server have? [18:05:02] moonmoon: config is https://www.irccloud.com/pastebin/oXfYRBBj [18:05:54] Ram total on grafana is 3.87 GB [18:06:05] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw8.miraheze.org&var-port=9100&viewPanel=78 [18:07:11] now I'm no expert here. But you have 32 PHP processes, each of which can take up to 0.5GB of RAM, for a total of 16GB consumed max. And you're wondering why a server with under 4GB of RAM is OOMing? [18:08:13] Hmm [18:09:24] granted the wiki won't consume nearly that much in normal operation sans memory leaks; I've personally only seen memory usage reach those levels when doing things like transcodes [18:09:46] moonmoon: well the working theory is a memory leak somehwere [18:10:03] but you're set up for failure even with everything operating normally [18:10:13] are you sure it's a leak or is it just object caches getting fuller? [18:10:24] I'm not sure on anything [18:11:03] But memory increases to a point everything crashes [18:18:53] Hi! When I use the visual editor in own MediaWiki on XAMPP, I received this error: Error contacting the Parsoid/RESTBase server (HTTP 500). I looked at the google search, but I could not to solve the problem. [18:19:13] RhinosF1: have you had any success with enabling any sort of profiling extension in PHP itself to see what memory is being allocated to over time? [18:19:22] that's probably a good next step if not [18:22:11] moonmoon: well I don't know where to store the files [18:22:15] Or output [18:22:30] Because we get like 10req/s [18:22:46] that'll vary by what you're trying to use, but typically you use sampling [18:22:53] That's a lot of debug files [18:22:54] (that is, it isn't run on every request) [18:23:19] at 10 req/s, if your sample rate is 1%, that leads to one sample every 10s on average [18:23:28] True [18:23:37] which may even be too frequent for what you need [18:23:48] But how do we find which file is useful [18:24:27] We need it to sample the worst ones [18:25:01] not sure which profiler you're using so I can't help much there. Chances are I'm not using the same one for my sites [18:25:29] Either tideways or excimer [18:25:49] if the configuration is code-based you can always monitor for some threshold of usage and only start logging once the threshold is exceeded [18:31:01] moonmoon: well I assume I just stick it with the current mediawiki config but extend it to an extra server [18:31:25] And log it rather than html output [18:34:49] hmm, yeah. I wouldn't expose APM details in html to end users [18:55:26] I solved my error with https://www.mediawiki.org/w/index.php?title=Topic:Wbcdqrzfrumju8t1&topic_showPostId=wbcqlv6eg8tcozp9&fromnotif=1#flow-post-wbcqlv6eg8tcozp9 [18:55:34] I am so happy. [19:02:44] moonmoon: it's only on test and with a url parameter normally [19:12:02] moonmoon: I'm pretty sure no one thought when making this config [19:12:58] It seems over time various values changed and it's drifted [19:13:27] Which is why the config says it can use 16/4GB [23:03:38] moonmoon: https://github.com/miraheze/mw-config/pull/4273/files does nothing [23:03:49] the directory isn't being populated [23:05:05] Why would you set wmgProfiler? [23:06:29] Reedy: because that's what it does above [23:06:59] ok but what's reading it? [23:07:09] the usual config var read by MW is $wgProfiler [23:07:12] https://github.com/miraheze/mw-config/blob/061ab79212ad2455a8b27a0fc388b5cf3edd7f7e/LocalSettings.php#L19 [23:07:23] moonmoon: it gets magically changed back later [23:08:00] * RhinosF1 is going to change that [23:09:31] just checking that php has write access to that directory? [23:10:03] https://www.irccloud.com/pastebin/p7doyqLs/ [23:10:17] moonmoon: ^ [23:18:18] beats me. Maybe error logs will shed more light on what's going on? [23:18:31] nothing seems obviously wrong about that [23:19:20] not that i can see [23:19:35] RhinosF1: oh, I'm guessing the reason you aren't using $wgProfiler directly is because you run that in an auto_prepend_file and so it gets assigned before DefaultSettings runs and initializes $wgProfiler to its default value of "nothin" [23:19:59] moonmoon: hmm [23:20:04] very likely actually [23:20:21] i did try with $wg directly and that made no difference though [23:20:41] CosmicAlpha: can you look while i sleep