[02:00:22] <ori>	 I noticed that 0.50% of index.php wall-clock time is LuaSandboxCallback::gsub, so I instrumented the PHP code to dump the args into a file, to do some statistical analysis and see what might be worth optimizing. Captured all calls from a single parse of 'Barack Obama' on enwiki -- just shy of 90k in total!
[02:01:13] <ori>	 The most common replacement is the empty string. OK, expected. Second place is a surprise:
[02:01:25] <ori>	 {'1': '1', '0': '0', '3': '3', '2': '2', '5': '5', '4': '4', '7': '7', '6': '6', '9': '9', '8': '8'}
[02:05:41] <ori>	 The search pattern in these cases is always '%d', so I'm guessing it's code to convert numerals from one language to another, which is missing a guard-close to skip if source and target numerals are identical.
[02:05:50] <ori>	 *clause
[02:32:37] <ori>	 ring a bell?
[02:57:35] <ori>	 think I found it
[02:59:17] <ori>	 https://en.wikipedia.org/wiki/Module:Citation/CS1 , line 4653
[02:59:32] <ori>	 k = mw.ustring.gsub (k, '%d', cfg.date_names.local_digits);		-- for enumerated parameters, translate 'local' digits to Western 0-9
[03:33:43] <ori>	 reported: https://en.wikipedia.org/wiki/Help_talk:Citation_Style_1#Redundant_ustring.gsub_calls_for_digits . this was fun
[12:15:18] <Krinkle>	 ori: nice catch!
[12:15:36] <Krinkle>	 how did you capture the arguments? Is that on mwdebug / mw-experimental?
[12:15:42] <Krinkle>	 Is/was
[21:12:38] <ori>	 Krinkle: I live-hacked a logging statement into LuaSandboxCallback:__call on the deployment server and exercised it via benchmarkParse.php quickly before reverting. Which might not be the current best practice.
[21:12:49] <ori>	 But the sanctioned workflows for this kind of debugging were daunting. https://wikitech.wikimedia.org/wiki/Mw-experimental guides you through applying helmfiles and managing systemd timers and shell scripts to control image building. The mw-experimental hosts don't have PHP in the host (outside the container). https://wikitech.wikimedia.org/wiki/Maintenance_scripts was even more
[21:12:51] <ori>	 frightening.
[21:15:23] <ori>	 I don't understand the value of having the debug server run under k8s.
[21:15:49] <Krinkle>	 ref T405688
[21:15:50] <stashbot>	 T405688: Support shell to mw-experimental pod - https://phabricator.wikimedia.org/T405688
[22:37:14] <tgr_>	 the value is not having to maintain separate k8s and baremetal deployment logic, I imagine?
[22:37:32] <tgr_>	 for MediaWiki developers it certainly is negative value
[23:40:12] <Krinkle>	 ori: RE https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219190, feel free to roll that out if you can. assuming that that doesn't break scap in beta, we can then look at enabling it on one o f the two appservers in beta and monitoring opcache/ram use and tweak php.ini settings accordingly there.