[15:41:31] <Krinkle>	 Amir1: looking at T328872 for perf impact
[15:41:32] <stashbot>	 T328872: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T328872
[15:41:38] <Krinkle>	 https://grafana-rw.wikimedia.org/d/000000559/mediawiki-action-api-breakdown?orgId=1&from=now-60d&to=now&timezone=utc&var-module=upload&var-query=&var-Percentile=0.5
[15:41:59] <Krinkle>	 Looks like Sep 11-12 is when a big improvement happened, since then, less clear.
[15:42:14] <Krinkle>	 the patches rolled this month though
[15:43:08] <Krinkle>	 https://sal.toolforge.org/production?p=0&q=s4&d=2025-09-12
[15:43:12] <Krinkle>	 s4 primary switch perhaps?
[15:45:04] <Krinkle>	 The improvement starts sudden on 01:00 AM UTC sharp on Sept 11
[15:45:05] <Krinkle>	 https://grafana-rw.wikimedia.org/d/000000559/mediawiki-action-api-breakdown?orgId=1&from=2025-09-10T20:13:52.850Z&to=2025-09-11T08:32:09.613Z&timezone=utc&var-module=upload&var-query=&var-Percentile=0.5
[15:45:32] <Krinkle>	 https://sal.toolforge.org/production?p=5&q=&d=2025-09-11
[15:45:54] <Krinkle>	 00:47 … Repooling after maintenance db2240 (T402763)', diff saved to https://phabricator.wikimedia.org/P83223
[15:45:54] <Krinkle>	 00:45 … finished single-replica PHP 8.3 pilot on shellbox-constraints - T403284
[15:45:54] <stashbot>	 T402763: Drop rc_new from recentchanges table in wmf production - https://phabricator.wikimedia.org/T402763
[15:45:54] <stashbot>	 T403284: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284
[15:46:16] <Amir1>	 I doubt they had any effect
[15:46:26] <Amir1>	 for my patch, I saw it in flamegraph
[15:46:52] <Krinkle>	 upload rate raised at that time
[15:47:04] <Krinkle>	 I guess a persistnet upload bot started around then with relaivelyh small files that upload quickly
[15:47:07] <Krinkle>	 and has continued to this day
[15:47:22] <Krinkle>	 nearly trippled
[15:47:36] <Krinkle>	 so nothing got faster there, just more fast requests were added
[15:48:14] <Amir1>	 https://performance.wikimedia.org/arclamp/svgs/daily/2025-10-23.excimer-wall.all.fn-Upload.reversed.svgz vs https://performance.wikimedia.org/arclamp/svgs/daily/2025-09-23.excimer-wall.all.fn-Upload.reversed.svgz
[15:48:48] <Amir1>	 consistency check was 20% of the time in all upload operations
[15:50:16] <Krinkle>	 https://performance.wikimedia.org/arclamp/svgs/daily/2025-09-23.excimer-wall.all.fn-Upload.svgz?s=consist
[15:50:18] <Krinkle>	 ack, nice
[15:50:42] <Krinkle>	 I saw that pre-merge as well, you did an excimer profile in the commit message
[15:50:58] <Amir1>	 yeah
[15:51:00] <Krinkle>	 but was hoping to see something absolute afterward, but harder to see
[15:51:33] <Amir1>	 I was hoping to see less GET on swift side but there are so many read GETs that the hange got drowned 
[15:53:08] <Amir1>	 actually maybe it can show itself in HEAD?
[15:59:30] <Amir1>	 nope, it doesn't do HEAD
[16:04:22] <Krinkle>	 duesen: hehe, you're welcome xD - RE: T404739
[16:04:22] <stashbot>	 T404739: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739