[00:57:46] 🎉 🥂 [01:07:30] $ curl -si --connect-to ::text-lb.codfw.wikimedia.org "https://www.mediawiki.org/?$(date +%s)" | grep server: [01:07:32] server: mw2365.codfw.wmnet [01:07:35] wooo [01:22:51] yeah, codfw is serving 2 or 3 req/s now, in addition to 25 req/s monitoring traffic [01:26:23] easiest to see with the time set to 1 day: https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red?orgId=1&var-site=codfw&var-cluster=appserver&var-method=GET&var-code=200&var-php_version=proxy:unix:%2Frun%2Fphp%2Ffpm-www.sock.*&from=now-24h&to=now&viewPanel=17 [01:55:28] congrats to AaronSchulz and Krinkle as well, it's been a long road and this is a nice milestone [10:28:01] Amir1: do you recall what process you used to generate the wikitechwiki.ico file? (the last change to a prod favicon) I can't seem to find any doc pages or scripts for it. [12:01:47] Krinkle: I have no memory of iT. Probably just used a standard Linux tool [14:37:59] I'm looking at writes in x2, in half a day since the incident, it had, 1.1M writes, 650K writes are just RL, 360K are captcha. [14:38:06] Are these numbers normal? [14:41:09] https://grafana.wikimedia.org/d/000000174/redis?orgId=1&from=now-6M&to=now&var-datasource=eqiad%20prometheus%2Fops&var-job=redis_sessions&var-instance=All [14:41:26] There were previously 30K - 10K remaining = 20K/s operations in Redis [14:42:01] of whch about 3K/s "set" [14:42:42] I would compare to the writes we had when we first deployed x2 [14:43:08] https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-site=eqiad&var-group=core&var-shard=x2&var-role=All&from=now-2d&to=now [14:43:10] where is the spike? [14:47:02] I'm not saying there is a spike, I was wondering if it could be optimized further. [14:47:31] I know there is a good reason for it and forgive me if I'm not understanding this well but why not going with WAN? [14:49:20] 3k/s writes is basically as much as a small core section. [14:50:02] sorry but I can't prioritize this right now, we've already spent over a year on this. There is a long list of other things to do before this is worth more time. Information exists within https://phabricator.wikimedia.org/T113916 https://www.mediawiki.org/wiki/ResourceLoader/Architecture and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/813120/ [14:50:21] Tim has proposed putting it on tmpfs if performance is an issue [14:51:05] it's not performance [14:51:13] thanks. I go read those [14:51:20] once multi-dc and php7.4 and php8.1 and save timing and pageview timing are in better state, we can look back and see what seems worth it in retrospect bigger picture to optimise further. Otherwise we may end up optimising something we will remove again.