[00:23:57] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530854 (10Scott_French) One additional point of interest: If we zoom out and look at //all// php-fpm request timeouts logged by the 8...
[01:05:25] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10530980 (10ssingh) On the Varnish side of things, I got some clarity from bblack. Specifically that `first_byte_timeout` only applies...
[07:04:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[07:09:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[07:45:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[07:50:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[08:42:42] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10531262 (10daniel) > This suggests we're getting stuck in some long-running C function, perhaps?  A shot in the dark, given DT's heavy...
[11:25:02] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10531509 (10mszabo) >>! In T385395#10531262, @daniel wrote: >> This suggests we're getting stuck in some long-running C function, perha...
[11:28:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[11:33:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[14:36:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[14:41:43] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10531858 (10Daimona) >>! In T385395#10529425, @Scott_French wrote: > These seem to indicate a large number of very slow (> 1000s) discu...
[14:44:06] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10531865 (10mszabo) I also took a glance at local and global AFs using `rlike` in case there might be a problematic regex there. I thin...
[14:45:17] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10531866 (10mszabo) >>! In T385395#10531858, @Daimona wrote: > How exactly are y'all reproducing this? Would trying to edit https://uk....
[14:51:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[15:31:17] <wikibugs>	 06Traffic, 06Content-Transform-Team-WIP, 06Data-Persistence, 10iOS-app-feature-Performance, and 7 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365#10532014 (10Jgiannelos) 05Open→03Resolved Closing this one for now since we have a comp...
[15:43:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[15:53:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[15:58:55] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10532058 (10Jhancock.wm)
[15:59:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[16:14:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[16:26:10] <wikibugs>	 06Traffic, 06SRE: create a puppetized abstraction for haproxy blocklist hysteresis - https://phabricator.wikimedia.org/T329331#10532148 (10CDanis) 05Open→03Declined Instead we accomplished this via requestctl. {T371144}
[18:27:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[18:37:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[19:13:46] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10532577 (10Daimona) I wanted to reproduce this locally. So I went over to https://uk.wikipedia.org/wiki/Special:Export, added the titl...
[19:51:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[20:06:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[20:29:31] <greg-g>	 Friday afternoon informal request: what is the process for procuring a new (sub)domain? We in FR have a need for one for our SMS sends (see https://phabricator.wikimedia.org/T379318 ). Should I just file a basic info task and let ya'll ask me questions or is there a form/process I should follow? Sorry, I couldn't find anything in my initial
[20:29:31] <greg-g>	 searches on mw.org/wikitech/etc
[20:35:00] <sukhe>	 greg-g: so are we looking for the .wiki gTLD or just a general domain?
[20:35:10] <sukhe>	 because the former is a bit more complicated
[20:35:25] <greg-g>	 I think folks requested it because of the shortness
[20:35:39] <greg-g>	 but I am open to other solutions if appropriate
[20:35:51] <sukhe>	 there is some history in https://phabricator.wikimedia.org/T88873 that summarizes it better than I can here
[20:36:53] <sukhe>	 in general I think our (Traffic's) position has been to not add more subdomains to .wiki
[20:37:08] <sukhe>	 but: 
[20:37:35] <greg-g>	 fair!
[20:37:43] <sukhe>	 a) if it is a generic new domain, the process is to get Legal to register it and then Traffic can take care of the DNS side of things
[20:37:55] <sukhe>	 b) for .wiki, I can check with the team
[20:38:04] <greg-g>	 gotcha
[20:38:15] <greg-g>	 that makes sense.
[20:38:19] <sukhe>	 I don't want to give you what might just be my (possibly outdated) opinion on the topic
[20:38:42] <sukhe>	 bblack's comment is from 2016 but IIRC in my last conversation with him, we discussed about the same thing https://phabricator.wikimedia.org/T88873#2651081
[20:38:57] <greg-g>	 yeah
[20:39:19] <sukhe>	 we are meeting in person next week and I can follow up then on the task
[20:39:30] <greg-g>	 again, I'm ok with it not being a dotwiki, it was just the FR team's first thought given the length and that these will be used in SMS messages
[20:39:38] <sukhe>	 yeah that's fine and I think a fair ask in that respect
[20:39:40] <greg-g>	 oh right, forgot about the SRE offsite next week
[20:40:18] <sukhe>	 but yeah, I can follow up on that later but if not, a generic domain is a fairly easy undertaking. you can email Chuck at Legal and CC sre-traffic basically.
[20:40:43] <greg-g>	 I'll let the team know the basic process (thank you!) and if you have any suggestions I think we're open to it :)
[20:40:52] <greg-g>	 I'll do that email to Legal/chuck soon
[20:40:56] <greg-g>	 thanks much sukhe !
[20:42:33] <sukhe>	 greg-g: thanks, I have made a note and will comment on the task (re: using [msr].wiki)
[20:43:41] <greg-g>	 oh, perfect, I was just summarizing what you said here :)
[20:51:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[20:52:02] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10532748 (10Daimona) I realized I also needed to install TemplateStyles, so I did that; I also already had Scribunto installed (althoug...
[21:01:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[21:11:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[21:21:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[23:20:25] <wikibugs>	 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533126 (10mszabo) @daniel was right—this is indeed a case of catastrophic backtracking, although not in DT but in AbuseFilter.  We st...