[06:27:40] <jinxer-wm>	 FIRING: VarnishHighThreadCount: Varnish's thread count on cp5018:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5018 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[06:47:40] <jinxer-wm>	 RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp5018:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5018 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[07:08:25] <wikibugs>	 06Traffic, 06Abstract Wikipedia team: Wikifunctions is down - https://phabricator.wikimedia.org/T374318#10431401 (10RhinosF1) >>! In T374318#10350479, @Feeglgeef wrote: >>>! In T374318#10162134, @DVrandecic wrote: >> I wonder if we can raise that ban again? I think the crawler in combination with T374241 w...
[07:10:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[07:30:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[09:30:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[09:35:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[12:36:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[12:46:09] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[12:52:09] <jinxer-wm>	 FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[13:01:24] <jinxer-wm>	 RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX
[15:27:41] <wikibugs>	 06Traffic, 06SRE: Reveal IP after click only on Varnish error pages - https://phabricator.wikimedia.org/T383062 (10Diskdance) 03NEW
[17:14:49] <wikibugs>	 10Wikimedia-Apache-configuration, 06Infrastructure-Foundations, 06Security-Team, 07SecTeam-Processed, and 2 others: https://www.mediawiki.org/.well-known/change-password redirects to HTTP - https://phabricator.wikimedia.org/T381625#10433283 (10sbassett)
[17:32:22] <wikibugs>	 06Traffic, 06Abstract Wikipedia team: Wikifunctions is down - https://phabricator.wikimedia.org/T374318#10433429 (10Jdforrester-WMF) >>! In T374318#10350479, @Feeglgeef wrote: > My bot (that runs the featured tool from the newsletter) that continuously tries to locally update every Z2 all the time, and has...
[17:35:35] <wikibugs>	 06Traffic, 06SRE: Reveal IP after click only on Varnish error pages - https://phabricator.wikimedia.org/T383062#10433442 (10Lucas_Werkmeister_WMDE) > For non-JavaScript fallback, we can just choose to show or hide the IP completely (Cloudflare chooses the latter).  A [<select> element](https://developer.mozill...
[19:15:19] <wikibugs>	 06Traffic, 06SRE: Reveal IP after click only on Varnish error pages - https://phabricator.wikimedia.org/T383062#10433802 (10ssingh) p:05Triage→03Medium
[20:40:34] <wikibugs>	 06Traffic, 06Data-Engineering, 06Data-Platform: GeoDNS: Pipeline from event.development_network_probe to operations/dns.git - https://phabricator.wikimedia.org/T380626#10434144 (10VirginiaPoundstone)
[21:13:56] <wikibugs>	 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, and 2 others: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#10434459 (10VirginiaPoundstone)
[21:16:13] <wikibugs>	 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 06SRE: Add a rolled-up cache_status field to druid webrequest_sampled_128 - https://phabricator.wikimedia.org/T319344#10434497 (10VirginiaPoundstone)
[21:17:10] <wikibugs>	 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, and 2 others: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10434511 (10VirginiaPoundstone)
[21:19:40] <wikibugs>	 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 06Movement-Insights: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463#10434547 (10VirginiaPoundstone)
[21:23:04] <wikibugs>	 06Traffic, 06Data-Engineering-Icebox, 06SRE: Webrequest x_analtics `wprov` value is incorrectly formatted - https://phabricator.wikimedia.org/T339910#10434633 (10VirginiaPoundstone)
[21:23:52] <wikibugs>	 06Traffic, 06Data-Engineering-Icebox, 06SRE, 06Trust and Safety Product Team, and 2 others: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947#10434654 (10VirginiaPoundstone)
[21:43:42] <wikibugs>	 06Traffic, 06Experimentation Lab, 10Experimentation Lab Radar: Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#10435272 (10VirginiaPoundstone)
[21:46:52] <wikibugs>	 06Traffic, 06SRE, 13Patch-For-Review: Reveal IP after click only on Varnish error pages - https://phabricator.wikimedia.org/T383062#10435370 (10BCornwall) My proposed patch seems the simplest method: Using basic HTML/CSS we can simply portion off any sensitive info into an expandable box. I added a red backg...