[00:45:27] 06Traffic, 13Patch-For-Review: Move varnish pseudo-headers to vmod_var variables - https://phabricator.wikimedia.org/T373550#11076334 (10BCornwall) [00:57:35] 06Traffic, 13Patch-For-Review: ncmonitor should ignore invalid duplicate MarkMonitor domains - https://phabricator.wikimedia.org/T393734#11076341 (10BCornwall) 05Open→03In progress [02:54:32] 06Traffic, 06SRE, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11076470 (10Midleading) Please be more clear about the UA policy enforced here. I am always setting the `Api-User-Agent` header in my code with my Wikipedia username inside, but... [07:29:21] 06Traffic, 06SRE, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11076853 (10TheDJ) @Midleading you are always supposed to have a user-agent. Api-user-agent is just for situations where you are unable to MODIFY that agent to provide additiona... [08:11:35] 06Traffic: New software: HttpHound - https://phabricator.wikimedia.org/T400244#11076976 (10Fabfur) [08:11:51] 06Traffic: New software: HttpHound - https://phabricator.wikimedia.org/T400244#11076978 (10Fabfur) Renamed to [[ https://gitlab.wikimedia.org/repos/sre/httphound | HttpHound ]] [08:39:24] 07HTTPS, 10wikiba.se website, 10Wikidata: Set HSTS on wikiba.se (force HTTPS) - https://phabricator.wikimedia.org/T232246#11077090 (10Valerie_Wollinger_WMDE) forwarded this, will be looked into shortly [11:42:55] Hey, when one of y'all gets the time, could we get your input on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1171994 ? (context: fractional routing for rest.php centralized routing migration https://phabricator.wikimedia.org/T400131) [12:17:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5019:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5019 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [12:22:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5019:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [12:25:04] 06Traffic: Migrate prometheus-rdkafka-exporter to Gitlab - https://phabricator.wikimedia.org/T400985#11077784 (10Fabfur) a:03BCornwall [12:32:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5019:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [12:42:40] RESOLVED: [3x] VarnishHighThreadCount: Varnish's thread count on cp5019:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [12:49:54] 06Traffic: Include golang-1.19 package in bullseye golang component - https://phabricator.wikimedia.org/T401685 (10Fabfur) 03NEW [13:21:27] 06Traffic: Include golang-1.19 package in bullseye golang component - https://phabricator.wikimedia.org/T401685#11078019 (10Fabfur) 05Open→03Declined Declined. Better upgrade purged to golang-1.23 [13:22:32] 06Traffic: Bump purged go version - https://phabricator.wikimedia.org/T401687 (10Fabfur) 03NEW [13:33:12] 06Traffic, 06Data-Engineering: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11078059 (10Ottomata) [14:03:09] FIRING: LVSHighCPU: The host lvs1016:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1016 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [14:08:09] RESOLVED: LVSHighCPU: The host lvs1016:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs1016 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [14:29:00] 06Traffic, 10MobileFrontend, 06Reader Experience Team: Decide how to configure $wgMobileUrlCallback during mobile domain sunset - https://phabricator.wikimedia.org/T400855#11078312 (10ssingh) >>! In T400855#11074571, @Krinkle wrote: > @ssingh Could you look-over the approach in the task description for anyth... [14:32:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5032:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5032 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:37:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:38:00] FIRING: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [14:42:40] FIRING: [7x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:47:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:48:00] FIRING: [3x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [14:58:00] FIRING: [5x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [15:03:00] RESOLVED: [6x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5027:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [15:32:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:32:45] sigh [15:42:40] FIRING: [10x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:47:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:52:40] FIRING: [10x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:57:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:00:15] 06Traffic: Migrate prometheus-rdkafka-exporter to Gitlab - https://phabricator.wikimedia.org/T400985#11078917 (10BCornwall) 05Open→03Resolved [16:02:40] FIRING: [10x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:07:40] FIRING: [10x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:08:39] (stale alerts, varnish thread count keeps recovering) [16:12:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:17:40] FIRING: [10x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:27:40] FIRING: [6x] VarnishHighThreadCount: Varnish's thread count on cp5027:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:32:40] RESOLVED: [4x] VarnishHighThreadCount: Varnish's thread count on cp5027:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:36:32] 06Traffic, 10MobileFrontend, 06Reader Experience Team: Decide how to configure $wgMobileUrlCallback during mobile domain sunset - https://phabricator.wikimedia.org/T400855#11079175 (10BBlack) @Krinkle - Diving into the specifics a bit (I think all of this will be clearer if you make a prototype VCL patch for... [18:42:30] Hey there traffic folks.... don't suppose anybody is around? [18:42:37] hello :) [18:42:54] I just added a bunch of new vlans / subnets in Netbox and realised I need to add the reverse DNS INCLUDEs for same now [18:42:57] hey :) [18:43:26] it's 1000% my own fault I'm left doing this after not having finished the solution. karma with the slap right to my face [18:43:52] it's all good -- how can we help [18:55:02] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11079827 (10ssingh) Hi folks: Thanks for confirming the extent of the changes from fr-tech's side. We discussed this a b... [19:15:56] sukhe: if you could give me a quick review on this that would be great [19:15:56] https://gerrit.wikimedia.org/r/c/operations/dns/+/1178065 [19:18:56] topranks: looking shortly! [19:27:32] topranks: looks good, +1! [19:27:47] thanks :) [19:30:03] authdns-update done and working [19:30:07] cmooney@cumin1003:~$ dig +short -x 10.192.44.1 [19:30:07] irb0-2053.lsw1-e2-codfw.codfw.wmnet. [19:30:10] thanks!! [19:34:37] nicely done! [19:52:17] 06Traffic, 10MobileFrontend, 06Reader Experience Team: Decide how to configure $wgMobileUrlCallback during mobile domain sunset - https://phabricator.wikimedia.org/T400855#11080062 (10Krinkle) For this task, I mainly want to review the purge strategy. Today, the MobileFrontend extension uses a single mechani... [21:16:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.30:9200 @ cirrussearch1122 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=elasticsearch - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [21:21:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.30:9200 @ cirrussearch1122 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=elasticsearch - https://alerts.wikimedia.org/?q=alertname%3DFermMSS