[01:39:57] (VarnishTrafficDrop) firing: 69% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [01:44:57] (VarnishTrafficDrop) resolved: 69% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [04:06:57] (VarnishTrafficDrop) firing: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [04:16:57] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [04:21:56] (VarnishTrafficDrop) firing: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [04:26:56] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [08:42:57] (VarnishTrafficDrop) firing: 55% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [08:57:57] (VarnishTrafficDrop) firing: (2) 40% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:07:57] (VarnishTrafficDrop) firing: (2) 42% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:09:52] folks I've seen the above alert firing a lot recently, every time it seems a false alert (increase + decrease in traffic, different from the avg) but I am wondering if we need to tune it [09:10:18] (I am worried that we stop checking it and we miss something important) [09:11:50] bblack: could you take a look at that, as you're working on the alerting? :) [09:12:57] (VarnishTrafficDrop) resolved: (2) 60% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [13:18:43] 10Traffic, 10SRE, 10vm-requests: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) Ready to create Ganeti VM durum4001.ulsfo.wmnet in the ganeti01.svc.ulsfo.wmnet cluster on row 1 with 2 vCPUs, 4GB of RAM, 15GB of disk in the private network. Ready to create Ganeti VM... [13:26:17] bblack: for https://gerrit.wikimedia.org/r/c/operations/puppet/+/721062, do i have to do anything other than just puppet-merge ? [13:29:17] ema: ^ maybe knows/ [13:30:31] ottomata: just puppet-merge is enough! [13:30:40] ok great, ty, proceeding! [13:48:56] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) @ssingh durum4001 and durum4002 are ready now as well. [13:49:40] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) [13:51:43] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10ssingh) >>! In T290672#7355397, @Dzahn wrote: > @ssingh durum4001 and durum4002 are ready now as well. Thank you @Dzahn; very grateful for all the help here! [14:47:16] 10Traffic, 10Analytics, 10Data-Engineering, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10nettrom_WMF) >>! In T288853#7352497, @Ottomata wrote: > The -1 is not for the idea, but for a specific imple... [16:21:22] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) Ready to create Ganeti VM durum5001.eqsin.wmnet in the ganeti01.svc.eqsin.wmnet cluster on row 1 with 2 vCPUs, 4GB of RAM, 15GB of disk in the private network. Read... [17:10:52] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) All done. You can add all hosts to site.pp now. [17:11:34] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10Dzahn) 05Open→03Resolved [17:46:57] (VarnishTrafficDrop) firing: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:51:57] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:54:56] (VarnishTrafficDrop) firing: 60% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:59:56] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [18:00:55] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create Ganeti VMs for durum - https://phabricator.wikimedia.org/T290672 (10ssingh) Thank you, again, for the help and for getting this done so quickly. [20:18:43] 10Traffic, 10MW-on-K8s, 10Performance-Team, 10SRE, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki) [20:19:42] 10Traffic, 10MW-on-K8s, 10Performance-Team, 10SRE, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10jijiki)