[04:34:17] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling) It looks like pywikibot is a decent test rig for session replication races. I installed pywikibot on a Linode instance in the Dallas region, which... [06:24:47] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10ori) We need to decide if we want to make this change, taking into consideration the fact that the resource savings (in dollar te... [07:16:54] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Joe) >>! In T211661#8139622, @ori wrote: > As several people have pointed out in conversation with me, the fact that we're stori... [08:49:19] <_joe_> hello traffic people, I prepared two patches for moving pybal etcd connections in relations to the PDU upgrades: https://gerrit.wikimedia.org/r/c/operations/puppet/+/821341 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/821602 [08:49:36] <_joe_> can I ask you to merge/restart pybal where needed? [08:52:24] _joe_: will do [08:52:36] should I merge them as well? [08:52:36] <_joe_> uh you're back? [08:52:39] yep [08:52:44] I was OoO till yesterday [08:52:45] <_joe_> vgutierrez: please do at your convenience [08:52:56] <_joe_> yeah I thought you were out the whole week too :) [10:08:16] (VarnishTrafficDrop) firing: Varnish traffic in eqsin has dropped 68.74373768737728% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [10:08:56] (HAProxyEdgeTrafficDrop) firing: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [10:11:50] 10netops, 10Cloud Services Proposals, 10Infrastructure-Foundations: Separate WMCS control and management plane traffic - https://phabricator.wikimedia.org/T314847 (10cmooney) p:05Triage→03Low [10:13:16] (VarnishTrafficDrop) resolved: Varnish traffic in eqsin has dropped 68.49699995727907% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [10:13:56] (HAProxyEdgeTrafficDrop) resolved: 68% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [13:51:52] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10MatthewVernon) While Disk Is Cheap (TM), container listing is not and our thumbs containers are the largest in terms of number-of... [14:58:18] 10Traffic, 10MediaWiki-General, 10SRE: Roll out query parameter normalization - https://phabricator.wikimedia.org/T314868 (10ori) [15:00:23] 10Traffic, 10MediaWiki-General, 10SRE: Roll out query parameter normalization - https://phabricator.wikimedia.org/T314868 (10ori) 05Open→03In progress [16:22:56] (HAProxyEdgeTrafficDrop) firing: (4) 26% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [16:27:56] (HAProxyEdgeTrafficDrop) resolved: (5) 65% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [16:41:21] 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Platonides) SQLite? I'm surprised it uses that, it may not be the best performant option for such a queue... [18:17:02] 10Traffic, 10MediaWiki-General, 10SRE: Roll out query parameter normalization - https://phabricator.wikimedia.org/T314868 (10ori) [18:32:56] (HAProxyEdgeTrafficDrop) firing: 47% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [18:42:56] (HAProxyEdgeTrafficDrop) resolved: 58% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:12:56] (HAProxyEdgeTrafficDrop) firing: (2) 62% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:13:16] (VarnishTrafficDrop) firing: Varnish traffic in esams has dropped 69.3914529772412% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [19:17:56] (HAProxyEdgeTrafficDrop) resolved: (2) 61% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:18:16] (VarnishTrafficDrop) resolved: Varnish traffic in esams has dropped 68.5717087644373% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop