[01:17:01] I think I'm for it. The "canonical" format which places the title first is ultimately also more readable and it's nicer to have URLs in this form flowing through the system for debugging. [02:37:10] 10Domains, 10SRE, 10WMF-Communications: Setup URL (soundlogo.wikimedia.org) for Sound Logo website - https://phabricator.wikimedia.org/T314626 (10Varnent) [02:37:35] 10Domains, 10SRE, 10WMF-Communications: Setup URL (soundlogo.wikimedia.org) for Sound Logo website - https://phabricator.wikimedia.org/T314626 (10Varnent) [09:40:34] <_joe_> ori: I agree FWIW [10:13:46] i uploaded a patch for soundlogo [13:55:56] (HAProxyEdgeTrafficDrop) firing: 59% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [13:57:16] (VarnishTrafficDrop) firing: Varnish traffic in eqsin has dropped 67.8666828953081% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [14:01:01] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [14:02:16] (VarnishTrafficDrop) resolved: Varnish traffic in eqsin has dropped 69.50450933279859% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [14:26:39] sukhe: if you are around, mind a review of https://gerrit.wikimedia.org/r/c/operations/dns/+/820667/3? [14:50:56] (HAProxyEdgeTrafficDrop) firing: 64% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [14:53:16] (VarnishTrafficDrop) firing: Varnish traffic in eqsin has dropped 65.4172128390591% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [14:58:16] (VarnishTrafficDrop) resolved: Varnish traffic in eqsin has dropped 67.4063035618256% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [15:00:57] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [15:03:24] sukhe: i copied all the others around [15:03:29] i see what you mean [15:03:34] let me fix to 5M now [15:04:04] or is 1H better [15:04:08] they not consistent [15:04:54] techblog is 5M [15:05:00] the other 3 i see are 1H [15:05:23] yeah I am not sure if there is a preference, I was just checking if there was a reason this is much higher than the others [15:05:33] I think 1H is better since it has the most votes :) [15:05:41] sukhe: no, I do IH then [15:06:07] sukhe: see PS4 [15:06:19] unless of course Automatticc dictates something else [15:06:28] > VIP recommends reducing the TTL of a DNS record to 300 seconds at least a day before [https://docs.wpvip.com/how-tos/check-dns-record-time-to-live/] [15:06:38] not very helpful [15:06:42] RhinosF1: checking [15:10:20] RhinosF1: thanks for the patch [15:10:21] dig soundlogo.wikimedia.org +short [15:10:21] wikimediasoundlogo.go-vip.net. [15:10:56] sukhe: I get ssl error which I assume expected so I guess resolve the task [15:12:46] yes that's expected till they finalize the installation on their end [15:21:48] 10Domains, 10Traffic, 10SRE, 10WMF-Communications, 10Patch-For-Review: Setup URL (soundlogo.wikimedia.org) for Sound Logo website - https://phabricator.wikimedia.org/T314626 (10ssingh) 05Open→03Resolved a:03ssingh ` dig soundlogo.wikimedia.org CNAME +short wikimediasoundlogo.go-vip.net. ` Thanks... [15:53:58] 10Domains, 10Traffic, 10SRE, 10WMF-Communications: Setup URL (soundlogo.wikimedia.org) for Sound Logo website - https://phabricator.wikimedia.org/T314626 (10Varnent) Thank you, @RhinosF1 and @ssingh! :) [16:23:25] I've been watching a few grafana dashboards to see if the Africa merge had much of an impact on lessening esams' load. Sadly, it doesn't look like it made much of a difference [16:28:16] was that the change that you merged yesterday? [16:37:44] Yeah [17:03:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) Note that the generate_dns_snippets.py script might need to be adapted, see the error during a test on netbox-next. ` 2022-08-05 18:55:05,373 [INFO] Gathering de... [17:29:28] brett: a good way to get an idea of that in advance is with turnilo webrequest_sampled_128 [17:29:47] (which can give you an idea of both per-country reqs and egress bytes) [17:31:41] cdanis: netflow [17:32:05] also that, but webrequest gives you good enough data on bytes while also offering http hits [17:32:46] agreed, especially as the change has been merged yesterday [17:33:01] oh, I meant for projecting the impact of such a change [17:34:07] brett: see for instance https://w.wiki/5YKa [17:34:29] you *did* lessen the load that Africa puts on esams, it just was very small already [17:38:54] but in general / in the future -- turnilo is a great tool for doing quick analyses like this. you can split the data a lot of different ways and play with it quickly [17:39:23] https://w.wiki/5YKi for instance :) [17:40:03] interesting that ZA has much more bytes than anything else, on average, even though EG has ~just as many hits [17:40:48] anyway, sorry for butting in, hope that was helpful at least :) happy to chat more if you'd like [17:45:07] cdanis: Thanks for that. I briefly looked at turnilo but not to a useful degree. This is a great example [17:45:33] np! happy to guide you around it some more if you like [17:45:41] webrequest_sampled_128 and wmf_netflow are probably the most useful tables for most SRE purposes [17:45:56] oh, and nowadays ofc the internal netflow one, although I'm not sure how complete that is yet [19:38:53] I see a significant uptick of traffic in drmrs did anything change? there is nothing in SAL [19:39:39] oh looks like it's the same everyday, but more so today, maybe because of the recent changes [19:39:57] see for example https://librenms.wikimedia.org/device/device=239/tab=port/port=23132/ [19:46:40] XioNoX: https://w.wiki/5YM5 :) [19:48:48] cdanis: what is that spike? [19:48:52] not sure [19:48:57] not in webrequest yet ofc [19:49:05] but the added traffic to Africa from drmrs is uh [19:49:07] Turnilo and rates is always a bit messy but that's about... 128*1.5e9 bytes/hour ~= 425 megabit/sec [20:00:26] XioNoX: happened to be looking at something else (average haproxy<>Varnish concurrency by site) and it increases for both esams and drmrs in a shape that approx matches that graph heh :) https://w.wiki/5YM8 [20:00:34] XioNoX: so it is actually an organic traffic spike of some sort [20:00:47] evening spike I guess [20:00:55] just looks weird :) [20:01:04] possibly a news event [20:01:13] or being linked somewhere popular [20:01:19] the evening spike usually isn't *so* sudden [20:15:25] Didn't the africa patch get merged today [20:16:00] Oh no yesterday [20:16:02] https://github.com/wikimedia/operations-dns/commit/97518c5dc5743c6d2b05cadbea2927e6cc1cbb2e