[01:11:10] We may want to oversample navtiming in those regions for a little while and get some before/after numbers [01:15:02] https://diff.wikimedia.org/2018/04/24/new-data-center-singapore/ [01:16:25] Hmm it seems the last Grafana 8 upgrade killed compat for the redirects for symbolic dashboard URLs [01:16:30] https://grafana.wikimedia.org/dashboard/db/performance-singapore-caching-center [01:16:53] https://grafana.wikimedia.org/dashboard/db/navigation-timing [14:33:55] folks I just roll restarted blazegraph on wdqs-public-codfw, the metrics https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&from=now-3h&to=now looks better now but I have not a complete idea about what caused blazegraph to stall in this way. The page look resolved/acked in vops' dashboard, so I am inclined to keep going with my Sunday [14:34:12] Cc: gehel, dcausse, ryankemper --^ [14:35:04] elukey: thanks ! [14:36:24] We have a few ideas of what can cause Blazegraph to crash, but none of them actionable in the short term [14:36:48] It seems to have recovered, so we should be good for now [14:37:32] ryankemper, inflatador : can you write an incident report tomorrow and share with the team ? (I'll be on the plane)