[00:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:34:56] 10Tool-global-search: cURL error 6 - https://phabricator.wikimedia.org/T358415#9574131 (10Varnent) [02:35:10] 10Tool-global-search: cURL error 6 when using Global Search tool - https://phabricator.wikimedia.org/T358415#9574141 (10Varnent) [02:36:20] 10Tool-global-search: cURL error 6 when using Global Search tool - https://phabricator.wikimedia.org/T358415#9574131 (10Varnent) [02:37:05] 10Tool-global-search: cURL error 6 when using Global Search tool - https://phabricator.wikimedia.org/T358415#9574148 (10Izno) [02:37:57] 10Tool-global-search: Global Search is down: 500: Internal Server Error / Could not resolve host: cloudelastic1004.wikimedia.org - https://phabricator.wikimedia.org/T358061#9574145 (10Izno) [03:02:05] 10Cloud-Services: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#9574177 (10Doc_James) We at Wiki Project Med have created NC Commons for NC and ND licensed material. It currently has about 6.3 million media files. We have consensus from Afrikaans WP to adopt its use. Wondering i... [03:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [03:39:35] 10Cloud-VPS (Project-requests): Request creation of logger-discord-bot VPS project - https://phabricator.wikimedia.org/T358337#9574213 (100xDeadbeef) A rough estimate I got from a person hosting the bot themselves is 1-2GB. So I suppose 2 or 3GB would suffice? [03:42:04] 10Cloud-VPS (Project-requests): Request creation of logger-discord-bot VPS project - https://phabricator.wikimedia.org/T358337#9574214 (100xDeadbeef) Looks like we're moving forward with Trove, and that would probably need to be associated with a tool on toolforge. I've pre-emptively created one at https://tools... [06:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [12:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [14:14:22] 10Cloud-VPS (Project-requests): Request creation of wikiauthbot-ng VPS project - https://phabricator.wikimedia.org/T358427#9574490 (100xDeadbeef) [14:29:08] 10Cloud-VPS (Project-requests): Request creation of wikiauthbot-ng VPS project - https://phabricator.wikimedia.org/T358427#9574502 (100xDeadbeef) Hmm, looks like [Dragonfly](https://github.com/dragonflydb/dragonfly) is able to outcompete Redis in terms of performance and resource usage. I'm interested in trying... [15:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [19:22:32] 10Tool-global-search: Global Search is down: 500: Internal Server Error / Could not resolve host: cloudelastic1004.wikimedia.org - https://phabricator.wikimedia.org/T358061#9574576 (10MusikAnimal) Okay, we're back up and running. I do not recall why we were querying cloudelastic1004.wikimedia.org directly instea... [21:10:49] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [22:06:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:11:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown