[07:26:56] greetings [09:24:43] I just filed T413185 while looking at the tf-infra-test firing alert JFYI [09:24:44] T413185: Increase signal of Tofuinfratest alerts - https://phabricator.wikimedia.org/T413185 [09:26:01] I see we have alerts for both tofu and terraform defined, I think the latter can be removed? https://phabricator.wikimedia.org/P86735 [09:28:37] yes that was the old name of the alert, it was probably not cleaned up after creating the new ones [09:29:30] ok I'll remove them, thank you dhinus [10:03:00] quick update on the "updatetools" errors from yesterday: the last ~50 runs were successful, despite the fact that toolsdb is still seeing elevated traffic and higher-than-average "connection problems" [10:03:09] https://grafana.wmcloud.org/d/PTtEnEyVk/toolsdb-mariadb?orgId=1&from=now-7d&to=now&timezone=utc&var-server=tools-db-6 [12:09:54] the urlshortener followup I was mentioning yesterday T413211 [12:09:55] T413211: Allow urlshortener for a selection of third-level domains from wmcloud.org and toolforge.org - https://phabricator.wikimedia.org/T413211 [13:49:12] heh T413219 have we seen this before and/or rings a bell? [13:49:13] T413219: Tofu refresh state error during wmcs.vps.create_project - https://phabricator.wikimedia.org/T413219 [13:50:49] related MR is https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/287 [15:31:29] yes this is a duplicate of T410265 [15:31:30] T410265: [tofu-infra] "tofu plan" failing in codfw - https://phabricator.wikimedia.org/T410265 [15:33:08] I will work around it by creating the project manually and then re-running the cookbook with "--skip-tofu-apply" [16:24:57] done, project created [16:25:52] T412865 [16:25:52] T412865: Request creation of wlmaz VPS project - https://phabricator.wikimedia.org/T412865 [18:32:23] I find that every now and then I get `debug2: bad passphrase given, try again...` when trying to ssh into Toolforge. It goes away after a while. Is this 'normal' with others?