[00:42:50] !log mediawiki-vagrant Removed self from project members [00:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [00:43:39] !log mwv-apt Removed self from project members [00:43:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwv-apt/SAL [07:11:58] Hi, I’m one of the managers of the Osmit project. We had a machine to update due to the buster deprecation. However, another machine is no longer working anymore on the assigned web proxy with a general error, see https://osmit-estratti.wmcloud.org [07:11:58] We did not change anything. Do you know what it could be correlated? to? [07:13:26] LorenzoStucchi: looking [07:21:45] LorenzoStucchi: there's nothing listening on port 80 of the machine that the proxy points to (root@osmit-estratti:~# ss -tlnp | grep :80) [07:25:56] LorenzoStucchi: there's a docker container running, from the docker-compose it seems that the `webapp` container that should be running is not (that seems to be the one exposing port 80) [07:26:24] I'll leave it for you to start it or not, let me know if that was the issue [07:34:07] Thanks for the analysis, I thought it was correlated to some process, but it was just a failure of the procedure. Thanks you so much [09:15:41] !log toolsbeta setting kyverno policies to Enforce (T368141) [09:15:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:15:44] T368141: toolforge: kyverno: change policies to Enforce - https://phabricator.wikimedia.org/T368141 [09:18:19] !log tools setting kyverno policies to Enforce (T368141) [09:18:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:25:46] !log taavi@tools-bastion-12 tools.wikibugs toolforge jobs restart phorge [09:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [10:03:09] !log admin taking cloudcephosd1006 out of the pool (T348643) [10:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:03:13] T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 [10:17:42] !log toolsbeta deploying toolforge-webservice 0.103.9 (T368463) [10:17:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:17:45] T368463: `webservice` (build 0.103.8) crashes on login-buster.toolforge.org (python 3.7) - https://phabricator.wikimedia.org/T368463 [10:18:24] !log tools deploying toolforge-webservice 0.103.9 (T368463) [10:18:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:35:00] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [11:35:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [11:40:32] !log tools update pywikibot image to 9.2 T363631 [11:40:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:40:36] T363631: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363631 [12:39:15] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [12:39:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:19:39] what is the best way to handle secrets between multiple cloud vps instances? I'm setting up my second instance and would like to have a single location where I define my application secrets, I couldn't find how to do it in wikitech [19:24:29] don-vip: the standard way to do that is with a common puppetserver, but you may find that to be more trouble than just copying them by hand [19:24:59] I was wondering if a tiny Trove database could work [19:25:11] (but mainly because that’s one thing I happen to know you can share between cloud VPS instances ^^) [19:25:44] sure, that would work [19:26:04] you'd want to be very careful about access rights [20:23:55] in case anyone was wondering, the current report of backfill_too_full on ceph is most probably a bug https://tracker.ceph.com/issues/39555 , should go away once the rebalancing is finished [20:27:15] shoot, ceph is in version 14 on the new reimaged host :/ [20:27:23] (should be fixed in v15) [20:28:44] I'll upgrade those tomorrow... I think there might be a race condition between the repos being set and the packages installed [22:32:22] !log admin disabled all g3.* flavors in eqiad1 [22:32:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:08:52] !log redirects Added redirects-nginx03 to "web" security group (T367550) [23:08:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL [23:08:55] T367550: Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550 [23:14:12] I am about to reboot both cloud-vps bastions. It'll be a short outage but will interrupt any work you might have in progress. [23:17:47] !log redirects Switched all proxies to point at redirects-nginx03 (T367550) [23:17:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL [23:17:50] T367550: Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550 [23:28:56] ok, all done [23:43:52] !log redirects Shutdown redirects-nginx02 (T367550) [23:43:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL [23:43:55] T367550: Cloud VPS "redirects" project Buster deprecation - https://phabricator.wikimedia.org/T367550 [23:48:32] !log redirects Deleted instance redirects-nginx02 (T367550) [23:48:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL [23:50:41] :) cool [23:52:09] thanks for getting that started mutante [23:56:05] yw, yes, for some reason I did not even think about having to apply the group to the instance individually [23:56:18] just "but it's HERE and it HAS the rules"