[08:06:53] <Sascha80>	 Hi all, I’m a newbie Cloud VPS user. I’ve created my first server instance, but cannot log ssh to the machine. (But I can log into the bastion just fine). When I run “ssh -v -J sascha@bastion.wmcloud.org sascha@webserver.qrank.equiad1.wikimedia.cloud”, I get this suspicious log message:
[08:06:55] <Sascha80>	 debug1: Authentication succeeded (publickey).
[08:06:55] <Sascha80>	 Authenticated to bastion.wmcloud.org ([185.15.56.13]:22).
[08:06:56] <Sascha80>	 debug1: channel_connect_stdio_fwd: webserver.qrank.equiad1.wikimedia.cloud:22
[08:06:56] <Sascha80>	 debug1: channel 0: new [stdio-forward]
[08:06:57] <Sascha80>	 debug1: getpeername failed: Bad file descriptor
[08:06:57] <Sascha80>	 debug1: Requesting no-more-sessions@openssh.com
[08:06:58] <Sascha80>	 debug1: Entering interactive session.
[08:06:58] <Sascha80>	 debug1: pledge: filesystem full
[08:06:59] <Sascha80>	 debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
[08:07:00] <Sascha80>	 debug1: client_input_hostkeys: searching /Users/sascha/.ssh/known_hosts for bastion.wmcloud.org / (none)
[08:07:00] <Sascha80>	 debug1: client_input_hostkeys: searching /Users/sascha/.ssh/known_hosts2 for bastion.wmcloud.org / (none)
[08:07:01] <Sascha80>	 debug1: client_input_hostkeys: hostkeys file /Users/sascha/.ssh/known_hosts2 does not exist
[08:07:01] <Sascha80>	 debug1: client_input_hostkeys: no new or deprecated keys from server
[08:07:02] <Sascha80>	 channel 0: open failed: administratively prohibited: open failed
[08:07:02] <Sascha80>	 stdio forwarding failed
[08:07:03] <Sascha80>	 kex_exchange_identification: Connection closed by remote host
[08:07:03] <Sascha80>	 Connection closed by UNKNOWN port 65535
[08:08:12] <legoktm>	 hi Sascha80
[08:08:20] <legoktm>	 you have a typo, it should be "eqiad1"
[08:08:42] <legoktm>	 (also in the future please use a pastebin when pasting multiple lines :))
[08:10:12] <Sascha80>	 Hi legoktm, thank you! Sorry for my confusion, but where’s the typo?
[08:10:28] <legoktm>	 > webserver.qrank.equiad1.wikimedia.cloud:22
[08:10:42] <legoktm>	 should be webserver.qrank.eqiad1.wikimedia.cloud
[08:12:21] <Sascha80>	 Oh! Thank you!
[10:35:06] <addshore>	 Could someone re enable this account please? https://phabricator.wikimedia.org/people/manage/31343/
[10:35:23] <addshore>	 it was blocked on wikitech because of lack of email (now unblocked and email being set)
[10:35:31] <addshore>	 but it looks like unblock doesnt equal unblock on phab
[10:42:12] <Lucas_WMDE>	 there’s a similar discussion going on in -tech right now fyi
[10:42:17] <addshore>	 :D
[10:49:58] <dcaro>	 -tech?
[10:50:12] <Lucas_WMDE>	 #wikimedia-tech IRC channel
[10:50:18] <Lucas_WMDE>	 I was lazy :P
[10:50:36] <dcaro>	 why I'm not in that channel? xd joining, thanks!
[10:52:09] <Sascha68>	 Another Cloud VPS question from a newbie user: How to handle “Duplicate RecordSet” when setting up web proxies? In my initial testing, I’ve set up my server (project “qrank”, instance “webserver”) and then successfully created a web proxy as qrank.wmcloud.org. This worked perfectly fine. Then, later during my testing, I destroyed my
[10:52:10] <Sascha68>	 backend instance, and set it up again (now more cleanly, as a systemd service). It appears OpenStack has automatically deleted the web proxy “qrank.wmcloud.org” when I deleted my backend instance, which is all good. However, when I now go to https://horizon.wikimedia.org/project/proxy/# I can create *new* proxies (I just tried with
[10:52:10] <Sascha68>	 qrank-test.wmcloud.org and qrank-test2.wmcloud.org, which work fine); but if I try to re-create “qrank.wmcloud.org” (project=qrank, hostname=qrank, domain=wmcloud.org., backend instance=webserver, backend port=8080) I get an error, with details “Duplicate RecordSet”. So, it seems that somewhere in OpenStack, the old record still exists, but
[10:52:11] <Sascha68>	 it’s not visible to me in the Horizon console even though it was me who created the proxy. In a quick web search, I found a seemingly related OpenStack bug [https://bugs.launchpad.net/designate/+bug/1500798] but that was fixed in 2015. — Can a Cloud VPS admin see what’s going on? If you can, free to delete “qrank.wmcloud.org” so I can set
[10:52:11] <Sascha68>	 it up again via Horizon. Thanks for your help. — Sascha
[10:56:03] <taavi>	 Sascha68: I removed the qrank-test.wmcloud.org. record manually.
[10:56:10] <taavi>	 what about qrank-test2?
[10:57:05] <Sascha68>	 Sure, but I can delete that too myself. My problem is that I can’t (re-)create qrank.toolforge.org.
[10:57:10] <Sascha68>	 Sorry, qrank.wmcloud.org.
[10:57:37] <taavi>	 oh, qrank not qrank-test? oops
[11:01:05] <Sascha68>	 Yes. In https://horizon.wikimedia.org/project/proxy/# I can see qrank-test and qrank-test2, but I cannot see (and therefore not modify delete) qrank. But I cannot create a new qrank either; if I try to do that, I get the “Duplicate RecordSet” error.
[11:02:04] <taavi>	 try now?
[11:03:03] <Sascha68>	 Perfect, thank you!
[11:09:17] <taavi>	 I also filed T298681 to fix this edge case
[11:09:17] <stashbot>	 T298681: nova wmfsink not deleting dns records for proxies whose name equals to project name - https://phabricator.wikimedia.org/T298681
[11:23:04] <Sascha68>	 Thank you! ❤️
[13:18:02] <taavi>	 !log cloudinfra shut down cloud-cumin-02 T255980
[13:18:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[13:18:05] <stashbot>	 T255980: Migrate cloudinfra Cumin hosts to Buster - https://phabricator.wikimedia.org/T255980
[14:53:23] <Yi7cuGhe>	 Hi y'all my account was blocked last night for not having an email address associated with my account. As I always had an email address linked in my wikimedia account or on phab and still do I am not sure how this happened. Is there any way to appeal the block?
[15:09:33] <wm-bb>	 <bd808> Yi7cuGhe: I’m not fully online yet today, but yes I’m happy to work with you to remove the block. The accounts I blocked for lack of email yesterday were based on the data in the wikitech db.
[15:09:56] <wm-bb>	 <bd808> So we just need to get that fixed for your account
[15:17:16] <wm-bb>	 <Deep_Crawler> Hi, it appears that I cannot execute this command from php's exec:
[15:17:16] <wm-bb>	 <Deep_Crawler> . ./../pwb_venv/bin/activate
[15:17:18] <wm-bb>	 <Deep_Crawler> it is in one of .sh files on the server. when I run it from PuTTY works, but doesn't work from php. Do you have any idea?
[15:18:37] <wm-bb>	 <lucaswerkmeister> IIRC the PHP container doesn’t have a python3 command (though it does have python2)
[15:18:55] <wm-bb>	 <lucaswerkmeister> and the venv just points to /usr/bin/python3 or something like that, it doesn’t include a full Python install
[15:19:39] <wm-bb>	 <Deep_Crawler> So how can I execute venv from php?! it's not possible? (re @lucaswerkmeister: and the venv just points to /usr/bin/python3 or something like that, it doesn’t include a full Python install)
[15:20:12] <wm-bb>	 <Deep_Crawler> Because I need python3 and some libraries which work with python3
[15:20:15] <taavi>	 No, running python using the php container is not supported. Use the python3 web service type instead (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Python)
[15:21:17] <wm-bb>	 <Deep_Crawler> Thanx a lot, I will work on it 🙏 (re @wmtelegram_bot: <taavi> No, running python using the php container is not supported. Use the python3 web service type instead (https://wikitech....)
[15:21:28] <bd808>	 Yi7cuGhe: Does the current editing block prevent you from adding an email address at <https://wikitech.wikimedia.org/wiki/Special:Preferences>?
[15:21:55] <andrewbogott>	 taavi: I'm briefly wondering if sink should delete proxies at all. If it left them alone then the user could still see the old proxy in the UI and delete it or edit it to point to a new VM...
[15:22:05] <andrewbogott>	 What do you think, uxwise?
[15:22:16] <taavi>	 bd808: we have $wgBlockDisablesLogin = true; on wikitech
[15:22:55] <Yi7cuGhe>	 yeah I can't login
[15:23:14] <bd808>	 taavi: ugh. right that's from a past attack.
[15:23:56] <taavi>	 andrewbogott: deleting it will show a "no proxy", not deleting will show a 503 without more context (or something completely different if the IP gets recycled)
[15:24:11] <andrewbogott>	 hm, true
[15:24:13] <bd808>	 Yi7cuGhe: I just unblocked you on wikitech. Please set an email address.
[15:24:20] <Yi7cuGhe>	 thank you
[15:24:23] <taavi>	 I think deleting it is better currently, ideally it would tell you that the instance has proxies and you need to delete the proxy before deleting the instance
[15:24:57] <taavi>	 bd808: btw, currently unblocking doesn't re-enable the gerrit/wikitech accounts, I wrote https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/751941 to fix that
[15:25:39] <taavi>	 some phab admins manually enabled people earlier today, but those accounts may still be locked out from gerrit
[15:26:53] <bd808>	 taavi: it looks like you are also accidentally fixing T222209 with that patch. I will review soon.
[15:26:54] <stashbot>	 T222209: Cleanup logging and curl use in wikitech post-block hooks - https://phabricator.wikimedia.org/T222209
[15:56:34] <Krinkle>	 Im using kubernetes/php7.3 in toolforge and noticing that when using webservice shell to run a composer command, I get "Warning from https://repo.packagist.org: Support for Composer 1 is deprecated "
[15:56:47] <Krinkle>	 curious what the recommended way is to use composer 2 at this point
[15:57:00] <taavi>	 by using the php 7.4 container
[15:58:52] <Krinkle>	 he, that's reasonable
[15:58:53] <Krinkle>	 thx
[16:21:38] <Reedy>	 heh
[18:50:38] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 57dc392b8f (l10n updates)
[18:50:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL