[14:16:06] !log dcaro@tools-bastion-13 tools.zoomviewer updated the toolforge script to use the latest stable api path [14:16:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoomviewer/SAL [15:18:13] facts for hosts in the puppet-diffs project are not available in pcc, would someone be able to run the upload facts scripts, from cloudinfra-cloudvps-puppetserver-1.cloudinfra.eqiad1.wikimedia.cloud, to see if that resolves the issue [15:18:18] https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Manually_update_cloud [15:26:10] !log cloudinfra Running /usr/local/sbin/puppet-facts-upload on cloudinfra-cloudvps-puppetserver-1.cloudinfra.eqiad1.wikimedia.cloud per request by jhathaway [15:26:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:26:25] * jhathaway gets excited [15:27:01] jhathaway: this seems to be taking a long time... I wonder what it is doing [15:27:25] it uploads all the facts for hosts that use that puppetserver [15:27:34] I'm not sure of the count of hosts [15:27:59] oh dear god. that will be something near 800 instances [15:28:00] though these facts upload on a systemd timer I believe, so my assumption is that something is broken [15:30:55] I'm going to stop the script and re-run with `-vvv` logging [15:32:29] lol. it basically immediately shells out to another script (/usr/local/bin/puppet-facts-export) [15:33:33] nothing like a wrapper script to sprinkle SRE joy on your day! [15:34:06] jhathaway: this is going to take a while... there are 17783 data files in /var/lib/puppetserver/server_data/facts that the script is now redacting and compressing for upload to the pcc server [15:35:26] apparently every test instance that is built leaves a facts file behind forever [15:35:27] weird, my back of the envelope calculation is only 12MiB, but that many files may be the result of cloud vps churn, i.e. hosts that no longer exist? [15:36:02] hmm, seems like we should do more filterning on the upload client side [15:36:21] we have some on the puppetdb side, but seems sensless to upload and then discard [15:36:33] many, many, many files like "fullstackd-20240716235835.admin-monitoring.eqiad1.wikimedia.cloud.json" [15:36:50] nod, that is definitely on the ignore list [15:37:08] if only we had some fullstack SREs around ;) [15:37:25] * bd808 just plays SRE in WMCS for fun [15:38:30] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/puppet_compiler/files/pcc_facts_processor.py#58 [15:39:36] having an equivalent exclude in `puppet-facts-export` would probably be helpful [15:40:27] yeah [15:40:48] jhathaway: it all ended with "413 Request Entity Too Large" :(( [15:40:56] chef kiss [15:41:49] /tmp/puppet-facts-export.tar.xz is 27M file. puppet-compiler.wmflabs.org rejected it [15:42:24] Let me mule the tarball over to the other node myself... [15:42:50] thanks, I figured it would end poorly, given that the upload is triggered daily, I'll have to find out how to obtain cloudinfra access in order to iterate on the process [15:47:57] jhathaway: pcc-db1001.puppet-diffs.eqiad1.wikimedia.cloud:/root/puppet-facts-export.tar.xz is the export file. [15:48:07] perfect, thanks! [15:50:03] !log cloudinfra Added jhathaway as a project member to support PCC work [15:50:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [15:50:22] jhathaway: ^ you should be able to get into the project easily now too [15:50:36] bd808: very much appreciated thanks [15:51:14] yw. all just part of my evil plan to unblock as many folks as I can :) [15:51:40] <3 [15:54:51] This error appeared for a few seconds on arwiki [15:54:51] `MediaWiki internal error. [15:54:53] Original exception: [2c651563-9808-482a-885c-b7f383ef1676] 2024-07-17 15:52:25: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [15:54:54] Exception caught inside exception handler. [15:54:56] Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.` [15:58:45] @GergesShamon: ack. I looked up the error log and it matches things that seem to be known in the #wikimedia-operations channel. It was a transient failure caused by the "circuit breaker" code that tries to protect the databases from cascading failures when overloaded. [16:00:24] Content wiki errors are generally off topic for this channel. I know a bunch of folks want to report such things from Telegram, but we can't guarantee anyone will forward things on the IRC side or do log lookups for you. [16:36:25] Quarry not allowing Wikibooks? [16:53:57] Yetkin: it should work, I can find wikibooks queries under "recent queries" [17:08:17] bd808: thanks! (for unblocking folks) [17:08:54] @GergesShamon: See "Activity > Newest Tasks" in the issue tracker at https://phabricator.wikimedia.org/ and https://www.mediawiki.org/wiki/How_to_report_a_bug as it's unrelated to WM Cloud Services [17:36:22] Yes, it works. The autocomplate feature when searching "trwikibooks_p" does not work, though. (re @wmtelegram_bot: Yetkin: it should work, I can find wikibooks queries under "recent queries") [18:03:24] andrewbogott: for copypatrol in T369723, I'd like to be around to stop processes before you update Trove. Mon/Wed/Fri (US time) works best for me this week or next. Could there be data loss from updating? (i.e., should I also do a backup?) [18:03:25] T369723: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723 [18:17:13] JJMC89: there should not be data loss but backups never hurt! [18:17:47] can we do 17:00 UTC on Monday? [18:21:08] andrewbogott: 1700 Monday works for me [18:21:42] great! [19:47:59] is there a way to block agressive scraping of a cloud vps project? [19:48:17] the only IP I see is from the proxy, so, unsure what my options are [19:48:48] (a) how to find if this is a single ip (b) how to block that single ip [19:50:14] seems like a lot of scrapers hitting patchdemo (running on patchdemo3.visualeditor.eqiad1.wikimedia.cloud ) from apache traffic logs I see a bunch of hits coming rapidly from the proxy...and that's about it. [19:54:31] I guess I could look for X-forwarded-for header... [20:13:16] thcipriani: I think we have to add patchdemo to a list of projects that are allowed to see the remote IP if you want to try IP range blocking. This is something that we do for a small number of projects such as xtools which have well informed sysops and nasty abuse problems. [20:14:17] One thing that almost always stops scraping is putting things behind an OAuth login, but that is not always an easy thing to add. [20:14:53] another option might be UA filtering like https://wikitech.wikimedia.org/wiki/Nova_Resource:Copypatrol#Web_server [20:15:04] hrm, user agent seems to be some "Amazonbot" maybe I can figure out blocking at the apache level for that... [20:15:50] you can follow what I linked for that [20:15:57] <3 [20:24:51] amazonbot is just a fraction, most of the abusive traffic i saw had random browser user-agents [20:28:28] and it stopped at the moment. you can pretty much track it by the load chart here: https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=visualeditor&from=now-3h&to=now&var-instance=patchdemo3&viewPanel=3 [20:31:15] hi, I'm trying to connect to the database replicas. I created a new tool and have logged in via ssh. The docs say that the database credentials are created automatically in the home directory but I don't see them. I've logged out of ssh and back in. I do see the credentials file when I run the "become" command for the tool. [20:31:40] I thought that the creds would be available for both the tools and user accounts [20:32:14] rmontoya-agi: it can take a few minutes for the bot that makes the credentials to do so. Have you been waiting more than say 30 minutes? [20:32:42] rmontoya-agi: it can take a few minutes for the bot that makes the credentials to do so. Have you been waiting more than say 30 minutes? [20:33:48] it's been about 15. I'll wait a bit more. I just wanted to make sure that the expectation was that the creds would also be available on my user account and that I didn't have to become tool account [20:34:54] I don't remember if we still provision database credentials for tool maintainers separately from tools anymore or not actually... [20:36:26] now that I read `I do see the credentials file when I run the "become" command for the tool.` I think you are seeing what you should see: a replica.my.cnf file in a tool's $HOME [20:37:16] rmontoya-agi: if you do not have a replica.my.cnf file in your own $HOME then I would guess we stopped handing those out at some point [20:37:52] ok, so far I don't see one. It was in the database wiki that I saw "Database credentials are generated on account creation and placed in a file called replica.my.cnf in the home directory of both a Tool and a Tools user account. This file cannot be modified or removed by users." [20:38:34] The file would have been created for your user at the point that you were granted access to Toolforge. [20:38:51] (assuming we still do that) [20:39:34] ok then I'll proceed with the "become" command. I do want to set up a tunnel for it so I'll see if that still works if there isn't a creds file in the user directory [20:39:43] The tool maintainer's credentials were separate from a tool's credentials always, so you are talking about 2 separate mariadb accounts [20:43:01] ok, I created a new replica.my.cnf file in my user account and just copied the contents from the tools conf and it works [20:44:04] rmontoya-agi: be aware that the tool's replica.my.cnf could be changed by the system at any time. There is no guarantee of stability other than reading the file or envvar at runtime. [20:44:55] ok, thank you. I'll make a note of that. [20:48:35] thcipriani: if you do decide you need to see the real visitor IP address, there are now instructions for admins on how to help you -- https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Web_proxy#Expose_real_visitor_IPs_to_a_proxied_service [20:51:52] bd808: as a result of the above: https://phabricator.wikimedia.org/T370361 [20:52:23] thanks JJMC89 [20:53:53] bd808: nice! thank you! [23:49:12] !log maps-experiments deleting project as per https://phabricator.wikimedia.org/T367539#9989573 [23:49:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps-experiments/SAL [23:55:49] !log wikicommunityhealth stopping all VMs due to lack of response on T367560 [23:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicommunityhealth/SAL [23:55:52] T367560: Cloud VPS "wikicommunityhealth" project Buster deprecation - https://phabricator.wikimedia.org/T367560