[13:19:53] legoktm: libup-web02.library-upgrader.eqiad1.wikimedia.cloud was left in 'VERIFY_RESIZE' -- I seemed to be working fine so I just now confirmed the resize because I needed it in state 'active' to complete some routine reboots. [13:33:45] !log library-upgraded unilaterally confirming the pending resize on libup-web02 because the pending resize prevented migration of the instance [13:34:19] oops, whither stashbot? [13:35:09] !log library-upgraded unilaterally confirming the pending resize on libup-web02 because the pending resize prevented migration of the instance [13:35:10] andrewbogott: Unknown project "library-upgraded" [13:35:19] !log library-upgrader unilaterally confirming the pending resize on libup-web02 because the pending resize prevented migration of the instance [13:35:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [13:35:41] stashbot returneth [13:46:46] !log tools.ircservserv test out the new components system T397929 [13:46:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ircservserv/SAL [13:46:49] T397929: Migrate ircservserv to components - https://phabricator.wikimedia.org/T397929 [19:00:12] I have a lot of messages in the error log of webservices "2025-06-26 18:57:12: server.c.980) [note] sockets disabled, connection limit reached" Tool: persondata, see /data/project/persondata/error.log … my fault? System limits or user limits? [19:50:15] wurgl: that message from lighttpd means that it hit the max-connections limit. Basically that your tool is getting a lot of parallel requests. [19:51:06] per https://github.com/wikimedia/operations-docker-images-toollabs-images/blob/master/shared/lighttpd/webservice-runner#L32 the limit is 300, so either your tool is massively popular, horribly slow, under attack by crawlers, or all 3 at once. [19:51:12] bd808 can I configure it different, preventing those messages [19:52:30] Today: 350343 entries in access.log [19:52:54] raising the limit probably won't make things better for users, but you may be able to override `server.max-connections` in your tool's $HOME/.lighttpd.conf file. There are some settings that can be overwritten and others that will just raise an error. I don't remember about this one. [19:53:50] Which value is default? [19:55:22] the config is generated by the shell script I linked to above. It puts in `server.max-connections 300` and then later appends whatever config is found in $HOME/.lighttpd.conf [19:56:31] That 11G access.log file is a lot of log to keep around for probably almost no value. [19:57:16] The `accesslog.filename` line in $HOME/.lighttpd.conf makes the server write it out [19:58:03] With crontab I made a backup every month and compressed it … but this is not possible anymore, since I cannot stop/start the webservices [19:58:19] do you actively review it? [19:58:37] I did von time to time [19:58:54] We turned these logs off by default because almost nobody actually uses them -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd#Web_logs [19:59:20] Looking for "bad" h4X0r-style access [19:59:58] what do you do when you find that? Do you keep a user-agent block list of something in the app? [20:00:56] I found some guys with ISO8859-Coding and tried to fix it. [20:01:55] The only app-fix I have is a sleep 60 when someone tries to inject code ;^) [20:04:19] That would keep one of your 300 possible connections open for 60 additional seconds when you could just return an HTTP 400 "Bad Request" instead [20:05:13] your attacker is not going to be bothered by a slow response [20:11:00] Log is off, max-connections is 400 Thanks