[05:30:22] <kuncung>	 Dear Toolforge admin, can you delete my unused tools? I've opened the ticket months ago but no response. T308963. Ty
[05:30:23] <stashbot>	 T308963: Delete tool jv-purge-cache - https://phabricator.wikimedia.org/T308963
[09:09:26] <taavi>	 !log tools.jv-purge-cache marked tool for deletion T308963
[09:09:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jv-purge-cache/SAL
[09:09:28] <stashbot>	 T308963: Delete tool jv-purge-cache - https://phabricator.wikimedia.org/T308963
[09:10:09] <taavi>	 !log tools.exportpdf marked tool for deletion T320254
[09:10:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.exportpdf/SAL
[09:10:11] <stashbot>	 T320254: Archive/delete tool exportpdf - https://phabricator.wikimedia.org/T320254
[09:10:49] <taavi>	 !log tools.devlibrarycard marked tool for deletion T320252
[09:10:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.devlibrarycard/SAL
[09:10:51] <stashbot>	 T320252: Archive/delete tool devlibrarycard - https://phabricator.wikimedia.org/T320252
[09:11:18] <taavi>	 !log tools.comidentgen marked tool for deletion T320250
[09:11:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.comidentgen/SAL
[09:11:20] <stashbot>	 T320250: Archive/delete tool comidentgen - https://phabricator.wikimedia.org/T320250
[09:12:07] <taavi>	 !log tools.paid marked tool for deletion T320246
[09:12:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paid/SAL
[09:12:09] <stashbot>	 T320246: Archive/delete tool paid - https://phabricator.wikimedia.org/T320246
[09:12:35] <taavi>	 !log tools.globalprefs marked tool for deletion T317459
[09:12:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.globalprefs/SAL
[09:12:37] <stashbot>	 T317459: Archive/delete tool globalprefs - https://phabricator.wikimedia.org/T317459
[09:13:01] <taavi>	 !log tools.gblrenamemon marked tool for deletion T317458
[09:13:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gblrenamemon/SAL
[09:13:03] <stashbot>	 T317458: Archive/delete tool gblrenamemon - https://phabricator.wikimedia.org/T317458
[09:13:39] <taavi>	 !log tools.extdist marked tool for deletion T317457
[09:13:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.extdist/SAL
[09:13:41] <stashbot>	 T317457: Archive/delete tool extdist - https://phabricator.wikimedia.org/T317457
[09:14:45] <taavi>	 !log tools.fab-proxy marked tool for deletion T317457
[09:14:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fab-proxy/SAL
[09:15:23] <taavi>	 !log tools.propertyuse marked tool for deletion T317067
[09:15:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.propertyuse/SAL
[09:15:25] <stashbot>	 T317067: Archive/delete tool propertyuse - https://phabricator.wikimedia.org/T317067
[09:15:52] <taavi>	 !log tools.pypidatabot marked tool for deletion T315739
[09:15:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pypidatabot/SAL
[09:15:55] <stashbot>	 T315739: Archive/delete tool PyPIDataBot - https://phabricator.wikimedia.org/T315739
[09:16:27] <taavi>	 !log tools.roysmith-test marked tool for deletion T313352
[09:16:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.roysmith-test/SAL
[09:16:29] <stashbot>	 T313352: Archive/delete tool roysmith-test - https://phabricator.wikimedia.org/T313352
[09:17:19] <taavi>	 !log tools.test-stats marked tool for deletion T310803
[09:17:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.test-stats/SAL
[09:17:21] <stashbot>	 T310803: Archive/delete tool test-stats - https://phabricator.wikimedia.org/T310803
[09:22:43] <taavi>	 !log tools.quotable marked tool for deletion T310788
[09:22:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quotable/SAL
[09:22:46] <stashbot>	 T310788: "Quotable" tool tries to load third-party content; HTTP 502 errors - https://phabricator.wikimedia.org/T310788
[09:24:17] <taavi>	 !log tools.translation-server marked tool for deletion T308587
[09:24:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translation-server/SAL
[09:24:19] <stashbot>	 T308587: Archive/delete tool translation-server - https://phabricator.wikimedia.org/T308587
[09:24:51] <taavi>	 !log tools.test-12 marked tool for deletion T308304
[09:24:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.test-12/SAL
[09:24:53] <stashbot>	 T308304: Archive/delete tool test-12 - https://phabricator.wikimedia.org/T308304
[09:25:41] <taavi>	 !log tools.zhwikiirccc marked tool for deletion T307975
[09:25:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zhwikiirccc/SAL
[09:25:43] <stashbot>	 T307975: Archive/delete tool zhwikiirccc - https://phabricator.wikimedia.org/T307975
[09:26:10] <taavi>	 !log tools.extensionservice marked tool for deletion T307721
[09:26:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.extensionservice/SAL
[09:26:12] <stashbot>	 T307721: Archive/delete tool extensionservice - https://phabricator.wikimedia.org/T307721
[09:26:50] <taavi>	 !log tools.wd-books-cleaner marked tool for deletion T307676
[09:26:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-books-cleaner/SAL
[09:26:52] <stashbot>	 T307676: Archive/delete tool wd-books-cleaner - https://phabricator.wikimedia.org/T307676
[10:30:14] <kuncung>	 taavi: \o/ _/|\_
[12:04:00] <taavi>	 !log admin taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # T320301
[12:04:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:04:03] <stashbot>	 T320301: Account recovery help needed for developer account DatGuy - https://phabricator.wikimedia.org/T320301
[17:21:32] <danmichaelo>	 Hi! Is there a new-ish limit on login sessions on tools-login.wmflabs.org? I'm getting "Too many logins for '[username]'"
[17:22:11] <andrewbogott>	 danmichaelo: I'm getting it too :)  I'll investigate a bit
[17:22:52] <danmichaelo>	 Thanks, glad I'm not the only one :)
[17:23:41] <wm-bb>	 <jeremy_b> forgot this is NFS (re @lucaswerkmeister: yeah, you’d need to run lsof on all Toolforge hosts to know if there are open references or not…)
[17:25:06] <andrewbogott>	 https://phabricator.wikimedia.org/T320352
[17:26:00] <taavi>	 looks like we're hitting some global limit of 100 users logged in at the same time?
[17:26:54] <andrewbogott>	 taavi: it's total users and not per user? I'm not understanding what the 'sd' is in 'Too many logins (max 100) for sd'
[17:27:31] <taavi>	 andrewbogott: I don't know for sure, but `who | wc -l` returns 103 which is suspiciously close to 100 in the log file
[17:27:39] <andrewbogott>	 hm, that would fit
[17:27:54] <andrewbogott>	 although the client message definitely implies that it's personal.
[17:28:09] <taavi>	 yeah..
[17:28:15] <taavi>	 quite a lot of people with idle tmux sessions
[17:29:20] <taavi>	 I'm going to kill a few of those to free up the limit
[17:29:26] <taavi>	 !log tools kill 10 idle tmux sessions of user 'hoi' on tools-sgebastion-10 T320352
[17:29:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:29:29] <stashbot>	 T320352: Users can't connect to login.toolforge.org: 'too many logins' - https://phabricator.wikimedia.org/T320352
[17:29:32] <andrewbogott>	 ok :) That will at least tell us if that's the limit
[17:29:55] <andrewbogott>	 it was!
[17:30:19] <taavi>	 next question: how do we raise that limit?
[17:30:20] <andrewbogott>	 that auth.log error message is weirdly invisible on google
[17:30:42] <andrewbogott>	 Supposedly pam gets limits from /etc/limits.d/ but we don't even have the dir here...
[17:31:58] <taavi>	 /etc/security/limits.d exists and is mentioned on limits.conf(5)
[17:32:18] <andrewbogott>	 yeah, and has *               hard    maxsyslogins    100
[17:32:48] * andrewbogott checks to see if that's puppetized
[17:32:49] <taavi>	 that might explain things
[17:33:21] <taavi>	 yep, comes from profile::toolforge::bastion::resourcecontrol
[17:36:17] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/840673
[17:37:00] <andrewbogott>	 danmichaelo: you should be able to connect now, barely :)
[17:37:15] <taavi>	 looks good
[17:37:42] <andrewbogott>	 I'll merge when CI catches up.
[17:37:50] <andrewbogott>	 thanks for the quick diagnosis :)
[18:01:54] <danmichaelo>	 looks good!