[02:30:44] excuse me, could I change Wikimedia developer account's UNIX shell username ? [02:52:15] while it is theoretically possible, it usually breaks things [05:41:52] Warning FailedCreate 84s (x72 over 30m) cronjob-controller Error creating job: jobs.batch "anticompositebot.nolicense-cron-1644469500" is forbidden: exceeded quota: tool-anticompositebot, requested: count/jobs.batch=1, used: count/jobs.batch=17, limited: count/jobs.batch=15 [05:42:03] huh, apparently my CronJobs haven't been running for three days [05:57:10] AntiComposite try scrolling back to 7 Feb 12:52 UTC. [05:57:10] T301081: toolforge: Fix job/cronjob quotas - https://phabricator.wikimedia.org/T301081 [05:57:24] I haven't read the ticket but sounds maybe relevant [05:57:40] yeah, that's what it was [05:58:25] which is fine, I had successfulJobsHistoryLimit: 1 set already for pretty much everything but had forgotten to actually apply the changes [06:03:38] the bigger problem is that I don't have a good way to monitor for that sort of failure [07:31:28] arturo: I'll take a look [07:32:48] ah looks like it was the acls [07:32:55] my bad [08:06:44] !log tools disable puppet globally for enabling puppetdb T214427 [08:06:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:06:48] T214427: Enable puppetdb in toolforge - https://phabricator.wikimedia.org/T214427 [08:16:46] !log tools enable puppetdb and re-enable puppet with puppetdb ssh key management disabled (profile::base::manage_ssh_keys: false) - T214427 [08:16:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:16:50] T214427: Enable puppetdb in toolforge - https://phabricator.wikimedia.org/T214427 [08:17:27] I'm going to wait an hour or so to ensure puppet has ran everywhere before going ahead with `profile::base::manage_ssh_keys: true` [08:45:26] !log tools set `profile::base::manage_ssh_keys: true` globally T214427 [08:45:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:45:32] T214427: Enable puppetdb in toolforge - https://phabricator.wikimedia.org/T214427 [12:54:19] !log tools trying to join node tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud to the grid cluster in tools. - cookbook ran by arturo@nostromo [12:54:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:03:51] !log tools trying to join node tools-sgeweblight-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:03:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:05:08] !log tools trying to join node tools-sgeweblight-10-3 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:05:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:06:26] !log tools trying to join node tools-sgeweblight-10-4 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:06:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:07:52] !log tools trying to join node tools-sgeweblight-10-5 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:07:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:24:14] !log tools trying to join node tools-sgewebgen-10-1 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:24:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:25:28] !log tools trying to join node tools-sgewebgen-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo [13:25:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:14:03] !log bastion deleted shutoff bastion-restricted-eqiad1-01 [14:14:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [14:30:35] Is there any way to customize the "No webservice" page? The one you get if your tool is down, "The URL you have requested, https://spi-tools-dev.toolforge.org/spi/, is not currently serviced...." [14:31:00] It would be nice to be able to put up something saying, "Down for maintenance, expect to be back up at XXXX" [14:41:16] not an answer to your question, but I usually put such messages in the tool’s SAL [15:07:31] !log tools shutdown tools-clushmaster-02 T298191 [15:07:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:07:34] T298191: Toolforge: Replace clush with cumin - https://phabricator.wikimedia.org/T298191 [15:07:41] roy649: no, that isn't currently supported [15:34:06] roy649: one idea could be to leave the webservice up, and replace your index.html with a maintenance one [16:58:54] roy649: that is an interesting idea. The page you are talking about is served by the fourohfour tool (https://toolsadmin.wikimedia.org/tools/id/fourohfour). The source code is at https://phabricator.wikimedia.org/source/tool-fourohfour/. That tool already does things to lookup information about the target tool in LDAP at runtime. If we could think of a way to pass other data to it then that could be used to do what you are describing. [17:00:24] How to pass the data is the tricky bit. Today it might be possible with some kind of well known file in the target tool's $HOME, but that will add new complexity to the hope of getting rid of NFS mounts into the webservice pods in the longer term. [17:53:05] “here are the latest log messages for the tool: