[00:04:24] !log tools.jarbot Increased count/jobs.batch quota to 30 [00:04:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot/SAL [00:06:35] !log tool.citationhunt Increased count/jobs.batch quota to 20 [00:06:36] bd808: Unknown project "tool.citationhunt" [00:06:45] !log tools.citationhunt Increased count/jobs.batch quota to 20 [00:06:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citationhunt/SAL [00:10:48] !log tools.jarbot-iii Increased count/cronjobs.batch to 60 and count/jobs.batch quota to 30 [00:10:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot-iii/SAL [00:12:46] The core toolforge-jobs issue from T308189 is resolved, but we are working on cleanup for some tools that are having trouble starting their jobs [00:12:47] T308189: Toolforge jobs stopped getting scheduled around the same time as the Toolforge k8s cluster upgrade - https://phabricator.wikimedia.org/T308189 [00:16:15] !log tools.citationhunt Increased count/jobs.batch quota to 30 [00:16:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citationhunt/SAL [00:19:17] !log tools.jarbot-ii Increased count/cronjobs.batch to 60 and count/jobs.batch quota to 30 [00:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot-ii/SAL [00:27:20] !log tools.oabot Set concurrencyPolicy: Forbid on oabotrefresh cronjob and deleted stale job pods [00:27:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.oabot/SAL [04:12:51] !log rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying [04:12:52] andrewbogott: Unknown project "rebooting" [04:12:59] !log admin rebooting primary bastion (bastion-eqiad1-03.bastion.eqiad1.wikimedia.cloud) in hopes of resolving a problem with ssh proxying [04:13:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:28:07] !log tools deploy jobs-api update T308204 [09:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:28:10] T308204: toolforge-jobs should set startingDeadlineSeconds by default - https://phabricator.wikimedia.org/T308204 [11:55:09] Hey everyone. Are the Bridgebot issues related to the current scheduled jobs issues? [11:59:55] Titore: what ar ethe bridgebot issues? [11:59:58] *are the [12:00:31] (I don't find any cronjobs in the bridgebot tool itself, so probably not?) [12:01:46] on #wikipedia-it-sysop we're not getting telegram messages [12:36:57] !log tools re-enable CronJobControllerV2 T308205 [12:37:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:37:00] T308205: Re-enable CronJobControllerV2 - https://phabricator.wikimedia.org/T308205 [13:12:23] Titore: I just tried restarting the bot, did that help at all? [13:26:49] yes taavi, thanks! [13:31:56] thanks taavi! [14:21:46] Raymond_Ndibe: \o/ [14:26:27] Hi all, I'm going to move all the traffic from dbproxy1018 to dbproxy1019 shortly, so that we can upgrade dbproxy1018 to bullseye: https://phabricator.wikimedia.org/T298940#7886420 [14:26:27] No action necessary on your part, but figured I should post in case anything stops working. Undoing any of these changes should be straightforward [14:29:15] thanks for the warning razzi [15:03:08] Hey crazy kids, I'm about to reboot some NFS servers which will make toolforge VERY upset and broken for a few minutes. Brace yourselves :) [15:06:32] !log admin stopping nfs-server on labstore1004 in preparation for reboot [15:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:09:58] !log admin rebooting labstore1004 for 304938 [15:10:55] * Lucas_WMDE screams, flails around, etc. [15:11:40] mood. [15:12:10] having a good day Lucas_WMDE? [15:12:31] I was just providing some appropriate feedback for toolforge going down ;) [15:13:08] * TheresNoTime saw the warning and thus was appropriately sated... for now :> [15:52:17] !log tools.stewardbots Deployed ef01194 (within the last hour) [15:52:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [16:17:53] hello, i stopped being able to use crontab command as my tool: it says `tools.wmcz@tools-sgecron-01.tools.eqiad.wmflabs: Permission denied (publickey).`. Does anyone know what happened? [16:18:18] (I can ssh in under my personal account and become the tool at cron server, so not a blocker, but still looks like a bug) [16:22:04] !log tools.stewardbots Deployed b30b346 & restarted SULWatcher [16:22:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [16:35:07] urbanecm: I can reproduce your problem with the wmcz tool. I'll try to poke around a bit when I'm out of meetings to see why the keyless ssh for that account to the cron server is failing. If you have time to open a bug that would make it easier to track. [16:42:31] ok, will do. thanks bd808 [16:44:29] filled as T308263 [16:44:30] T308263: tools.wmcz: cannot use crontab - https://phabricator.wikimedia.org/T308263 [16:50:01] bd808: maybe https://github.com/wikimedia/puppet/commit/acaf2557d89f6745fb6cc5fef5665dfde965e900 breaking things by accident? [16:52:58] oh yes, fixing [16:53:56] urbanecm: try now? [16:55:34] taavi: perfect, thanks! [17:22:25] nice find taavi [21:17:08] !log mwoffliner resizing/rebooting mwoffliner4 as part of hypervisor maintenance [21:17:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwoffliner/SAL [22:06:56] !log bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797 [22:06:57] inflatador: Unknown project "bking@deployment-elastic05" [22:06:57] T299797: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 [22:07:24] well , dang [22:08:21] inflatador: !log [22:09:26] start with !log deployment-prep ... [22:09:30] Thanks mutante , any idea what the beta cluster ...ah ! [22:09:48] !log deployment-prep bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797 [22:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:11:16] yea, in hindsight just calling it actually "beta" or "betacluster" might have been easier [22:12:42] NP, thanks again