[09:47:46] > ssh hoi@login.toolforge.org [09:47:46] > -bash: exec: fish: not found [09:47:46] > Connection to login.toolforge.org closed. [09:48:28] I cannot login. I need someone help to `mv /home/hoi/.bashrc /home/hoi/.bashrc.bak`. [09:55:32] Hoi: done [09:55:55] taavi: Thx! [10:33:19] !log lucaswerkmeister@tools-bastion-13 tools.bridgebot Double IRC messages to other bridges [10:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [12:54:13] !log anticomposite@tools-bastion-13 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected [12:54:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [13:57:51] !log anticomposite@tools-bastion-13 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected [13:57:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [18:55:48] * bd808 lunch [21:22:47] !log codesearch Attempt to fix puppet failure since 48h `codesearch8 puppet-agent[21081]: (/Stage[main]/Codesearch/Git::Clone[labs/codesearch]/Exec[git_pull_labs/codesearch]/returns) error: insufficient permission for adding an object to repository database .git/objects` [21:22:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [21:27:16] thanks, saw those mails too [21:31:52] using this command btw: [21:31:52] https://github.com/Krinkle/dotfiles/blob/03eff82411c0213e3b7173b7add44b1ded8b97ef/Extended-Mind.md#chown [21:32:01] as stolen from https://gerrit.wikimedia.org/g/operations/puppet/+/b1721a0871515a55f5d9eefdba1447ec30da1bc1/modules/profile/files/mediawiki/deployment/fix-staging-perms.sh [21:32:05] seems to work [21:32:26] although there are still other remaining issues, they don't seem to prevent a succesful puppet run right now [21:32:37] 1) puppet-agent-cron says something about a debian channel being undefined [21:32:51] 2) puppet-agent says something about puppet7 being unavilable on buster. [21:32:56] but I'm ignoring that right now [21:33:16] mutante: do you know more about those or what would be an appropiate response to them? [21:34:42] https://www.irccloud.com/pastebin/PhBFFvIX/ [21:35:22] I guess the first one can be silenced by removing the repo, but it seems like puppet should be doing that on its own so I rahter not touch it [21:35:26] heh, I was about to ask if 1) referred to buster-backports :/ that went away a few days ago, T362518 [21:35:36] I have no idea what to do about it, though, all I can offer is that pointer [21:35:43] yeah, that seems like the same issue [21:35:48] nope [21:35:49] presumably it's using that repo for non-zero packages [21:35:55] or not :) [21:36:46] the first is because for some reason the codesearch module is pulling the docker package from a local repo that toolforge used years ago (but also moved away years ago, so it was removed) [21:37:04] and the second is harmless but will go away once the instance is migrated to a newer debian release [21:52:55] Krinkle: there is a puppet 5 -> puppet 7 migration. machines can opt-into the puppet7 on Hiera. but they shouldn't if they are still on buster. best fix would be to stop using buster [21:53:57] and/or the "bug" would be that they got opted into puppet7 though they aren't suitable for that yet [21:54:28] it's possible this was done on role-level.. or on instance level [21:55:43] you can ignore it if the puppet run still finishes [21:59:16] regarding the docker package installation and repo.. we should probably just include profile::ci::docker to get that .. if it works on buster [22:00:12] or reuse parts from that class [22:01:30] [apt1002:~] $ sudo -E reprepro ls docker-ce [22:01:30] docker-ce | 5:20.10.12~3-0~debian-buster | buster-wikimedia | amd64 [22:01:30] docker-ce | 5:23.0.6-1~debian.10~buster | buster-wikimedia | amd64 [22:06:06] I would recommend switching from Docker to Podman, but I don't know if anyone has written Puppet modules for that yet. [22:06:08] on a buster host like deploy1002 there is docker.io instead of docker-ce. [22:06:40] https://wikitech.wikimedia.org/wiki/Developer.wikimedia.org#Demo_server was magically easy [22:07:18] docker.io is the package name in the main debian repos, docker-ce is used in the repo by docker the company that we sometimes import stuff from to get a newer version [22:09:48] "sudo podman pull docker-registry.wikimedia.org" .. makes me think docker-registry wasn't the perfect neutral name :p [22:10:22] bd808: that does look good [22:42:45] Krinkle: lucaswerkmeister: an attempt to fix: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1022215/1/modules/codesearch/manifests/init.pp [22:48:09] (I don’t think I actually contributed anything useful to this discussion tbh ^^) [22:52:19] !log melos@tools-bastion-13 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC [22:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [22:56:19] you did, Lucas. you pointed to the ticket that shows it's the entire backups that went away [22:56:23] happy Earth Day.. afk now [22:56:32] s/backups/backports