[00:35:53] pymysql.err.OperationalError: (1356, "View 'commonswiki_p.page' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them") [00:35:57] well that's new [00:51:23] AntiComposite: Its due to T60674 [00:51:24] T60674: Drop page.page_restrictions column from Wikimedia wikis - https://phabricator.wikimedia.org/T60674 [14:39:07] bd808 FYI bridgebot sends double messages from IRC to Telegram again :( [14:42:01] !log tools.bridgebot Double IRC messages to other bridges [14:45:17] * Lucas_WMDE half-seriously ponders the idea of the bot having a webservice where anyone can trigger a reconnect [14:45:23] (or whatever it is that temporarily fixes the issue) [14:51:21] @Lucas_WMDE: it's a hard restart of the bot to fix the doubling bug. The command I run on the server is `bb.sh restart 'Double IRC messages to other bridges'`... which I'm going to document on the tool's wikitech page right now. [15:01:03] Thanks :) [15:02:58] !log tools.stewardbots Restart StewardBot, disconnected from IRC [15:03:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [15:11:31] bd808: thanks, that’s useful to have on the wiki page at least :) [15:13:30] @Lucas_WMDE: would you be willing to be a co-maintainer of the bot to help with this bug and similar problems when I'm not available? I have several weeks of vacation staring soon and will feel bad if the telegram bridges become unusable while I'm away. [15:14:03] I’m happy to restart the bot (volunteer account would probably make more sense), sure [15:14:11] don’t know if I could look into the issue on the code side [15:15:49] !log tools.bridgebot Add lucaswerkmeister as co-maintainer [15:15:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [15:17:18] @Lucas_WMDE: You have the power to restart now :) #someday I will find a way to fix the irc client code, but for now having more folks who can restart is the easiest solution. [15:17:46] What exactly is the issue? Is the source code public, I may be able to take a look? [15:18:26] thanks! `become bridgebot` is working, I’ll probably have an opportunity to try a restart soon enough ^^ [15:19:33] Zppix: it uses matterbridge (https://github.com/42wim/matterbridge) [15:21:23] Zppix: https://phabricator.wikimedia.org/T305487 is the Phabricator task, it has some details I think [15:21:52] Lucas_WMDE: Oh, thanks for the link, I’ll take a look and see if that’s something thats within my skill level. [16:51:48] !log tools delete tools-sgeexec-0904 for T309525 experimentation [16:51:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:51:51] T309525: Toolforge: Create a cookbook to decomission a SGE node - https://phabricator.wikimedia.org/T309525 [18:23:18] roy649: possibly worth looking at https://doc.wikimedia.org/puppet/puppet_classes/rsyslog.html etc as well? :) [18:47:11] Yeah, I think what I need is a modification of https://doc.wikimedia.org/puppet/puppet_classes/rsyslog_3A_3Areceiver.html [18:48:00] I'm just getting up to speed on the details of puppet. I've used it before, but only in the sense of "Let me shoot an email to the devops guys and tell them what I need" :-) [18:52:44] roy649: you will need git clone "https://gerrit.wikimedia.org/r/operations/puppet" [18:53:37] then after you have a local copy, go to: [18:53:47] ./puppet/modules/profile/manifests/rsyslog [18:54:04] you will find a bunch of "profile::rsyslog::foo" classes there [18:54:17] check out which one is closest to what you need [18:54:52] then you'd want a new profile:rsyslog::whatever for your needs [18:55:07] basically it's just one "rsyslog::conf" snippet [18:55:19] which defines a source for a config file [18:55:50] the actual config files are in ./puppet/modules/profile/files/rsyslog/ [18:56:19] once you have that I could merge such a config for you and then you can use it without having to run your own puppetmaster [18:56:46] OK, that mostly makes sense. I'll give it a go. [18:57:02] roy649: when did spi-tools get a VPS? ^^ [18:57:24] I've had it for a while, don't make much use of it. [18:57:37] The main spi-tools app runs in toolforge. [18:59:12] Right now I'm on one of my periodic "I need to get logging working in some rational way", so I'm trying to get it to log via syslog, and one thing led to another which eventually led to, "This seems like a good time to bite the bullet and learn how puppet works" :-) [19:00:19] ahh, yak shaving :) [19:01:27] roy649: a core principle of puppet is you don't tell it what to do, you declare what the _desired state_ is. Then you let puppet worry about how to get to that state. [19:01:51] so for example you say "ensure this package is installed" and then it does whatever is needed to make that happen on this type of system [19:02:17] yeah, that part I got. It's a simple concept, which then gets wrapped up in a DSL with templates and classes and modules and a zillion other details :-) [19:02:40] I used to run Ansible on some of my hobby projects. Pretty much the same idea. [19:02:54] But Puppet seems to be what all the cool kids are using these days. [19:03:04] yea! so for this case, copy existing code and then slightly change it will be the easiest way [19:03:08] Yes, yak shaving :-) [19:03:29] start by making the rsyslog config you want [19:03:34] before you get to actual puppet [19:03:50] then copy that config file into the modules/profile/files/rsyslog/ dir [19:04:07] *cough* if someone wants to take a quick look at the patch for T309449 ... [19:04:08] T309449: Package 'cgroup-bin' has no installation candidate on Debian 11 (modules/mediawiki/manifests/cgroup.pp) - https://phabricator.wikimedia.org/T309449 [19:04:14] But, if I start hand-editing the rsyslog config files, won't puppet just keep overwriting them? [19:05:16] let me check that [19:08:43] roy649: yes, if you put a new file into /etc/rsyslog.d/ that is not puppetized, puppet will remove it [19:08:54] sadness :-( [19:09:03] roy649: but.. it's acceptable to disable puppet for a short time to test somehting [19:09:14] as long as it's not permanently disabled [19:09:21] ah, that sounds like a plan [19:10:07] so you can "sudo puppet agent --disable 'testing rsyslog config'" [19:10:16] just don't forget to re-enable it [19:10:17] Ah, excellent. [19:10:22] thanks. [19:11:20] the actual puppet code will just end up like: [19:11:20] # rsyslog::conf { 'hadoop': [19:11:21] # content => template('hadoop/hadoop.conf.erb'), [19:11:21] # priority => 90, [19:11:21] # } [19:11:37] it will have a name, a priority and a path to a template [19:11:45] but you can make that easier by not even using a template [19:12:42] the existing files in modules/profile/files/rsyslog/ are just simple files with config for it as well [19:13:24] so once you have the config itself working.. the puppet part should not be super complex [20:20:30] TheresNoTime: reviewed :) [20:20:58] thank you kindly :) [21:12:02] Is anyone using the S3 API with our internal swift? Trying to set it up for ES snapshots and not sure where to start [21:18:55] inflatador: probably a better question in #wikimedia-sre. For what it's worth I don't know of any S3 gateway usage myself in prod. [21:20:13] bd808 cool, I'll give a shout-out. Thanks [21:20:15] inflatador: g.odog used to be the swift knower of things™. Not sure if that's still a thing he manages or not these days. [21:22:41] yeah, that's what I was told as well , about to email him