[09:06:15] Hi. Would appending a # before a crontab line be sufficient to "suspend" the job or shall I just remove the line? [09:08:43] !log tools.bridgebot Double IRC messages to other bridges [09:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [09:09:16] hauskatze: that should be sufficient as far as I understand [09:09:35] Lucas_WMDE: thanks, I'll try that :) [09:11:11] !log tools.mabot Temporarily suspend archivebot operations refs. T313785 [09:11:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mabot/SAL [11:11:41] !log commtech initiate soft reboot of tts.commtech.eqiad1.wikimedia.cloud, unresponsive [11:11:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL [19:05:25] hi cloud folks! I can't find a phab about it, but perhaps you're already aware that https://php-security-checker.wmcloud.org/ is down? [19:05:45] That's a service which warns us about vulnerable composer packages in our repos [19:06:14] and it's connected to a daily jenkins job for each repo that emails the owner when the job fails [19:06:41] anyway, for a week we in fundraising tech have been getting a false-positive failure email for each repo we manage [19:06:59] ejegg: hey! looks like that url is served by the security-checker1.packagist-mirror.eqiad1.wikimedia.cloud instance, which was shut down last week due to https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation [19:07:17] these are the people you need to reach out to if you need it running again: https://openstack-browser.toolforge.org/project/packagist-mirror [19:07:43] Ahh, so nobody else relies on it? [19:07:59] I see there was a suggestion to move it to toolforge: https://phabricator.wikimedia.org/T296967 [19:10:46] I don't think any of us cloud admins know more about that service, sorry [19:42:58] is there a preferred method to make services HA on network level, other than boxes that sit in the middle (such as load balancers)? [20:02:59] Southparkfan: the standard for wmcs infra is keepalived with a vip, unfortunately that needs some manual work by a cloud admin to set up [20:03:49] If you have the option, you should always do HA on the application level [20:05:59] if it weren't for the syslog standard... [20:07:01] Just send it to multiple (geo-redundant) hosts? That's what I normally do (re @wmtelegram_bot: if it weren't for the syslog standard...) [20:08:37] as long as the syslog clients support that + you don't mind the on-disk duplication at the syslog server side, that's an option [20:09:52] given https://jmagnin.github.io/2020/10/10/syslog-load-balancing-with-haproxy.html putting a haproxy in front of it may be a viable option nowadays, but I need a second option [20:09:58] You can use Kafka for transport and dedupe so you don't have to worry about that downstream [20:10:12] taavi: are those vip's public addresses or internal ones? [20:10:59] Southparkfan: in our network setup instances will never have public IPs directly assigned to them. the VIPs are private ones, but floating IPs can be mapped to them like any other host [20:12:44] But easiest is to just have 2 syslog servers sitting far away from each other so you still have syslog when a site fails [20:16:43] right, the use case is proving a somewhat reliable central syslog receiving/storage service for cloudvps instances acting as syslog clients [20:18:35] My experience is with syslog that you really need it in time of outages so it's worth to have the redundancy. You can always set a more aggressive logrotate strategy on one of the hosts if you are afraid about disk usage [20:18:53] Also useful when you want to do some planned activity on you syslog vm [20:19:14] for the purpose of storing audit logs somewhere else - uptime is not as huge of a concern as in production, but a cold standby that requires fiddling with IP addresses on syslog clients seems too much of a hassle to me [20:22:46] just duplicating everything to two hosts seems like the simplest option here, and I don't think we're talking about so much data that it'd become problematic [20:23:13] Send a copy of your audit logs to production? That way it's also more tamper proof [20:25:01] I am not sure if such cross-realm traffic is preferred [20:26:29] but I recall the logs aren't that huge (@ the 'send to two hosts' idea), let me take a look... [20:26:38] Depends on how important the logs are. If you get pwned, the logs will be gone too [20:28:00] I understand that concern, the goal is to at least maintain an integrity level in case of node or project-wide compromise [20:29:14] protection against tampering via a wmcs-wide compromise? fair to me, actually [20:39:30] around 80 GB per server with 28 days retention, so 160 GB space if we were to get these logs to two nodes