[02:01:52] !log bd808@tools-bastion-12 tools.schedule-deployment Built new image from 7da2df29 [02:01:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [02:02:29] !log bd808@tools-bastion-12 tools.schedule-deployment Restart to pick up new container image [02:02:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [09:06:49] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed 253d1b0f45 (l10n updates: pa) [09:06:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [09:20:06] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 24a82d8701 (search items in interface language) [09:20:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [09:24:07] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 88402067dd (update GitHub actions), pulled without webservice restart [09:24:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [10:31:29] !log deployment-prep deployment-puppetserver-1 - in /srv/git/operations/puppet cherry-picked I477c4b72297d8e740461a029e0fd1c7bca818c2f to test wikivoyage.beta.wmcloud.org domain handling - T355281 [10:31:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [10:31:33] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [10:38:58] I think I have a problem, somehow the deployment-puppetserver-1 stopped responding, I cannot SSH into it. could anyone check if they can reach deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud ? [11:10:06] confirmed [11:13:40] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [11:13:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [11:14:17] !log deployment-prep proceeding with soft restart deployment-puppetserver-1 [11:14:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [13:01:46] Do we know how much time does it take for SSL certificxates to update on deployment-prep? Some time ago I added new domains in hiera configs (*.wikimedia.beta.wmcloud.org) and it refreshed certificates pretty quickly. [13:03:11] https://test2.wikimedia.beta.wmcloud.org shows it has a correct certificate. Yesterday I added another wildcard in hiera `deployment-acme-chief` section - `*.wikivoyage.beta.wmcloud.org`, but https://pl.wikivoyage.beta.wmcloud.org/ still reports invalid cetificate. [13:09:21] hmm [13:09:27] cert has been issued [13:09:44] https://www.irccloud.com/pastebin/TxIl9du4/ [13:11:02] but puppet is broken on deployment-cache-text08 so it never got deployed [13:11:53] actually I broke it :) [13:12:04] Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::lvs::realserver::ipip::interfaces' (file: /srv/puppet_code/environments/production/modules/profile/manifests/lvs/realserver/ipip.pp, line: 20) on node deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud [13:15:32] thanks for quick response [13:16:04] ok, so cert is there. I also wonder if we want to start creating a new cert for wmcloud.org domain instead of adding everything to wmflabs.org [13:17:46] do you know how we can fix it? [13:18:59] It's first time I'm doing such changes on hiera/working with puppet. I did some puppet work long time ago - but what we have is a totally different level :) [13:21:22] I'll fix puppet [13:22:29] thanks, Do you know who would be the best person to talk to about splitting certificates into two? not needed for now, but I assume, if we start adding more alternative names we will quickly cross that 40 domains mark [13:24:03] we enforce that limit to avoid getting a huge certficate that requires several RTTs during the TLS handshake [13:24:19] probably that optiimization isn't required on the beta cluster [13:24:24] but don't quote me on that :) [13:32:24] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040158 [13:32:48] no idea what's wrong with PCC on cloud but at least the ipip related errors that are shown on the production catalog don't appear on the change catalog [13:34:30] good to know regarding the limit. Will see, I think for now we're good with two domains on wmcloud.org but I heard there is an idea to move more stuff from wmflabs.org to wmcloud.org. [13:42:04] pmiazga: tested that CR on cloud by applying the changes via horizon, cache-text08 refreshed the TLS material already [13:42:46] https://www.irccloud.com/pastebin/xolhaWwz/ [13:42:52] it should be ok now [13:43:08] yes, it works now [13:43:19] https://pl.wikivoyage.beta.wmcloud.org/wiki/Strona_g%C5%82%C3%B3wna shows correct certs [14:46:23] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/StewardBot because of a connection loss [14:46:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [16:30:19] bikeshed coloring opportunity! I want to make a Toolforge tool that will maintain a collection of build service created containers for use by other tools. Things like IRC bouncers and cache services that could be reused by a number of projects. What would you call that tool? [16:30:28] Keep in mind that the tool name will be repeated twice when the containers are used (tool-$TOOL/$TOOL-$SERVICE) [16:31:39] i don't think the tool name necessarily has to be used as a prefix on the container name? [16:31:49] "containers"? "shared-containers"? [16:32:30] One snarky idea: "bstools" which totally stands for "build service tool services" [16:34:16] "containers" isn't bad. And you are right taavi that the name doesn't have to be repeated. [16:34:48] So we could have tool-containers/bnc, tool-containers/redis, tool-containers/memcached, ... [16:35:01] I kind of like that [16:35:37] that works surprisingly well even with the tool- prefix [16:37:46] the bikeshed has been colored -- https://toolsadmin.wikimedia.org/tools/id/containers [16:38:17] see also: https://bikeshed.toolforge.org/ [17:16:12] something sends us emails when systemd timers fail on a cloud VPS instance. just like in production. is it new that we have this in cloud? [17:18:51] it's cool. there is just one difference to production. the mail comes from generic noreply@wikimedia.org not from noreply@${hostname}. and since the host name isn't mentioned in the mail body you might have no clue where the issue happens [18:09:52] mutante: cloud vps hosts being able to send mail as @wikimedia.org seems like a Problem.. do you have a sample mail (with headers) that I could look at? [18:11:48] taavi: I just forwarded it to you. and ACK, but it's just soft fail [18:11:50] Received-SPF: softfail (google.com: domain of transitioning noreply@wikimedia.org does not designate 185.15.56.18 as permitted sender) [18:12:29] meanwhile I found out the part that it is not coming from @$hostname is due to https://phabricator.wikimedia.org/T358675 [18:12:35] left a comment there [18:13:02] it's also possible that I see this the first time now because of work of Jesse on mail delivery ? [18:13:24] but more likely the changes by Ben in the ticket above [18:13:33] forwarding only sends the content, not the headers I'm interested in [18:14:16] pasted the headers in a new mail to you [18:14:41] thanks [18:20:38] !log language shut off language-translate2 instance running an open registration mediawiki instance filled with spammers [18:20:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Language/SAL [22:13:00] !log bd808@tools-bastion-12 tools.containers Built tool-containers/bnc:latest from https://gitlab.wikimedia.org/toolforge-repos/containers-bnc/-/tags/v1.0.0 [22:13:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.containers/SAL