[00:15:55] <andrewbogott>	 !log  image-suggestion-api deleting project after discussion with bpirkle
[00:15:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Image-suggestion-api/SAL
[00:16:51] <andrewbogott>	 Southparkfan: the host you made is it's 'ircd03' not 'irc03'  :D
[08:00:57] <jminguillona>	 hello everybody, I'm having problems to log into toolforge using putty, is this the right place to find a solution? thanks a lot
[08:02:43] <dcaro>	 jminguillona: yep, did you follow https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart#Set_up_an_SSH_client_and_a_key ?  What error are you seeing?
[08:02:54] <dcaro>	 hello :)
[08:04:37] <jminguillona>	 hello, yes I created a key using puttygen, followed all instructions, open the ssh session, asks for my private key, but it suddenly closes the session 
[08:05:26] <jminguillona>	 if I provide a wrong password it complains, so it's something related to establishing the session, but no idea
[08:07:46] <dcaro>	 did you add your key to https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ ?
[08:08:24] <jminguillona>	 yes
[08:08:31] <dcaro>	 did you request membership? https://toolsadmin.wikimedia.org/tools/membership/apply
[08:08:59] <dcaro>	 you have to have a tool in mind already, if you have some code that would help a lot with the approval
[08:09:15] <jminguillona>	 oops, no, thanks
[08:09:54] <dcaro>	 np, make sure to follow the first steps https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart#Get_access 
[08:10:01] <dcaro>	 (if you haven't yet)
[08:10:02] <jminguillona>	 we have a script to scrap some wikipedia web pages regularly, we were told that tooolforge was the best place to do so
[08:11:55] <dcaro>	 that sounds good yes, make sure to explain a bit there in the request what you use those scrapes for and how it helps the movement (and if you have the script code somewhere a link to it, everything running on toolforge must have an FLOSS license and be published, you can use gitlab.wikimedia.org for that if you want, once the tool is approved you can automatically create a repo there)
[08:12:27] <dcaro>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules
[08:13:03] <dcaro>	 note that publishing your code anywhere else is ok also as long as it's public (ex. github, gerrit, ...)
[08:14:48] <jminguillona>	 ok I just made the request, with some info about the project
[08:14:51] <jminguillona>	 thanks a lot!
[08:16:52] <arturo>	 jminguillona: request has been accepted
[08:18:24] <dcaro>	 jminguillona: make sure to subscribe to https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ to get notices of platform changes and maintenance windows and such
[08:18:56] <dcaro>	 give it a minute also to ssh, as it has to create your tool home dir, etc. xd
[09:13:45] <Southparkfan>	 andrewbogott: yeah, it was too late for me I guess, sorry :)
[09:15:31] <jminguillona>	 thanks a lot, I already logged in!!!
[09:28:16] <Southparkfan>	 !log deployment-prep create deployment-jobrunner05 with Bullseye image, T370487
[09:28:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[09:28:20] <stashbot>	 T370487: Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487
[09:33:03] <Lucas_WMDE>	 !log wikidata-dev wikidata-icinga: shut down instance, will try to bring up a replacement on newer base OS
[09:33:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[09:35:27] <Lucas_WMDE>	 !log wikidata-dev wikidata-icinga-2024: instance created, will try to set up using https://github.com/tarrow/wmcs-icinga/tree/master/infrastructure
[09:35:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[09:44:58] <Lucas_WMDE>	 !log wikidata-dev wikidata-icinga-2024: set up instance (see https://github.com/tarrow/wmcs-icinga/pull/1), wikidata-icinga hopefully no longer needed
[09:44:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL
[11:03:27] <Southparkfan>	 !log deployment-prep switched over cpjobqueue (running on deployment-changeprop-1) to deployment-jobrunner05 T370487
[11:03:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[11:03:31] <stashbot>	 T370487: Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487
[14:31:15] <andrewbogott>	 Southparkfan: do you have gerrit patches I can review? Or is there anything else I can do to help w/your rampage?
[14:33:23] <Southparkfan>	 andrewbogott: https://phabricator.wikimedia.org/T370487 has two patches
[14:34:04] <Southparkfan>	 and https://phabricator.wikimedia.org/T369919 has one patch
[14:34:33] <Southparkfan>	 after merging, I can delete the old jobrunner (controlled, to not break the beta sync-world job again) and release one floating IP
[14:34:43] <Southparkfan>	 and remove one RRset from the wmflabs.org zone
[14:35:34] <andrewbogott>	 Southparkfan: what about that '-1 do not merge yet' comment?
[14:35:42] <Southparkfan>	 which one?
[14:35:49] <andrewbogott>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1055394
[14:36:03] <Southparkfan>	 yeah, because it was a WIP
[14:36:14] <andrewbogott>	 but no longer, right?  So you can unset?
[14:36:16] <Southparkfan>	 yep
[14:36:21] <Southparkfan>	 cpjobqueue is now using jobrunner05
[14:36:26] <andrewbogott>	 (gerrit sort of doesn't want me to merge while it's set WIP)
[14:37:45] <andrewbogott>	 ok, +2'd all of those
[14:56:40] <andrewbogott>	 so, chown -r
[14:56:57] <Southparkfan>	 hm
[15:01:48] <Southparkfan>	 alright, I have to go in a few minutes for a party, but two new appserver nodes are being set up
[15:02:17] <andrewbogott>	 nice!
[15:05:50] <Southparkfan>	 southparkfan@deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud: Permission denied (publickey).
[15:05:53] <Southparkfan>	 sad
[15:09:18] <andrewbogott>	 I did the thing with puppet certs on the local puppetserver and am re-running puppet, maybe it'll sort out logins
[15:09:32] <andrewbogott>	 (It has a ton of work to do, though, because appserver)
[15:10:28] <andrewbogott>	 hmmmm or maybe ssh requires some special group membership for those hosts?
[15:10:44] <andrewbogott>	 My personal key doesn't work either
[15:12:19] <Southparkfan>	 I don't have time to look at it right now
[15:13:00] <Southparkfan>	 but we're down two Buster machines now, and after fixing the login issue moving the two appservers should not be impossible either
[15:13:57] <andrewbogott>	 hm, personal key works on the old app servers (11,12) but not on the new ones (13,14) so something interesting is happening :(
[15:14:16] <andrewbogott>	 I'll leave a note here about what I find
[15:35:24] <wm-bb>	 <deadbf> Is sqlite actually the best option for my tool? Inserts/updates are rare and very unlikely to be concurrent, reads are not highly concurrent and I'd cap it at 3 queries/s at maximum. Currently the .db file is less than 1MB and I don't expect it to grow much larger. I read on wikitech that sqlite would be using NFS which should have performance problems, but at first I
[15:35:24] <wm-bb>	 used the to <clipped message>
[15:35:25] <wm-bb>	 <deadbf> olsdb but it was slower
[15:37:19] <andrewbogott>	 Southparkfan: fixed now, some kind of foul-up with cloudinit
[15:38:07] <Southparkfan>	 thanks!
[15:39:58] <andrewbogott>	 deadbf, sqlite would probably work for you in the short term but the less you rely on NFS the better. If things are working with toolsdb best to stick with that.
[15:43:55] <wm-bb>	 <deadbf> Thanks for the reply, andrewbogott. I moved off from toolsdb because it was quite slow for reads. It takes about 0.5s for a single query while it appears to be instant for sqlite. I'm suspicious that most of the latency came from the network connection to the database, but I haven't done extensive testing.
[15:45:19] <andrewbogott>	 I would not expect it to be nearly that slow but haven't tested recently
[15:56:25] <Southparkfan>	 andrewbogott: could you +2 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1055464 ? :)
[15:57:09] <andrewbogott>	 yes but you should also go to your party
[15:57:52] <Southparkfan>	 indeed, that's why I'm working in the train
[15:58:21] <andrewbogott>	 ok, fair excuse
[16:06:39] <Southparkfan>	 yep, puppet repo doesn't want to sync yet again
[16:06:57] <Southparkfan>	 despite chown -R using gitpuppet and oneshot-ing puppet-git-sync-upstream
[16:07:52] <Southparkfan>	 complaining about permission errors yet again :) I guess that python sync script should do a better job of su'ing as gitpuppet
[16:10:05] <andrewbogott>	 yeah, I'm doing some tests there now
[16:17:21] <andrewbogott>	 it's in sync again for the moment
[16:56:39] <Southparkfan>	 !log deployment-prep switching trafficserver backends from mediawiki11 and 12 to 13 and 14 - T361387
[16:56:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[16:56:44] <stashbot>	 T361387: Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387
[16:56:59] <Southparkfan>	 so far so good
[17:01:37] <Southparkfan>	 andrewbogott: ^ going to stop for today, the buster appservers are still used in a few places, but I'll take care of that later
[17:01:58] <andrewbogott>	 sounds good, thanks again for working on this
[17:02:10] <Southparkfan>	 as far as I can see the references to the former appservers are solely in horizon hiera, shouldn't be too hard once I find more time
[17:02:35] <Southparkfan>	 my pleasure
[17:03:49] <Southparkfan>	 (but no guarantees for the future!)
[17:07:57] <andrewbogott>	 you can grep horizon hiera by checking out https://gerrit.wikimedia.org/r/admin/repos/cloud/instance-puppet,general
[17:27:41] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 4757a1f91e (l100n updates: de, nb, uk)
[17:27:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[17:28:03] <wm-bb>	 <lucaswerkmeister> wow, that’s like 10× the amount of updates!!
[17:32:25] <wm-bb>	 <lucaswerkmeister> (in case anyone was wondering if those messages are logged automatically or written by hand :P)
[19:00:53] <ebernhardson>	 What would be an appropriate ip range to identify request that come from within the cloud infra and received by beta-cluster? This is related to test deploying https://www.mediawiki.org/wiki/Extension:NetworkSession
[19:01:45] <ebernhardson>	 It will end up creating an account, but the account will only have the 'read' permission so it shouldn't really matter
[19:05:23] <ebernhardson>	 the main goal of the ip range is to be able to clearly see it working in one place and rejected in another
[19:05:50] <mutante>	 ebernhardson: https://phabricator.wikimedia.org/P66844
[19:05:58] <mutante>	 ebernhardson: https://gerrit.wikimedia.org/g/operations/puppet/+/51110996bfafcaff929c78bd5b14bf5d4938b2bf/modules/network/data/data.yaml
[19:06:07] <mutante>	 in that second link, see lines with "cloud-hosts"
[19:06:17] <ebernhardson>	 mutante: excellent, thanks!
[19:08:39] <mutante>	 also line 29 with that 172.16.0.0/12
[21:18:03] <bd808>	 !log tools.ircservserv Created https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config via Striker
[21:18:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ircservserv/SAL
[21:18:11] <bd808>	 !log tools.ircservserv Created https://gitlab.wikimedia.org/toolforge-repos/ircservserv via Striker
[21:18:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ircservserv/SAL
[21:20:49] <jhathaway>	 andrewbogott: unfortunately it looks like taavi's 8438917b952480ec20c2cf6667a1d6e9e07e1973 commit is breaking ci
[21:20:54] <jhathaway>	 https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster/5820/console
[21:29:54] <jhathaway>	 andrewbogott: is reverting safe?
[21:30:30] <andrewbogott>	 jhathaway: yes, go ahead and revert
[21:30:56] <jhathaway>	 ok, will do thanks
[21:31:03] <andrewbogott>	 how is it affecting CI though?  Doesn't it only apply on cloudvirts?
[21:31:49] <andrewbogott>	 huh
[21:31:56] <jhathaway>	 If you look at https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster/5820/console
[21:32:02] <andrewbogott>	 Yeah, I see.
[21:32:04] <jhathaway>	 you see that some of the rspec tests are now failing
[21:32:07] <andrewbogott>	 I don't follow yet but it's fine to revert
[21:32:17] <jhathaway>	 ok
[21:32:25] <jhathaway>	 reverting..
[21:34:18] <jhathaway>	 reverted, thanks
[21:35:12] <jhathaway>	 I've been bitten by the rpsec tests one to many times