[09:03:47] ejoseph: meeting? [09:15:36] errand, back in a bit [10:49:20] I just had "I don't remember visiting this place" moment with https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/745206 :D [10:49:52] it happens :) [10:50:00] errand&lunch [11:01:28] lunch [11:56:39] errand&lunch [14:07:28] dcausse: how do we get the proper timestamp for the kafka reset after streaming updater bootstrap? [14:08:41] zpapierski: from the dump in hdfs, see the hive query in https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#First_run_(bootstrap) [14:09:16] just switch `date` = '20210718' and wiki='wikidata' to the right dump date and "commons" [14:10:29] ah, cool, thx [14:12:04] greetings [14:14:15] o/ [14:17:22] o/ [14:23:13] inflatador: I think that elastic2051 is still tagged as active in netbox, can you move it to failed? [14:27:36] gehel done! [14:27:53] thanks! [14:30:26] ottomata dcausse: did we already discuss change the retention on wcqs mutation topic? [14:31:07] zpapierski: we discussed together on wednesday but we still need to ask Andrew to do it :) [14:31:29] yep, that's me asking :D [14:32:37] does that topic exist already, actually? [14:34:35] I don't think so since we never started the pipeline on kafka-main [14:34:41] but did not check [14:53:38] inflatador: I might be a couple minutes late for our meeting I need to pick up my son [14:54:18] helllloOoOOO [14:57:26] dcausse no problema, just ping me whenever you're ready [15:00:49] ottomata: hey - first question - how to create a topic without pushing anything to it, second question - can I ask you to change the retency policy on it once I make use of the knowledge gained from the first question :) ? [15:01:15] to create a topic, you need to log into the kafka host and run kafka topic --create ... [15:01:20] smae for changing the retention [15:01:28] oh, I can do it by myself? [15:01:29] i can do, maybe make a phab ticket for paper trail? [15:01:33] i dunno, probalby not? :) [15:01:41] actually you probably could via the kafka API...if that worked [15:02:07] I have even two topics, one even mentions you - https://phabricator.wikimedia.org/T296470 [15:02:22] I think you can use that one - normally running the updater would create that topic [15:03:02] I'm not sure if mutation topics have a dc prefix, now that I think about it [15:03:35] ok, yes they do [15:03:42] hum! okay [15:03:49] so two topics, one per DC [15:03:55] looking [15:03:59] (all mentioned in the ticket as well) [15:04:18] k, and it only needs 30 day retenion on main cluster, right? [15:04:38] we do mirror everythign to jumbo [15:04:47] main is fine [15:04:49] but guessing you don't need extra retention there? [15:04:50] kay [15:05:14] going to create the topics and do https://wikitech.wikimedia.org/wiki/Kafka/Administration#Alter_topic_retention_settings [15:09:44] zpapierski: done [15:09:49] https://phabricator.wikimedia.org/T296470#7604918 [15:09:55] awesome, thank you ! [15:33:51] inflatador: still want to chat about puppet today? [15:42:08] gehel Y, ready when you are [15:42:31] meet.google.com/jsq-eqct-ryk [16:00:47] \o [16:06:03] o/ [16:07:01] o/ [17:13:06] quick workout, back in ~30 [17:48:24] sorry, been back [18:01:02] week-end time, see you next week! [18:19:13] btw, regarding upgrading deployment-prep elasticsearch instances: https://phabricator.wikimedia.org/T298252#7587752 [18:29:30] ooh that sounds cool [18:54:19] we don't have much left in our nginx configuration for elasticsearch, so it should not be too difficult to replace it with envoy, as long as it works both on deployment-prep and production [19:33:23] lunch, back in ~30 [20:19:34] and back [21:00:39] lunch [21:40:01] back [23:15:54] Does anyone know how to reach our ES cluster in the "deployment-prep" env? I found relforge hostnames in the puppet repo, but not deployment-prep [23:20:25] inflatador: are you in the deployment prep project? [23:20:46] Not sure, I am a new hire [23:21:01] How can I tell? [23:21:23] inflatador: https://ldap.toolforge.org/user/bking [23:21:29] it looks like you are [23:22:15] That refers to an openstack project? [23:22:21] it is yes [23:22:23] inflatador: yup, specifically https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep [23:22:31] the ES nodes are deployment-elastic0[5-7].deployment-prep.eqiad.wmflabs [23:22:50] you'll be able to ssh in using your cloud VPS key [23:23:11] Thanks, I was looking at ebernhardson -linked article but there's no host info there ;) [23:23:36] i used puppet https://github.com/wikimedia/puppet/blob/production/hieradata/cloud/eqiad1/deployment-prep/common.yaml [23:23:45] https://github.com/wikimedia/puppet/blob/production/hieradata/cloud/eqiad1/deployment-prep/common.yaml#L213 [23:24:08] Deployment prep isn't well maintained though sadly so [23:24:27] got it, believe me when I say I have experience in poorly-maintained Openstack environments ;P [23:24:39] deployment-prep/beta in general is [23:24:44] no one 'owns' it [23:24:50] inflatador: one oddity is going to be that for production all hosts are listed in that site.pp file, in in wmcs it's much more self service, roles are selected commonly through the horizon UI (horizon.wmflabs.org). Deployment-prep though is yet more special, in that it's more integrated to prod puppet but still awkwardly different [23:25:01] https://openstack-browser.toolforge.org/project/deployment-prep has all instances on [23:26:40] hmmm I'm getting NXDOMAIN for deployment-elastic0[567].deployment-prep.eqiad.wmflabs [23:27:13] ah, I added the .org and it's at least resolving [23:27:21] s/eqiad.wmflabs/eqiad1.wikimedia.cloud/ [23:27:40] inflatador: the .eqiad.wmflabs are kinda historical, as taavi mentioned it's been replaced by real names that can resolve :) [23:27:47] otherwise you need ssh config magic [23:28:11] you need it anyways to use the bastion [23:28:25] ahh, yea i suppose [23:28:32] ah good point, I assume there is a 'cloud bastion' separate from the prod bastion [23:28:37] yes [23:29:09] OK, I see https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances [23:29:31] yeah I was just about to link that [23:30:15] OK, time to conjure up some SSH magic [23:30:35] taavi: do wikimedia.cloud work in exchange for .wmflabs if it's never been at .cloud ? [23:30:45] * RhinosF1 just copied from puppet [23:31:31] we created wikimedia.cloud names for all existing VMs, so in theory yes (and I've never had ssh config for wmflabs) [23:32:00] but you might have issues with tls certificate names or similar, which forces you to be careful [23:32:14] ok [23:32:57] for context: before recently, we had ~3 different things all called "Labs", and we're trying to get rid of that term now [23:34:14] no, ~4 [23:34:26] LOL [23:37:45] https://wikitech.wikimedia.org/wiki/Help:Labs_labs_labs [23:38:01] we were also told we couldn't call something the relevance lab :P [23:39:19] Another generic tech term that's so overloaded that it's meaningless? NAAAH ;) [23:47:27] OK, calling it a week! Thanks everyone for your help and see ya Monday [23:48:05] i'm taking off too. take care