[09:00:12] dcausse: i'm traveling back home [09:00:22] I wont be able to join by 10 [09:06:05] ejoseph: ok, feel free to re-schedule [10:32:06] sigh... just upgraded my system and now I get "send_pubkey_test: no mutual signature algorithm" when accessing gerrit :/ [10:36:15] might be related to ssh-rsa deprecation... [10:45:55] was T253824 in the end, added the workaround and it worked [10:46:02] T253824: planned upstream deprecation of the ssh-rsa signing algorithm (RSA with SHA-1) - https://phabricator.wikimedia.org/T253824 [10:46:32] nope I meant T276486 [10:46:33] T276486: gerrit's sshd is incompatible with RSA pubkeys + Fedora 33 clients (and future versions of OpenSSH proper) - https://phabricator.wikimedia.org/T276486 [10:58:56] lunch [13:18:48] greetings [13:45:36] o/ [14:19:29] dcausse LMK if you have time to look at some puppet stuff with me, trying to plumb thru 'default_java_version' var https://gerrit.wikimedia.org/r/c/operations/puppet/+/787505/3/modules/elasticsearch/manifests/instance.pp#b175 which comes from https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/java.pp#L45 but it's not working [14:41:42] inflatador: sure, looking [14:43:37] dcausse thanks! If it helps, it lkooks like we pull in the java profile here: https://github.com/wikimedia/puppet/blob/e1787fadbb2764a4036f2c4fcab79000861696c7/modules/profile/manifests/elasticsearch.pp#L37 [14:46:27] inflatador: quickly looking I don't think you can access $default_java_packages as it's a local var of the java profile [14:48:18] Ah OK. Could we add it as a lookup as we're doing in the profile above? https://github.com/wikimedia/puppet/blob/e1787fadbb2764a4036f2c4fcab79000861696c7/modules/profile/manifests/elasticsearch.pp#L21 [14:49:15] afaics this info is pulled out of the $facts var which seems global [14:51:13] not sure I know puppet enough... but naively I would inspect this $facts var again but this is kind of repeating the same logic in another place... [14:52:19] perhaps a function can be extracted? [14:56:34] this java profiles seem to support multiple jvm being installed [14:56:54] but the elasticsearch manifest assumes that only one is installed I guess [14:57:33] yeah, I'm looking at the logstash class?module?profile? (not sure of terminology) and it looks like they're doing something smilar [14:58:04] modules/logstash/manifests/init.pp:32 [14:59:47] https://github.com/wikimedia/puppet/blob/e1787fadbb2764a4036f2c4fcab79000861696c7/modules/logstash/manifests/init.pp#L32 [14:59:56] ok so they pass the java package explicitely [15:02:07] oh looks like you can access this via $profile::java::default_package_name [15:02:33] this might just work? [15:03:12] ref modules/profile/manifests/logstash/gelf_relay.pp:21 [15:03:35] awesome, will take a look. Thanks! [15:07:04] unsure but you might have to add new arg that will passed to modules/elasticsearch/manifests/init.pp:21 and then to elasticsearch::instance. Because you might only access this $profile::java::default_package_name from a profile [15:08:26] usually $::profile::java::default_package, although maybe that `contain profile::java` line brings it into scope without th eleading :: [15:08:54] but yea, what i would expect puppet needs is to make the reference across profiles in a profile, passing it down through the elasticsearch module until it gets to elasticsearch::instance [15:43:04] FYI: going directly from Dublin to the Giant's Causeway is a little over 3 hour's drive one way. So that would definitely be an all-day event. [16:06:26] quick workout, back in ~30 [16:31:44] dinner [16:40:49] back [16:47:07] ebernhardson dcausse Sorry for being late, I'm in the puppet meeting now. If you do need anything merged, let me know [16:49:17] inflatador: i dont think i have anything this week [16:50:02] Cool, hit me up anytime if you do need anything [17:44:32] lunch, back in ~45 [18:19:59] back [19:03:39] gehel ryankemper (or anyone else) what is the "right" number of disk for WDQS hosts? ref https://phabricator.wikimedia.org/T280382 . Looks like we have a mix of 4 and 8 disks . Newest deployed have 8 disks, but newest quoted but not deployed have 4 disks. And to make it extra confusing, oldest deployed have 4 disks [19:04:30] https://phabricator.wikimedia.org/P26916 shows current disk layouts [19:05:43] inflatador: so far, we haven't cared much for the number of disks, but we care about the overall available disk space. [19:05:56] looks like we can either have 3.5 TB of RAID10 (8 disks) or 2.6 TB of RAID0 [19:06:30] (4 disks) [19:06:47] Older hosts are raid0 as a way to increase disk space as disk usage has grown faster than expected [19:06:52] blazegraph data is ~1.1T ATM [19:07:12] checking grafana for growth rate [19:07:54] We have enough redundancy in the cluster so that raid0 is not a big issue. That being said, we'd like to plan for raid10 as much as possible [19:10:06] existing quote is for 4x 1.92TB , that'd give us ~1.92 TB at RAID10 or ~8TB at RAID0/JBOD . Is that OK or should we get a new quote? [19:10:50] I have no problem using raid0 or raid1 , looks like we have enough space for it although I haven't checked storage growth rate yet [19:15:13] looks like storage growth rate isn't in dashboard yet , I think we should just move fwd with the current 4x disk quote. gehel or anyone else, let me know if that is not OK [19:17:11] 1.9TB in raid10 seems low. That does not even 2x the current journal [19:17:42] And from memory, that's less than the newest WDQS servers. [19:18:23] gehel agreed, I was thinking of using RAID0 again, but if that is not OK, I can ask for a new quote with 8x disk/server [19:19:39] How urgent is that quote ? Can we discuss next week when I'm back from vacation ? [19:19:51] * gehel only has a phone at the moment [19:20:10] rob-h msg from yesterday says "Please advise, as we have to place this to order (not just approvals, but to order) for any chance of it arriving this fiscal year." [19:20:30] I will ask him if it can wait [19:20:38] Then let's move it forward ! [19:21:23] We want to have as much storage as the latest servers we added to the cluster. We don't care about the number of disks. [19:22:34] OK, do we care about RAID level? RAID0 or RAID1 will get us 8 or 4 TB respectively, RAID 10 2 TB [19:22:40] Depending on the current market prices, it might make more sense to have larger disks instead of more disks. Rob should know. [19:23:33] How do you get less space on raid10 ? [19:23:59] RAID 10 is stripe+mirror , so 1/4 of total disk space [19:24:16] RAID1 is 1/2 total disk space, RAID 0 (JBOD) is all, or RAID 0 (stripe) is 1/2 [19:24:57] In the content of the current quote, which is 4x 2 TB disks [19:25:47] I think you should check your calculations ! [19:26:47] http://www.raid-calculator.com/default.aspx gives results that match my expectations [19:27:42] agreed, double everything I just said, except for RAID1 [19:27:52] which we can ignore [19:28:11] 4x 2TB in raid10 should give us 4TB of usable storage, which is on par with what we have. So that should be good [19:28:34] Cool, sorry for rousting you. [19:29:02] See you next week! [19:38:58] Enjoy ! [20:08:19] Hi team! I usually do my product team updates on Thursday afternoons, and I use standup notes to help with that. Could you add any updates if you haven't? thanks! [20:27:26] mpham apologies, will get on that [20:49:24] out a little early, 1/2 day tomorrow...see you Friday afternoon (my time)!