[01:32:42] <inflatador>	 the snapshot failed with errors: https://phabricator.wikimedia.org/P30699
[01:34:25] <inflatador>	 starting it again as snapshot snapshot_t309648_attempt_2
[08:01:27] <gehel>	 dcausse: I'm writing a few Java guidelines. Do we have a standard for HTTP requests? Apache HC? OkHttp? Is there one that we want to push as a standard?
[08:02:50] <gehel>	 eventutilities is using HC, so that is probably already trickling down to a number of projects
[08:09:50] <dcausse>	 gehel: we have this http route thing for calling mw api servers internally, lemme check
[08:11:47] <dcausse>	 but apache HC is the one we the most
[08:11:51] <dcausse>	 *use
[08:18:49] <dcausse>	 if you get to lombok we have nice setups in wdqs/rdf/tools that works well with jackson (e.g. using a minimal @Data object that works well with jackson) 
[08:44:23] <gehel>	 first draft available on https://www.mediawiki.org/wiki/Manual:Coding_conventions/Java
[08:46:16] <gehel>	 Errand + lunch
[10:14:45] <dcausse>	 lunch
[13:01:28] <inflatador>	 greetings!
[13:01:48] <inflatador>	 looks like the snapshot was a success, will update notes and start on the restore shortly
[13:02:33] <inflatador>	 may need some help determining how to point commonswiki_file alias to the latest version of the index if that doesn't happen automatically
[13:24:56] <dcausse>	 o/
[13:34:04] <inflatador>	 <o/
[14:50:17] <inflatador>	 cloudelastic errors suggest it's not using path style access for the S3 repo, doing a rolling restart to (hopefully) enable our custom plugin (which does support path-based access)
[15:27:19] <ebernhardson>	 \o
[15:27:28] <ebernhardson>	 inflatador: the alias will happen automagically on restore
[15:27:50] <ebernhardson>	 (it's stored as part of the index settings)
[15:27:52] <inflatador>	 <o/
[15:28:14] <dcausse>	 o/
[15:28:30] <inflatador>	 cool, will let you know how the snapshot goes then
[16:17:24] <inflatador>	 cloudelastic1006 failing to start its main ES service, not a perms issue...checking it out now
[16:28:29] <inflatador>	 cloudelastic1006 is our only bullseye host, it had something weird going on with the s3 plugin
[16:28:44] <inflatador>	 reinstalling the wmf-elasticsearch-search-plugins package and restarting the services seems to have fixed it
[16:29:59] <inflatador>	 (going to work out for real now)
[16:34:41] <ebernhardson>	 :)
[17:02:07] <inflatador>	 aaand back
[17:23:21] <inflatador>	 Lunch, back in ~1 hr
[19:26:20] <ebernhardson>	 lunch
[20:13:50] <inflatador>	 quick break, brb
[20:28:40] <inflatador>	 back
[20:31:29] <ebernhardson>	 back
[21:28:29] <inflatador>	 ebernhardson I successfully registered the snapshot repo in cloudelastic. Per https://www.elastic.co/guide/en/elasticsearch/reference/7.17/restore-snapshot-api.html , do you think we need to include any params in the restore besides the name of the index to restore? 
[21:49:30] <inflatador>	 ebernhardson went ahead and started the restore from cloudelastic1002, cmd used: https://phabricator.wikimedia.org/T309648#8045498
[21:49:41] <inflatador>	 based on the restore API, seems to be working
[21:54:23] <inflatador>	 I am seeing some "allocation failed" for some shards occasionally
[21:58:56] <inflatador>	 Nothing in explain, so I'm guessing they'll get reassigned
[22:16:29] <inflatador>	 Explain says "primary shard for this replica is not yet active" on a lot of 'em, I imagine that will clear up eventually
[22:28:28] <ebernhardson>	 hmm, i don't remember seeing those last time but if it's only from cloudelastic during the restore seems sane
[22:48:44] <ebernhardson>	 I'm a bit surprised how slow that restore is running, ~50MB/s seems pretty minimal. But i guess it's just waiting and not that big a deal. I just always think they have more in them and like to tune things :P