[16:03:45] moritzm: still on to try the logstash cookbook? [16:06:14] sure, running it now, ok? [16:07:45] Sounds good! [16:09:14] it's running now with the default arguments (batch size 1) [16:10:21] * cwhite watches grafana [16:12:43] it started with 2023 and it's trying to recover the "OpenSearch shard size check - 9200" [16:12:50] currently at attempt 11/15 [16:13:39] The shard size check won't leave warning state right now. [16:15:17] ah, so unrelated to the current restart, right? [16:15:23] right [16:16:02] then we're actually running into https://phabricator.wikimedia.org/T319277 [16:17:37] so it would it hit the max retries (15, with some linear retry interval increase, the last one waits 42 seconds) and then it interrupts the restart to interactively ask to proceed or abort [16:18:08] given that we have another 11 nodes, I think I'll rather abort the cookbook, then, would be a little tedious otherwise :-) [16:19:04] but seems that the restart per se of a single node worked fine [16:19:50] so that's good to know, although the cookbook would be of little use until T319277 is implemented or the shard check fixed [16:21:15] Yeah, that sounds right to me [16:22:15] It will be a while before that check is fixed, but it is being worked on (https://phabricator.wikimedia.org/T327308)! [16:23:42] ok, we can just do another smoke test whenever T327308 or T319277 are fixed