[07:47:02] Morning! [07:47:06] o/ [08:37:38] bd808: I'm taking on T435223 since I still want to start shutting down the old cluster today-ish as announced [08:37:39] T435223: Migrate storage to new OpenSearch v2 Toolforge cluster - https://phabricator.wikimedia.org/T435223 [09:30:59] stashbot and friends are now running on the new elastic cluster [09:30:59] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [09:33:02] which means we can go ahead with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328539 [10:55:46] and one final cloud-announce post about the shutdown: https://etherpad.wikimedia.org/p/toolforge-opensearch-announcement [11:26:09] taavi: lgtm, maybe mention for how long it will be around if we know [11:30:10] dcaro: i'm thinking of taking dumps of the indexes onto some durable storage before deleting the old machines so the answer is probably 'for a while' but i don't want to commit to keeping them around indefinitely in public [11:31:00] sounds good [11:31:22] so sent [11:31:32] 👍 [15:25:59] thilp: I'm mulling over adding the job definition to the job created event log, and it turns out that loki's structured metadata can't be nested or other than `dict[str, str]`, so it might have to go somehow in the message instead :/, I think that for now might be ok that the message is something like "started job : {definition}" [15:26:43] the alternative is to stringify (json dump or similar), and put it as value in a metadata field, but that does not give much more value (yet at least) [15:27:19] metadata fields have limits too though, so would have to be careful there too [15:28:02] yes, I’d rather consider json dump directly as the message in that case [15:28:16] one of the metadata fields could be "message type" then [15:29:23] metadata are more useful for filtering, and we wouldn’t filter by nested data [15:30:13] never say never xd [15:30:53] sometimes we should :D [15:34:17] thilp: do you mind checking T432570? I think it matches this [15:34:18] T432570: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570 [15:36:01] that looks good to me! 👍 [15:54:34] taavi, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1318779 is ready for your eyes again (that's the kick-network-when-down thing) [16:25:52] If we want a WMCS project fully shut down, do we need to file a Phab task? I can’t see a means to do it in horizon (I shut down all the instances last week). [16:26:20] James_F: yes please. [16:26:24] Ack. [17:24:01] * dcaro off [17:24:04] cya tomorrow!