[06:54:20] <dcausse>	 clear failed airflow tasks from the week-end, opensearch-semanticsearch@codfw is having issues looking
[07:21:20] <dcausse>	 ben will look into it
[12:11:44] <dcausse>	 sigh... somehow the indices switched back to qwen3....
[12:14:21] <dcausse>	 bah ofcourse I forgot to delete the DagProp variable from the airflow db...
[12:32:27] <dcausse>	 re-importing all this
[13:12:32] <inflatador>	 <o/
[13:16:15] <dcausse>	 o/
[14:08:35] <ebernhardson>	 \o
[14:10:34] <dcausse>	 o/
[16:21:52] <dcausse>	 dinner
[17:00:15] <ebernhardson>	 curious...maybe i need to keep waiting.  applied image update to opensearch-semantic-search-test and it hasn't restarted yet
[17:03:18] <inflatador>	 ebernhardson did you run the cirrus toolbox script? The operator isn't smart enough to restart on its own after an image change ;(
[17:12:30] <ebernhardson>	 inflatador: oh! i didn't realize that. but makes sense...i guess it's been awhile since i restarted any of these
[17:12:41] <ebernhardson>	 will try out the toolbox command, thanks!
[17:26:04] <inflatador>	 Cool, let me know if you gives you any problems
[19:18:57] <ebernhardson>	 hmm, this is going to be tedious...i wonder if we need to somehow change the opensearch build
[19:19:31] <ebernhardson>	 the problem is, the docker-registry.wikimedia.org/repos/data-engineering/opensearch image builds from a base image that has a snapshot, meaning we can only build the cirrussearch-opensearch-image on top of it for 7 days after the image under the opensearch image was released
[19:19:46] <ebernhardson>	 otherwise it fails with: Error: Release file for http://snapshot.debian.org/archive/debian/20260906T040000Z/dists/trixie-updates/InRelease is expired (invalid since 8d 17h 5min 46s). Updates for this repository will not be applied.
[19:24:15] <ebernhardson>	 i think (currently testing) fix is to re-trigger publish on the latest opensearch image so it rebuilds and publishes...but annoying
[19:27:36] <ebernhardson>	 sigh, no now it says:  Release file for http://snapshot.debian.org/archive/debian-security/20260914T053800Z/dists/trixie-security/InRelease is expired (invalid since 1d 7h 14min 32s). Updates for this repository will not be applied.
[19:29:31] <ebernhardson>	 i feel like something is fundamentally wrong with our setup with this new restriction, if there is a chain of images from different repos they have to all build in the same week, but we don't publish that frequently. 
[19:46:42] <ebernhardson>	 oh i'm a dumbdumb...it's not the opensearch image, it was the base trixie image which we pin