[16:11:02] good news, it seems that we solved the bugs with s3/apus/ceph, the docker registry's restricted backend seems to work as expected (all the hanging etc.. was related to the radosgw s3 daemon in need of a restart on the apus hosts) [16:19:27] nice! [16:30:33] I am a little concerned that all the problems were "solved" with a radosgw roll restart, since from what we know it can happen anytime [16:30:47] that is not really great, and there was no indication in the logs about what was happening [16:31:12] maybe something worth to discuss in the ceph sig [16:43:14] FYI: the ml-build1001 machine can't login to the registry if we only allow it push to the /v2/ml namespace since. when docker logs in /v2 is called so we might need to add the ml-build docker registry user to the `regular-push.htpasswd` file