[15:24:16] I have two concerns about this email that I'm planning for Monday: [15:24:29] https://etherpad.wikimedia.org/p/floating-ip-aliaser [15:24:44] 1) Is that explanation even remotely comprehensible? [15:25:05] 2) Is that edge case (listening on an internal IP and not the public IP) serious enough that I should give notice before the change [15:25:38] I have kind of a hard time believing that #2 would be much of a thing, and even if it is that anyone would anticipate it until something breaks... [15:41:19] Rook or dhinus I'd appreciate thoughts about ^ if you have 'em (and are working today) [15:44:51] Reading... [15:47:51] I think it is worth mentioning. Perhaps leaving a hint in the tldr that would point people to what is happening in case they do have some service that could be impacted and would be inspired to read on? [15:50:21] The long did read description is comprehensible to me, though I may not be an ideal sample [15:54:12] do you think I should give notice to people to change their listening addresses before I roll out the change? [15:54:31] andrewbogott: I had to read it a few times but I don't know how to make it easier... I'm tempted to just remove the last paragraph, as it's just a speculation but if things do break, they might do it in more bizarre and unpredictable ways :) [15:55:13] ok, so it sounds like you think the listening-ip issue is no more likely than random-unanticipated-other-thing? [15:55:30] yes that's what I mean, but it's just my 2c [15:57:01] ok, thank you! [15:58:40] re: notice, maybe I would schedule this for next week, a few people are still on holiday [15:58:57] oh yes you planned the email for Monday [15:59:42] yeah, I'm traveling this weekend, don't want to leave things broken :) [16:00:22] I would probably send the email Monday, but do the change later next week, just in case somebody reads it and realizes magically they need to fix something :) [16:00:53] ok! [16:01:53] andrewbogott: a service can't be listening on a floating IP address, since the target VM always sees its own private address and never the floating one [16:02:05] oh great! [16:02:14] * andrewbogott strikes that paragraph [16:03:27] hey, since I briefly have you here taavi, can I ask you about the horizon updates you made when I was on sabbatical? I'm wondering if you found the state of the top-level deployment repo to make sense, or to not make sense [16:03:53] for example, I tend to make new branches when I merge with upstream code, but I could just have a single 'main' branch and delete everything else [16:04:32] I was looking at rolling out a patch, and became confused by my own branches and then thought, oh god, david and taavi tried to understand this while I was away :( [16:04:58] the branches were somewhat confusing, it seemed like different repos were using different ones and none had the actual branch in use marked as the default branch [16:05:40] also i changed the deploy repo to publish an image on push to the default branch since i got tired of manually making tags there [16:06:14] Oh! that's interesting, so we no longer need a puppet change to deploy? [16:06:28] Does that mean that we can't deploy to codfw1dev for testing before deploying to eqiad1? [16:06:54] no, you need to update the docker image tag in puppet as before [16:07:22] what i mean is that the docker image tag is generated from the commit timestamp like in most wikimedia repos that publish containers, instead of requiring a git tag for each deploy [16:07:41] oh, ok [16:08:03] I think that I don't ever pay attention to/think in terms of 'default branch' so it doesn't surprise me that that was inconsistent. [16:08:24] How would you have me organize all of that, in a perfect world? [16:09:45] either a single 'main' branch with the currently live code, or per-openstack-version branches but with the old branches periodically cleaned and the default branch as the currently deployed one, i think [16:10:09] probably a single main branch would be the cleanest, since we don't tend to be running multiple versions at once very often [16:10:19] ok [16:10:39] that was how I was leaning yesterday as well. [16:11:19] I guess the submodules could still have branches since we aren't usually deploying openstack head [16:15:32] * andrewbogott makes T382957 [16:15:46] thx taavi [17:19:36] I fell into a rabbit hole yesterday that ended up mostly resurrecting the https://striker.wmcloud.org/ demo server. Things work until you try to actually approve a join request. At that point the keystone problem from T359972 makes everything go boom. [17:19:36] T359972: Dev environment Keystone crashes - https://phabricator.wikimedia.org/T359972 [17:44:46] what is the actual interaction with keystone that kills things? [17:50:55] the keystone container in the striker development environment segfaults when you try to add a member to a project [17:51:45] rude [17:52:41] especially since keystone is just python, so why a segfault? [23:02:29] Summary of my afternoon: puppet pki never fails to surprise me :(