Replies: 11 comments 80 replies
-
| One thing to clarify here: This is just for new resources, not for new fields on graduated resources, right? |
Beta Was this translation helpful? Give feedback.
-
| If you had an experimental API in the |
Beta Was this translation helpful? Give feedback.
-
| @JoelSpeed that's correct. I'll rephrase this to clarify that it would avoid those errors when anyone was trying to install standard channel CRDs. Edit: Actually already had "when upgrading from experimental to standard", open to other ways to clarify this. |
Beta Was this translation helpful? Give feedback.
-
| I am not sure I understand how this proposal solves that problem. The schema cannot change between versions without a conversion webhook...? |
Beta Was this translation helpful? Give feedback.
-
| This proposal ensures that installing standard channel CRDs will always be safe and smooth at the cost of removing any theoretical in-place upgrade path from experimental to standard channel. They would be entirely different resources with different API groups and thus the upgrade path would be to create equivalent standard channel resources. |
Beta Was this translation helpful? Give feedback.
-
| This does not solve the "breaking changes need a conversion web hook to not be breaking changes" problem, and isn't intended to. As Rob says, this is currently only working towards making sure that Standard is always safe, and that using Experimental can't affect that. This is not the last step, to be clear, it's the first. But we want to see if this approach is worth the cost to implementation and users before pushing too hard. |
Beta Was this translation helpful? Give feedback.
-
| @robscott would be good to also gather feedback from implementations on who will support this new |
Beta Was this translation helpful? Give feedback.
-
| Yeah, thanks for the clarifications here and on chat. I am good with this provided we do as much as possible to make experimental truly unstable, notably the expectation that implementations should not support experimental and v1 concurrently (I am not sure if we can somehow enforce this, since its kind of a prisoners dilemma -- see companies selling LTS, once one does it (for $$$ usually) there is an expectation others do it too). I do, however, have concerns about the proposals that use this as a stepping stone, around "Kubernetes comes with the CRDs enabled by default and you cannot get experimental fields". That certainly doesn't need to be linked to this decision though. |
Beta Was this translation helpful? Give feedback.
-
Do you mean concurrently in the sense of "actively for a single installation within a cluster" (this feels like it might be possible to enforce with conformance tests, and could possibly encourage safer patterns like using dev/staging clusters for testing experimental functionality), or "within a project's codebase"? |
Beta Was this translation helpful? Give feedback.
-
| @mikemorris the big concern with splitting groups (from a maintainer POV) is the cost to support 2 groups is orders of magnitude larger than to support 2 versions of the same group (since versions are identical). So I really don't want to get into a world where an implementation has to deal with that. The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable). Implementations supporting v1.5 should replace xFooRoute support with FooRoute support. tl;dr "within a projects codebase" |
Beta Was this translation helpful? Give feedback.
-
I remember this coming up before and I think there was some consideration that it might be possible to work with SIG API Machinery or improve tooling to make this easier?
That wouldn't solve for new experimental fields on existing stable resources though I think? |
Beta Was this translation helpful? Give feedback.
-
Correct -- this proposal is not covering that though |
Beta Was this translation helpful? Give feedback.
-
| Posted in slack but I'll surface this here Circling back to the discussion on which group |
Beta Was this translation helpful? Give feedback.
-
| Yeah this specific proposal doesn't really provide isolation for experimental fields. My previous proposal that did (#3106) was not particularly popular, so I'm trying to start smaller here. |
Beta Was this translation helpful? Give feedback.
-
| Agreed this is a bit awkward for features spanning across new resources and additions to existing resources, where perhaps either resources from the x-k8s.io group would be available but only alongside standard channel resources, or new fields on existing experimental channel resources in the newest release of the main group may become available without support in the implementation for x-k8s.io resources. This could lead to "dead fields" as @shaneutt describes in #3576 (comment) |
Beta Was this translation helpful? Give feedback.
-
| To clarify, the changes I'm proposing here are quite tiny. The only difference from today is that new resources in experimental channel would start in a different experimental API group. Everything else about experimental channel would stay the same. In practice that means that installing experimental channel would continue to get you resources were fully experimental, and resources that were GA with experimental fields. Although I'd like to pursue broader changes like a full separation between release channels to eliminate problems like dead fields, this specific proposal is really small. |
Beta Was this translation helpful? Give feedback.
-
| Gotcha, that makes sense - I was jumping ahead a bit to thinking about which resources a cloud provider might allow or install by default on a cluster. |
Beta Was this translation helpful? Give feedback.
-
| The APIs represent a common way to solve a specific problem. If 2 implementations use the same API - and 15 don't implement it - it is still better than the 2 implementations using different API, and if it is a rarely used feature - it is better than attempting to force the 15 others to implement it, and a good signal that the feature is not important enough to be part of the standard. My main concern is with calling this 'experimental' - like we called a lot of APIs 'alpha' before but still encouraged users to use in production. Should be treated just like vendor-specific v1 APIs, with the main difference that more than 1 vendor is using the same API. I used to be very enthusiastic about this proposal - but right now I think a better approach is to do nothing, and let each implementation define their own v1 API - perhaps with some communication and discussion on a common behavior. Let them try different things, get feedback, wait to see adoption and convergence - and only then attempt to standardize. Defining any standard API (or protocol) by committee, before having real world experience is never working well. …On Wed, Jan 29, 2025 at 5:55 PM Rob Scott ***@***.***> wrote: Yeah this specific proposal doesn't really provide isolation for experimental *fields*. My previous proposal that did (#3106 <#3106>) was not particularly popular, so I'm trying to start smaller here. — Reply to this email directly, view it on GitHub <#3497 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2S67EXLO4WRTOHXS432NGBBRAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBQGE4TENY> . You are receiving this because you commented.Message ID: <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12001927@ github.com> |
Beta Was this translation helpful? Give feedback.
-
| Not sure how supporting 2 groups is 'orders of magnitude' - I would expect at worst double, and no worse than what Istio does with HttpRoute and VirtualService. All implementation I know have vendor extensions - and I would expect they would remain supported, not thrown away even if a subset or variant of the extension is adopted ( and this is likely to be on a permanent basis for most implementations - expecting that upstream implements every single features that any vendor needs is not viable). Maybe the confusion is 'experiments' ( which should not be used by anyone in production ) versus features like FooRoute - that some vendors may adopt and maintain on their own for their customers. There is no expectation that a vendor FooRoute will go away and be replaced by an upstream FooRoute, just like there was no expectation that VirtualService will be deprecated when HttpRoute was v1. Since 'experiments' should never be in production - I am not at all concerned with their fate, as long as we avoid the istio confusion between alpha and v1. The (single or multi-vendor) FooRoute can be v1 and can have more fields or behave differently - and may have its own vendor-defined support lifecycle, and since common APIs are expected to be implementable by everyone - we can't expect implementations that can support more features to drop that. …On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***> wrote: @mikemorris <https://github.com/mikemorris> the big concern with splitting groups (from a maintainer POV) is the cost to support 2 groups is orders of magnitude larger than to support 2 versions of the same group (since versions are identical). So I really don't want to get into a world where an implementation has to deal with that. The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable). Implementations supporting v1.5 should *replace* xFooRoute support with FooRoute support. tl;dr "within a projects codebase" — Reply to this email directly, view it on GitHub <#3497 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI> . You are receiving this because you commented.Message ID: <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@ github.com> |
Beta Was this translation helpful? Give feedback.
-
| supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂 I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo |
Beta Was this translation helpful? Give feedback.
-
| I get where you are coming from and would agree a lot more if that is what was being proposed - but it doesn't seem to be. |
Beta Was this translation helpful? Give feedback.
-
| In other words: I read x-k8s.io as "eXtended" - not 'eXperimental'. A collection of gateway API extensions that have common definition across few vendors - and are as v1 as the core APIs - but can't be supported by majority/all vendors, or are too corner cases. If some of those extensions become common or popular - the same or slightly different API can be added to core, of course - and the vendors will have a 2x effort to keep both. …On Fri, Jan 31, 2025 at 5:14 PM Costin Manolache ***@***.***> wrote: Not sure how supporting 2 groups is 'orders of magnitude' - I would expect at worst double, and no worse than what Istio does with HttpRoute and VirtualService. All implementation I know have vendor extensions - and I would expect they would remain supported, not thrown away even if a subset or variant of the extension is adopted ( and this is likely to be on a permanent basis for most implementations - expecting that upstream implements every single features that any vendor needs is not viable). Maybe the confusion is 'experiments' ( which should not be used by anyone in production ) versus features like FooRoute - that some vendors may adopt and maintain on their own for their customers. There is no expectation that a vendor FooRoute will go away and be replaced by an upstream FooRoute, just like there was no expectation that VirtualService will be deprecated when HttpRoute was v1. Since 'experiments' should never be in production - I am not at all concerned with their fate, as long as we avoid the istio confusion between alpha and v1. The (single or multi-vendor) FooRoute can be v1 and can have more fields or behave differently - and may have its own vendor-defined support lifecycle, and since common APIs are expected to be implementable by everyone - we can't expect implementations that can support more features to drop that. On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***> wrote: > @mikemorris <https://github.com/mikemorris> the big concern with > splitting groups (from a maintainer POV) is the cost to support 2 groups is > orders of magnitude larger than to support 2 versions of the same group > (since versions are identical). > > So I really don't want to get into a world where an implementation has to > deal with that. > > The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had > xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable). > Implementations supporting v1.5 should *replace* xFooRoute support with > FooRoute support. > > tl;dr "within a projects codebase" > > — > Reply to this email directly, view it on GitHub > <#3497 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI> > . > You are receiving this because you commented.Message ID: > <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@ > github.com> > |
Beta Was this translation helpful? Give feedback.
-
| On Fri, Jan 31, 2025 at 5:23 PM John Howard ***@***.***> wrote: supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂 I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo Right, most vendors will implement APIs defined by another vendor - and that is the problem, each vendor defining a XXXFoo API that is slightly different from each other - all v1 and supported but subtly different. Having a common gateway-extensions.io/Foo is far better for users, if 2-3 vendors agree on a common Foo. From the 'core' API perspective, each single vendor extension and each multi-vendor extension are 'experimental', in the sense that there is no clear evidence it is a universal problem or consensus across all vendors. I think that is the main purpose (and benefit) of vendor extensions, it allows vendor to 'experiment' by releasing features and APIs. If x-net.io/Foo (v1) was defined - maybe both Istio and GKE and other could implement it, users can adopt such a feature with confidence it will be supported as a v1 API, and when a core Foo API is defined we can use the lessons learned from x-net.io/Foo and make changes as needed. Istio implementing ExperimentalFoo - and dealing with migrations and users who (as usual) adopt ExperimentalFoo in production ignoring the alpha and experimental labels - is indeed order of magnitudes harder than support an x-net.io/FooV1 or an istio.io/FooV1 API long term and not dealing with any migration. |
Beta Was this translation helpful? Give feedback.
-
Alternative ProposalWhat?I propose an alternative solution which I will call "Full Copy". In summary this solution suggestions that we need to move to having all APIs copied to a new experimental group, rather than just new ones.
Why?In the past we've lived in a world where an implementation manages the CRDs. Increasingly since GA we live in a world where these APIs are deployed and managed by platforms, and more and more treated like core APIs as multiple implementations need to be using the same APIs. In this future where platforms need to serve Gateway API to multiple implementations, I do not see how we can effectively "overlay" experimental fields with the standard APIs. This means that fields may or may not be present on a "core-like" API and this is not a good situation to be in. Ultimately too, I think the complexity of one API potentially being one, or a very different experimental version on the same production cluster will drive platforms to block experimental to control API surface. How?High Level Steps
Other details, questions and considerations
To keep things cleanly separated we will use separate Go types. To avoid unintended drift between types and to ensure that experimental remains a true super set of standard, we will add CRD schema checks/comparisons to CI (crd-schema-checker supports this today. It can be used as a (Go) library and there's desire to move it into upstream).
We will make even stronger guarantees about this new group being "Experimental" in a more classical sense of the word, and less "Almost Standard". We will do this by updating our GEP process and "probationary period" to reflect a faster on-ramp and off-ramp. This means that we will optimize for moving things into experimental status quicker, but we also will be strict about when things need to move out (or change significantly). As such all APIs in this new group will be Platforms will be advised to avoid managing the experimental CRDs if feasible. Implementations will be advised to only enable experimental on clusters where they're certain to have clean control of the experimental CRDs.
If it meets our criteria for promotion, it gets manually copied over to the corresponding standard types in a new feature branch. To help reduce situations where "something that wasn't happening in experimental with a specific field is now happening on standard", conformance tests need to pivot over to the new standard implementation and submit reports prior to merging the feature branch into After that's complete the process continues as normal. Generators are run and the fields get published in an upcoming release.
We do not support references between standard and experimental API types. We advise implementations to keep those cleanly divided (e.g. don't support attaching an |
Beta Was this translation helpful? Give feedback.
-
I am sure there are other options that could work, but if I were to chose I would, yes. The users of my project do not have issues with experimental - at all.
I would hope that if I don't at all think this is the intent, but just to view this from my perspective, this comes off as the platform saying "I see you are perfectly happy with the status quo, but I am not, so I am going to force the APIs to change to the point where they are unusable". Why are we so keen to push a problem that (IMO) only impacts platforms onto everyone else? Why not consider the platforms taking on the burden of supporting the community? (To re-iterate, just showing my raw perspective -- I 100% understand that is (hopefully) not the actual intent). A related question - are the platforms that are pushing for this change even going to implement all the experimental API types? I don't think I have seen any here actually indicating they would. If not, it comes across more like "removal of experimental via making it unusable so I no longer need to deal with it" than "improving experimental".
The cost is indefinite. Parallel controllers doing the ~same thing, forever. I don't care about a one time cost, I would easily spend 168 hours if it solved all of our problems and then was done, but its not the case at all. The cost is ongoing and will come at a risk to our users. We have concrete examples of doing this in the past in Istio, with Ingress v1 promotion (for other APIs we just cut over at once, but Ingress had too short of a window from v1 addition to beta removal). There it wasn't so bad since Ingress code ~never changed, and it had a lifetime of a couple releases before we could clean it up. Even in that time, we had a few bugs related to the split. Gateway would be far different, since its one of the most actively changed areas of our codebase and has no "end" time when we can converge.
I've considered a few of these but ultimately don't see any that aren't at risk of destabilizing our stable users (which is, of course, our top priority and something we cannot compromise on) |
Beta Was this translation helpful? Give feedback.
-
| Ultimately, Gateway API was about a bunch of implementations coming together to share a common API and unify the ecosystem around this. Its disappointing to see this shatter into a second sub-ecosystem with a mirror universe of APIs that will fragment the ecosystem we worked hard to build. |
Beta Was this translation helpful? Give feedback.
-
I appreciate that clarification, unfortunately I agree with Nick in that I'm struggling to see how it would be tenable.
Well if it helps, I do come from the perspective of a platform (OpenShift) and so I can speak directly from my own personal perspective which will hopefully be helpful: As Gateway API has become ubiquitous we have a requirement to ensure multiple implementations (and other projects that rely on Gateway API) can all work on the same cluster. It is not reasonable for one single implementation to be managing the life-cycle of these CRDs anymore, or users just installing them from the web, because at best other implementations could not consistently rely on what will be present for them, and at worst things could become broken. Naturally this means the platform has to take responsibility, and treat the CRDs as "core-like" APIs so that implementations can reliably depend on the API. OpenShift will be doing this very soon as GKE has been doing for some time. So back to experimental specifically: From the perspective of the platform I work on we can actually accommodate any of the solutions to the experimental problem ("Full Copy", "New Only", or "Do Nothing") with relative ease, we just wont be providing any level of support for experimental (which is exactly how experimental should be so it's almost not worth saying). Suffice to say it's more with my maintainer hat on and a desire to protect the user and implementer experiences that I have this conversation. And from that perspective I see it as nearly the opposite of it "only impacts platforms", I see it as "only impacting implementations and users of experimental". Given the above: In a future where "Do Nothing" can't happen (which implies separate groups happens regardless), I struggle to see a slow transition to the new group where things can be in two places is the best overall solution for everyone involved. For users I think the separation might mean they don't have to play hell to get the experimental CRDs in place on a cluster which already has the standard ones, so maybe that's nice. For implementations I rather expect they'd prefer to have one way to implement the experimental features, and not have to accommodate both the experimental channel AND the experimental group (maybe this is particularly true for new implementations, or implementations that are less invested in experimental), though I'm keeping myself aware that some implementations anticipate having to do a significant overhaul and we must include that in the discussion. The platform can easily accommodate either, but which would be best for all the users and implementations involved? Still an unanswered question in my mind.
Indefinite, as in the cost continues to be paid? 🤔 Assuming we're already transitioned though, and kinda in a "steady state" the main thing that will come up is "new experimental field X has been added to resource Y, I need to add support". In that instance, the workflow is:
Aren't those steps pretty similar in either case? I mean I understand we have the initial upfront cost to convert, and when the field finally gets promoted you have to copy from experimental over to the standard implementation, but otherwise is it kind of a wash?
Ok, but if we were able to come up with a stable alternative to the "extra controllers" implementation, would this significantly improve your feelings towards having separate groups? |
Beta Was this translation helpful? Give feedback.
-
Being a force for unification means we must work to ensure fairness for everyone we aim to unify. It's costly, but we owe it to them to work together, share perspectives and talk openly to find the most equitable solution.
I am truly sorry that it pains you in this way, I don't want that for you. It was us who did this in the past though and it's not just a couple of us anymore with a few users, it's like 30 implementations and craploads of users now, so our responsibilities and costs to keep receiving the value are definitely different now. As has been said, these are ALL bad options and we're not trying to sell any of them as good, we just want "least bad" and we need your help to make sure we pick right. |
Beta Was this translation helpful? Give feedback.
-
| I'm certainly late, as this whole GH discussion has become a huge rabbit hole, but I feel I need to add my two cents about the proposal. I feel like I am a bit in a middle ground: as a maintainer, I need to advocate and endorse the decisions that are best for our users and the API itself, while as a person who works on implementation, I hear the concerns about putting the burden of this decision on the implementation's shoulders. I won't repeat all the concepts, the concerns, and everything as I wouldn't be able to do a better job than what already all of you have done listing all the pros and the cons and discussing them. I agree with the analysis that @shane made about the available options we have and with @youngnick when he says that "no action" here is a non-viable option. So, the other two options remain. The "new only" option is a step ahead for sure, but from both the user's and our perspective, it's not sufficient in my opinion. I'm very aware of the burden posed on implementations, but I do think that the "full copy" solution is the only one that really improves the situation and has some chance to at least alleviate all the problems already broadly discussed here. However, I understand @robscott's point about having the "new only" as a stepping stone to the broader vision. I am totally fine with that, but I think the path we are getting into should be made clear and our final destination well defined. |
Beta Was this translation helpful? Give feedback.
-
| I think we are still using the word 'experiment' in a very confusing way. In your '3 kinds of experiments' - for 1 you mention 'are used in production', so perhaps you mean the same thing as I do, a v1 vendor extensions that are considered as an 'experiment' for the core API, in the sense that other vendors and the WG waits to get feedback on how the v1 vendor feature behaves and what adoptions it gets. Assuming we mean the same thing - vendor v1 APIs, that multiple vendors support but in slightly different ways - having a common API, even if not part of this WG - is a good thing for users, and it doesn't really matter if Gateaway API is planning or not to adopt it in core, so (2) is the same as (1), stable v1 APIs that users can safely rely on with no risks or problems, but is outside of this WG but still cross-vendor. (3) is the only tricky one - if Gateway API decides to adopt an API from (1) or (2) category, there will be some pain for the user in using both the original (1/2) API and (3) at the same time - but it is reasonable to assume the vendor API will remain a super-set of the Gateway API and remain supported long-term by the vendors. The only source of pain is if a user touches any 'alpha'/experimental CRD in prod, and that is something every vendor and K8S community should do a better job of preventing and educating users about. …On Fri, Feb 7, 2025 at 11:20 AM Arko Dasgupta ***@***.***> wrote: @kflynn <https://github.com/kflynn> the cost is high for both cases - full-copy and new only proposals Regarding Ana 1. if envoy gateway doesnt implement the full copy or new-only proposal, because it already supports the feature using 1. or 2. then Ana shouldnt mind, because her use case is solved 2. If many users voice the fact that they want to use the Gateway API version of the feature, that may be a consideration for Envoy Gateway but its a stronger signal for Gateway API to move the feature into standard im just trying to highlight how this situation we are in, is not a win-win one for the API group and implementations — Reply to this email directly, view it on GitHub <#3497 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2SP6FOK34FCXDQMFPT2OUBQLAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBZHAYDMMI> . You are receiving this because you commented.Message ID: <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12098061@ github.com> |
Beta Was this translation helpful? Give feedback.
-
| I think the only real problem is legitimizing the use of unstable, experimental APIs in production environments. Any user who understands production is not for experimentation and no alpha API should be touched outside of test clusters will be safe. As long as implementations understand that once they declare an API or feature as 'ready for production' or 'v1' they can't remove fields - but they can create new stable APIs or use v1 APIs from a 'standard' group while maintaining the API they used - users will be safe, and there is little pain for implementations. …On Mon, Feb 10, 2025 at 4:04 AM Joel Speed ***@***.***> wrote: I was trying to think about what the copy all approach gains us in terms of our ability to make breaking changes, so ran through some thoughts on what it would like with some changes on Gateway Full Copy XGateway Field is added and dropped Lets assume in this instance we add a field Foo, but later decide that we want to drop it in v1.N - On upgrade to v1.N, any cluster that has been leveraging Foo will have data populated in the field, but implementations will not read the field - Any write to Foo will wipe the persisted data in Foo - Dropping a feature *is* a breaking change, but we are in experimental, so who cares? - Gateway has not been affected as XGateway had the field first Field is added but we want to change the shape Lets assume in this instance we add a field Bar first as []string but now want []struct - This change *needs* a version bump, or a new name - So we have to move XGateway from v1alphaN to v1alphaN+1 - If we do not bump the version, then serialization issues could ensue, which is no fun for anyone - We have to implement conversion between the two alpha versions and support that - Or because this is experimental, we bump the version but don't implement conversion - This leaves users to have to remove their gateway API install before they can upgrade - Gateway has not been affected, XGateway v1alphaN is now dead, impls need to now all move to v1alphaN+1 - When we do eventually promote the feature, Gateway gets the correct shape first time Adding new fields to Gateway Field is added and dropped Lets assume in this instance we add a field Foo, but later decide that we want to drop it in v1.N - On upgrade to v1.N, any cluster that has been leveraging Foo will have data populated in the field, but implementations will not read the field - Any write to Foo will wipe the persisted data in Foo - Dropping the feature *is* a breaking change. This is a stable resource and while the feature is marked experimental, we don't know how all implementations are handling it(?) - The risk here seems way higher here than it does for the other approach, but again we would argue that the field was experimental and we reserve the right to drop it - Gateway can NEVER re-use this field name if we come up with some reason to later - How do we make sure that we never re-use the field name? Field is added but we want to change the shape Lets assume in this instance we add a field Bar first as []string but now want []struct - The field must be dropped (see above) and then introduced under a new name - We have no option to change the shape without breaking clients - Again we have a need to understand how to make sure we never use a name again ------------------------------ So to me, there's a couple of things we probably want to work out where we are ok with this. For the copy all approach, when we decide we need to make a change of shape to an API, are we going to be ok bumping the API to a new version, and what's the impact of that? Would we rather just say, "name is spent" and move on in this case? And if so, how would we track spent names? If we go down the "name is spent" route, then the benefits of the copy all approach for existing APIs, in terms of our ability to make breaking changes are lessened. Though I do see other benefits to the copy all approach, from the end user and platform management perspective. A reduction in the dead fields problem or stale fields waiting to be pruned problem is certainly a nice benefit. — Reply to this email directly, view it on GitHub <#3497 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUR2UY6IQ6UVAXSXQA7632PCIWVAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMJRHAYDCOI> . You are receiving this because you commented.Message ID: <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12118019@ github.com> |
Beta Was this translation helpful? Give feedback.
-
| (I've been slow to write here, but I have been trying to keep up.) We're at a point in this discussion where the various parties appear far enough apart that it's worth sitting back and asking where the disconnects are, and where we're making assumptions that aren't shared, so let me take a quick crack at getting that started -- and let me apologize in advance for any places where folks think I'm misunderstanding y'all. We agree on a lot of things. We all seem to want to have a place for experimentation with the API to happen, we all seem to recognize the value of having real users actually use the experimental version, and I think we all agree that the standard version is the one that's most important for actual users. We also seem to agree that we're expecting managed Gateway API from the cluster providers soon, and that we don't expect the managed offerings to include the experimental channel. (Anyone who doesn't think these are points of agreement, please speak up!) On the other hand, we seem to have a substantial disconnect around something I would've expected to be a point of agreement: whether we expect the experimental channel to be stable. That probably sounds silly on the face of it, given the many times that we've said that experimental can receive breaking changes -- but historically, that's happened only rarely, and in fact there was a point before Gateway API had a stable release where I remember many of us (including myself) downplaying the idea that experimental was unstable, because we wanted folks to use Gateway API at all. It's not hard to see this kind of history leading folks to see more stability in experimental than we've actually promised. And now, this discussion includes folks who seem to be saying both that their users are fine using experimental, and also that they don't want to destabilize those users. To me, those two together imply that the users are fine with experimental because they believe that the experimental APIs are in fact stable APIs (apparently to the point of being in production with them). (That may seem like a bit of a stretch, so for more context: if your users of experimental APIs are truly expecting breaking changes, then they won't consider breaking changes destabilizing. A user who finds breaking changes destabilizing is a user who didn't expect breaking changes... which means that they think they're using a stable API.) I want to be clear here that I'm not trying to pass judgment on these users. All I'm saying here is that as long as we don't agree even on whether or not experimental is meant to be stable, we're going to talk past each other till the cows come home, and that won't really get us very far. I think that there are other disconnects worth discussing, too, but this is the one that seems most fundamental to me. More on the others later. |
Beta Was this translation helpful? Give feedback.
-
| This analysis resonates a lot with me, and I share the same concern you are talking about here. It seems that users are fine with the experimental channel, and probably it's like that because of the very low amount of breaking changes we've introduced, or maybe it's our fault that haven't been sufficiently vocal that experimental is unstable. From the Gateway API survey, the 42.5% of the responding users use the experimental channel (37 out of 87 respondents). It's a huge number in my opinion as if we had data about the number of users enabling FG |
Beta Was this translation helpful? Give feedback.
-
| I very much appreciate this refocus Flynn, it resonates strongly with me as well. |
Beta Was this translation helpful? Give feedback.
-
| Thanks Flynn, I agree. A lot of the point of making changes here is to make it so we actually can make breaking changes in Experimental. Right now we basically can't, because as you say, it's basically stable with extra steps. We've ended up at the current status quo because, historically, we've been choosing the things that are easiest for implementor and power users who are following the discussion, or are prepared to accept the risks of using Experimental APIs in production. This has some really nice properties that @howardjohn and other folks have gone over before - implementations can use the single Go type for both Experimental and Standard, and once you add code to handle the Experimental things, there are no code changes to move things to Standard. So, when we move things to Standard, there are no changes for implementations, and if any users were using Experimental, then there are no changes for them either. This is a really nice place to be for the happy path where we never make any API mistakes. What we have today is easy for both implementors and power users, and none of those folks have felt the pain that those of us proposing the changes are worried about. So it's easy to discount and only see the pain of the new approaches ("The current state is not that big a deal"). Maybe the most important thing is to write down what I (and others) see as the nightmare scenarios, and compare and contrast the different approaches. Then we can have a more fair comparison about if the significant additional engineering cost, both upfront and ongoing, is worth it. I'm going to think about that some more. |
Beta Was this translation helpful? Give feedback.
-
| Agreed on all counts, @youngnick, especially about how the history makes it really easy to think about the happy path and discount the nightmares. 😐 I really like the idea of writing down the nightmare scenarios, actually: there's nothing like digging into those to unearth where we're assuming things that others may not be. I'd love to help get that ball running -- though I'm out next week, I should be able to do some async work there. |
Beta Was this translation helpful? Give feedback.
-
| That's a fair point @youngnick; enumerating those nightmare scenarios will certainly help me to more accurately weight the pros and cons here. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
As a follow up to #3106, I'd like to propose a smaller subset of that to address many of the same problems:
Importantly this means that when a new resource graduates to GA, users would need to manually copy their config over from x-k8s.io (alpha) to k8s.io (v1) CRs. Although this is rather onerous, I'd argue that it's preferable to the current experience which can result in confusing error messages when trying to install Gateway API standard channel. This new approach would ensure that it was always safe to install standard channel CRDs. For example, this could allow Kubernetes to include Gateway API standard channel by default (x-ref #3576).
I don't really like 3) here as it means we're not solving the full set of problems that fully separate API groups for experimental channel and standard channel would, but this more limited approach does come with some advantages. Notably, controllers would not need to support both experimental and standard channel API groups at the same time for the same resource. Instead, they could transition to the
k8s.ioAPI group at the same time the API does. If they want to support a broader set of versions, they can support both API groups for a limited period of time, but importantly any Gateway API release would only ever include a single API group (and version) per resource.Note: All mentions of API groups here are just focused on the suffix. In all cases, they would also include the
gateway.networking.prefix they already do.Beta Was this translation helpful? Give feedback.
All reactions