Per-pod native sidecar enablement

[dwj300]

(This is used to request new product features, please visit https://github.com/istio/istio/discussions for questions on using Istio)

Describe the feature request
We’d like the ability to control which pods use native sidecars, as we run a multi-cluster setup, and not all clusters are running new-enough kubernetes (1.28). Ideally this would be an annotation on the pod spec.

Describe alternatives you've considered
Waiting until all of our workload clusters are running k8s 1.28
Affected product area (please put an X in all that apply)

[ ] Ambient
[ ] Docs
[ ] Dual Stack
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Additional context

[howardjohn]

#48794 could also solve your probably in a different manner. But per-pod is probably also useful

edit: actually not quite, only in some scenarios which probably don't apply

[dwj300]

I think there are (at least) two somewhat natural options for said per-pod enablement (for arguments sake, let's call it injectNativeSidecar):

1. Add it to proxyConfig, similar to HoldApplicationUntilProxyStarts
2. Create a new annotation, similar to sidecar.istio.io/rewriteAppHTTPProbers

Do you have a strong preference between them? I have a slight preference towards the annotation, as proxyConfig is already somewhat of a pain to merge in our tooling, but if required would be fine.

[howardjohn]

My current assumption is native sidecars are going to something that we want ~universally, and we just have some transition pain due to versions. Based on that, a more temporary solution (annotation) seems appropriate. I would call it sidecar.istio.io/native-sidecar: true/false perhaps (the 'sidecar' bit is repeated which is a bit awkward, but 'native' on its own is odd as well).

[dwj300]

note to self, use nativeSidecar instead of native-sidecar

[AndresPinerosZen]

Even if native sidecars are going to be most likely something enabled on the whole cluster long-term, it's important to allow people to enable a few pods initially so they can test the stability. Keep in mind that this is still a beta feature of K8s and it is risky to set the whole cluster without previous testing.

[howardjohn]

Even if native sidecars are going to be most likely something enabled on the whole cluster long-term, it's important to allow people to enable a few pods initially so they can test the stability. Keep in mind that this is still a beta feature of K8s and it is risky to set the whole cluster without previous testing.

This was implemented in #53490 so you should be able to do this.

FWIW, its not easy but you can always test out global settings like this with Istio canary installs as well.