Support running SDP on k8s with ResourceQuotas #368
Description
This affects all operators and all products. Users reported problems such as
Warning FailedCreate 64m (x43 over 18h) daemonset-controller (combined from similar events): Error creating: pods "secret-operator-daemonset-ndrqx" is forbidden: failed quota: not-terminating-pods: must specify limits.cpu,limits.memory Warning FailedCreate 54m daemonset-controller Error creating: pods "secret-operator-daemonset-6zndn" is forbidden: failed quota: not-terminating-pods: must specify limits.cpu,limits.memory I guess this is because they have set up ResourceQuotas
We need to install a ResourceQuota into a k8s cluster and get our stuff working on it.
### Tasks - [ ] https://github.com/stackabletech/secret-operator/issues/263 - [ ] https://github.com/stackabletech/operator-rs/issues/596 - [ ] #394 - [x] product main containers should be fine, as we currently always configure the request + limit - [x] Deploy default requests + limits for all operator Pods