Guide: giving each team its own namespace¶
banlieue controller serves only the namespaces you choose
(ADR-0130).
A team gets a namespace, you label it, and the controller starts reconciling
that team's VirtualMachinePool, VirtualMachineClaim and VirtualMachine
objects. Nothing is served until it is chosen.
This is scope, not isolation
The selector decides which namespaces are served. It does not stop a
team naming another team's VMClass, VMImage or Provider selector, and
it applies no quota. Isolation today is the namespaced Role you give each
team; admission policy and quota per tenant are a follow-up decision. See
the accepted risks in the threat model.
How the scope is chosen¶
In order, the first that applies wins:
| # | Setting | Serves |
|---|---|---|
| 1 | --namespace-selector / BANLIEUE_NAMESPACE_SELECTOR |
every namespace whose labels match |
| 2 | --namespace / BANLIEUE_NAMESPACE |
exactly that namespace |
| 3 | neither | the namespace the controller runs in |
The controller never defaults to every namespace. If it cannot find any of the three it refuses to start. The startup log names the rule that applied:
A selector the API server cannot parse also stops startup, with the server's error.
Onboard a team¶
-
Set the selector on the controller. In
deploy/controller/configmap.yaml: -
Create the team's namespace and label it. The label is what opts it in:
-
Give the team a namespaced
RoleandRoleBindingfor its own objects. Use aRole, never aClusterRole. -
If the team's
VirtualMachines name aspec.userDataSecret or ConfigMap, let the controller read them there. The controller resolvesuserDatain the VM's own namespace, and the shippeddeploy/controller/rbac/role.yamlgrants that only inbanlieue-system. Copy thatRoleand itsRoleBindinginto the team's namespace (changemetadata.namespace, keep the subjectbanlieue-controllerinbanlieue-system). Without it, such a VM reports auserDataresolution failure; a VM with inlineuserData, or none, needs nothing here. Thevirtualmachine-userdata-authorizationadmission policy (ADR-0042) still decides which Secrets a team may name.
The controller notices the label at once: no restart, and everything already in the namespace is reconciled.
Who may label
Do not give tenants update or patch on namespaces. That permission
is what lets a namespace opt itself in. Select on a label only a platform
admin can set.
If the controller's own namespace holds the Providers, it must carry the
label too, or the controller will not see them.
Take a team out¶
The controller stops reconciling that namespace. Nothing is deleted, and
status is left as last written, so a VirtualMachine there can still show
Ready while it is no longer being watched. Deleting an object still works:
its finalizer runs even in an unserved namespace.
Permissions the controller needs¶
Selector mode lists and watches Namespace objects. The shipped
deploy/controller/rbac/clusterrole.yaml grants get, list and watch on
namespaces and nothing else; the controller cannot relabel a namespace. If
that grant is missing the controller fails at startup and never reports ready.
The operator and the providers¶
The controller is one of three binaries that read a namespace scope
(ADR-0131).
A team's Providers live in the team's namespace, so the operator has to
serve that namespace too, or its Providers get no workload.
The operator takes the same three-step choice, with its own flag names
(its --namespace means where it runs, so it is not reused as the scope):
| # | Setting | Serves |
|---|---|---|
| 1 | --watch-namespace-selector / BANLIEUE_WATCH_NAMESPACE_SELECTOR |
the Providers in every namespace whose labels match |
| 2 | --watch-namespace / BANLIEUE_WATCH_NAMESPACE |
exactly that namespace |
| 3 | neither | the namespace the operator runs in (--namespace) |
It no longer watches every namespace by default. To serve tenants, give it the same label as the controller:
If you install with banlieue bootstrap operator, pass it at install time
instead of editing the Deployment afterwards:
Keep the two selectors aligned. If the controller serves a namespace the
operator does not, a VirtualMachine there can schedule onto a Provider
that never gets a workload. Un-labelling a namespace stops the operator
reconciling its Providers but deletes nothing; a Provider being deleted
still has its workload torn down. The operator's ClusterRole grants
read-only get, list and watch on namespaces for this.
The providers keep their credentials in one namespace, their home
namespace (--namespace, or the namespace they run in). That is where their
Provider, credentials Secret and CA bundle live, and they never read a
Secret anywhere else. Every provider can also serve other namespaces
(vsphere, libvirt, proxmox and cloud-hypervisor), which is how one
provider serves many teams without any team ever holding the hypervisor
credentials. See the next section.
One shared provider for every team¶
Putting a Provider and its credentials in each team's namespace would hand
the team's Secret readers the vCenter login. Instead keep one Provider in the
platform namespace and let it declare the namespaces it serves
(ADR-0133).
Everything else stays in the team's namespace: the pool, its VMs, the claims,
the AgentSandboxes and the infrastructure machine CRs. Only the reference
to the provider crosses a namespace.
-
Declare who the
Providerserves. In the platform namespace, on theProviderthat holds the credentials:apiVersion: banlieue.io/v1alpha1 kind: Provider metadata: name: vcenter namespace: banlieue-system spec: providerClassRef: name: vsphere namespaceSelector: matchLabels: banlieue.io/tenant: "true" connection: endpoint: https://vcenter.example.com/sdk credentialsRef: name: vcenter-credsThe operator renders
--watch-namespace-selectoronto the provider from this field, so the provider and the controller read the same selector and cannot disagree. An absent selector serves the home namespace only, and an empty one ({}) serves nothing: banlieue never defaults to every namespace. -
Tell the controller where the shared providers are:
-
Label the team's namespace, as above. A team namespace with no
Providerof its own is now scheduled onto the shared one.
The lookup is: the VM's own namespace first, then the provider namespace. If a
team namespace holds any Provider, those are the only candidates, so a
selector that matches nothing there fails with NoProviderMatched instead of
landing on the shared platform. Only a Provider that declares the VM's
namespace is ever a fallback.
Apply the admission policies
A machine that names a provider in another namespace is a placement that
skipped scheduling. Apply both policies in
deploy/admission/:
machine-provider-ref.yamllets only the controller setproviderRef.namespace, so a team cannot write a machine that names the platform's provider.provider-namespace-selector-authorization.yamlmakes settingnamespaceSelectorneedupdateonproviders/namespaceselector. Bind thebanlieue-provider-namespaceselector-adminClusterRole it ships to platform admins only.
Give teams no verb on machine CRs (vspheremachines, libvirtmachines,
proxmoxmachines, cloudhypervisormachines). They need to see
their pool, VMs and claims, not the machines behind them.
Cloud Hypervisor hosts are opt-in and read the selector at startup
A Cloud Hypervisor provider runs on the KVM host with a bound
ServiceAccount token, not as a pod
(ADR-0134).
It reads its namespaceSelector from its own Provider once, at
startup, so changing the selector needs a provider restart. A host with
no selector keeps a token that reaches machines in its own namespace only.
With a selector, the operator binds a second ClusterRole,
banlieue-provider-cloud-hypervisor-shared, which lets the host's token
read and patch machines in the namespaces it serves (RBAC cannot be scoped
by label, so it can in every namespace). It holds no Secret or ConfigMap.
That is a wider stolen-host-token reach than a single-namespace host has,
so enable it only on hosts that must serve tenants.
What the shared provider can and cannot reach
Its ClusterRole can read and patch VSphereMachines in any namespace and
list and watch Namespaces. RBAC cannot be scoped by label, so the
selector is the gate and the ClusterRole is the ceiling. It has no
Secret or ConfigMap access outside its home namespace, so a team that can
read Secrets in its own namespace cannot read the credentials. A team's
userData is rendered by the controller into the machine in the team's own
namespace; the provider never reads it from a Secret.
Checking it¶
kubectl get namespaces -l banlieue.io/tenant=true # what is served
kubectl -n banlieue-system logs deploy/banlieue-controller | rg serving
A pool created in an unlabelled namespace gets no members and no status update. Label the namespace and the pool fills within a reconcile.