Skip to main content

Enforcing Policies with OPA Gatekeeper

Context

The configuration of Kubernetes suffers from several vulnerabilities. For instance, any developer can escalade his privileges using this tutorial

here

With k8saas, the common risks are reduced by builtin security mechanisms, like:

  • all accesses are managed using Central Authentication (Azure Active Directory with Multiple Factor Authentication Enforced).
  • Role Base Access Control (RBAC) is enabled on Kubernetes. Meaning, no-one has full admin access to the cluster (except level 2 / core team).
  • By design, the cloud resources are dedicated to a project (logical isolation and limited transversal permissions). Rebound to another cluster is very complex.

To improve the security of our clusters, k8saas has put in place from Atik 1.18 release, enforced policies. From Babel 2.0, the management of these policies has moved from Azure Policy for AKS to OPA gatekeeper

img

Use case

  • Perform a pentest to test the robustness of k8saas
  • Protection against credentials steeling
  • Protection against malware in kubernetes

What to do ?

Nothing ! The service is enabled by default.

Concept

Before Babel 2.0, the k8saas policy was based on Azure Policy for AKS. The rational was to use "ready-to-use" feature properly integrated with Azure.

Azure portal provide IHM to manage policies.

here

Azure Policy for AKS uses gatekeeper of the Open Policy Agent Project.

Limitations:

  • Azure Policy for AKS supports officially only azure builtin rules
  • Azure Policy for AKS is complex to debug. During the engineering phase, the iterations takes too many times

From Babel 2.0, k8saas has decided to use the same product (behind azure policy for AKS), ie Gatekeeper; but deployed, configured and monitored directly by k8saas team.

List of policies enabled by default for k8saas:

  • no-privileged-container
  • block-node-selector
  • ingress-host-max-lenght
  • nginx-ingress-classname

List of available policies:

k8saas policy nameDescriptionEnable by default
No-privileged-containerControls the ability of any container to enable privileged mode.
Corresponds to the privileged field in a PodSecurityPolicy. For more information, see
https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged
true
all-ns-must-have-gatekeeper-labelPut a gatekeeper label (tutorial)false
block-node-selectorrefuse pod with node-selectortrue
ingress-host-max-lengthLimit the host length when submitting ingress objectstrue
containers-shall-only-use-allowed-imageslimit the usage of specific pre-allowed artifact repositoriesfalse
nginx-ingress-classnameBlock the legacy ingress class name annotationtrue

HOWTO

Disable the policy for a specific cluster ?

Please send a request on the TrustNest K8SaaS Service catalog with:

  • the cluster used by the project
  • the reason
  • the contact of your security partner & Product Owner

Ask for a new policy ?

Please send a request on the TrustNest K8SaaS Service catalog with:

  • the cluster used by the project
  • the list of policies to enabled
  • the contact of your security partner & Product Owner

Monitor my policies

There are 2 useful dashboard that you can use to monitor your policies:

  1. K8saas / K8SAAS / OPA Gatekeeper

This dashboard shows you the type of policies deployed on your cluster (deny, warn etc...), and if they have blocked some deployment on your cluster.

img

  1. Flux Cluster Stats

This dashboard shows you the kustomizations deployed on your cluster, including the policies. With the naming convention and the description of the available policies (table just before) you can understand what are the deployed policies.

img

Especially, on the left table:

img

TROUBLESHOOTING

How to know when I'm blocked by a policy ?

When submitting a pod (manually or from a deployment/statefulset), you should see a log like:

Error from server ([denied by azurepolicy-container-no-privilege-6134f47be9a69edbacdd] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}
[denied by azurepolicy-container-no-privilege-7f83897b6cbbf9956e71] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}): error when creating "noderoot.yml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by azurepolicy-container-no-privilege-6134f47be9a69edbacdd] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}
[denied by azurepolicy-container-no-privilege-7f83897b6cbbf9956e71] Privileged container is not allowed: noderootpod, securityContext: {"privileged": true}

Those logs could be in the events:

$ kubectl get events -n <your_namespace> -w

Next Steps

  • Learn on the Open Policy Agent language to understand builtin policies. doc
  • Discover the AKS builtin policies doc