Outshift Logo

PRODUCT

9 min read

Blog thumbnail

by Laszlo Bence Nagy

Published on 06/23/2020
Last updated on 03/21/2024

Deploying Istio with restricted Pod Security Policies

Share

Thanks to the gradual maturation of Istio over its last few of releases, it is now possible to run control plane components without root privileges. We often use Pod Security Policies (PSPs) in Kubernetes to ensure that pods run with only restricted privileges. In this post, we'll discuss how to run Istio's control plane components with as few privileges as possible, using restricted PSPs and the open source Banzai Cloud Istio operator.

Introduction

In order to accomplish what we're setting out to do here - guarantee that Istio control plane components use the fewest privileges possible - we recommended using the Istio CNI plugin. To enforce the restricted privileges in question, we recommended you use Pod Security Policies. Before we get started, let's do a brief overview of these two concepts: Istio CNI and Pod Security Policies:

Istio CNI

The problem

By default, Istio uses an injected initContainer called istio-init to create iptables rules before the other containers in the pod can start. This requires the user, or service-account that's deploying pods in the mesh, to have sufficient privileges to deploy containers with NET_ADMIN capability - a kernel capability that allows the reconfiguration of networks. In general, we want to avoid giving application pods this capability. Istio POD Initialization without CNI Plugin

The solution

One way of mitigating this issue is to take iptables' rules configuration out of the pod using a CNI plugin.
CNI (Container Network Interface), a Cloud Native Computing Foundation project, consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers, along with a number of supported plugins.
The Istio CNI plugin replaces the istio-init container, which provides the same functionality, but without requiring Istio users to enable elevated privileges. It performs traffic redirection in the setup phase of the Kubernetes pod's lifecycle, thereby removing the NET_ADMIN capability requirement for users deploying pods to the mesh. Istio POD Initialization CNI Plugin
For a more detailed explanation on how Istio CNI works, check out our Enhancing Istio service mesh security with a CNI plugin blog post.

Pod Security Policies

Pod Security Policies provide fine-grained authorization for pod creation.
Let's try to quickly summarize how they work:
Note: You must enable the PSP admission controller in your Kubernetes cluster to use PSPs.

Missing Pod Security Policy

If you try to create a pod without a PSP resource attached to its Service Account (by using Role and RoleBinding), then the admission controller will deny the pod's creation. Missing PodSecurityPolicy

Denied by Pod Security Policy

If you try to create a pod with a PSP resource attached to its Service Account, but the pod does not satisfy the constraints set in the attached PSP resource, then the admission controller will deny the pod's creation. PodSecurityPolicy denied

Allowed by Pod Security Policy

If you try to create a pod with a PSP resource attached to its Service Account and the pod satisfies the constraints set in the attached PSP resource, then the admission controller will allow the pod to be created. PodSecurityPolicy
This is the gist of PSPs, but you can find a more in-depth explanation of how they work in an earlier blog post, which includes working examples and a detailed list of the security aspects used to constrain them: An illustrated deepdive into Pod Security Policies.

Deploy Istio with least privileges

Brief history

In Istio 1.5 the Istio control plane was redesigned as a monolith, and istiod was introduced. Per that release, SDS was also simplified and the NodeAgent component began to undergo deprecation. Istio 1.6 continued to make architectural changes by removing all the legacy control plane components. With these alterations, by the time of Istio 1.6's release, there were many fewer control plane components to secure. In the meantime, further efforts were made to run the remaining components without root privileges (e.g. root port ranges and root file paths ceased to be used). Let's take a look at the remaining Istio components in Istio 1.6.

Components with fewer privileges

These are the components that are run without root privileges by default in Istio 1.6:
  • istiod
  • istio-coredns
  • istio-telemetry
  • istio-policy
  • istio-proxy sidecar container
It's now also possible to run istio-gateway deployments without root privileges. Instead of the istio-init container, the Istio CNI plugin can be used in such a way that the NET_ADMIN capability can be dropped. These are the bare minimum privileges necessary for the remaining components. Let's take a look at how we can enforce them with PSPs.

Restrict with PSPs

For all the control plane components (except for the CNI) we can use the following restricted PSP to make sure that the they run only as non-root users. 
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp-restricted
spec:
  allowPrivilegeEscalation: false
  forbiddenSysctls:
    - "*"
  fsGroup:
    ranges:
      - max: 65535
        min: 1
    rule: MustRunAs
  requiredDropCapabilities:
    - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  runAsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    ranges:
      - max: 65535
        min: 1
    rule: MustRunAs
  volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
The CNI component needs a different PSP, since it needs access to the host network and host path volumes, so we'll use this one: 
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp-host
spec:
  allowPrivilegeEscalation: true
  fsGroup:
    rule: RunAsAny
  hostNetwork: true
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
    - secret
    - configMap
    - emptyDir
    - hostPath
With this setup (gateways run as non-root, CNI, PSPs), we can make sure that when the Istio control plane components run, they do so with alleviated privileges.
All the necessary resources, including Roles and RoleBindings, for this setup can be found in the Banzai Cloud Istio operator repo, here.

Try it out!

In this section, we'll setup an Istio mesh with the minimum necessary privileges by using the Banzai Cloud Istio operator.

Create cluster

First, let's create a GKE cluster with Network Policy enabled (for CNI) and pod security policy enabled (for PSP): 
$ gcloud beta container clusters create psp-demo --enable-network-policy --enable-pod-security-policy --cluster-version=1.16.9-gke.2 --node-version=1.16.9-gke.2 --num-nodes=5
NAME      LOCATION        MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
psp-demo  europe-west1-b  1.16.9-gke.2    35.187.123.4  n1-standard-1  1.16.9-gke.2  5          RUNNING

Deploy the Banzai Cloud Istio operator with PSP

We will use Kustomize to deploy the operator and the necessary PSP resources to the cluster. Create a kustomization.yaml file with the following content in any directory: 
bases:
  - github.com/banzaicloud/istio-operator/config?ref=release-1.6
  - github.com/banzaicloud/istio-operator/config/overlays/psp?ref=release-1.6
Apply the resources to your cluster based on that file: 
$ kubectl apply -k .
namespace/istio-system created
customresourcedefinition.apiextensions.k8s.io/istios.istio.banzaicloud.io created
customresourcedefinition.apiextensions.k8s.io/meshgateways.istio.banzaicloud.io created
customresourcedefinition.apiextensions.k8s.io/remoteistios.istio.banzaicloud.io created
role.rbac.authorization.k8s.io/istio-operator-psp-host created
role.rbac.authorization.k8s.io/istio-operator-psp-restricted created
clusterrole.rbac.authorization.k8s.io/istio-operator-manager-role created
rolebinding.rbac.authorization.k8s.io/istio-operator-psp-host created
rolebinding.rbac.authorization.k8s.io/istio-operator-psp-restricted created
clusterrolebinding.rbac.authorization.k8s.io/istio-operator-manager-rolebinding created
service/istio-operator-controller-manager-service created
statefulset.apps/istio-operator-controller-manager created
podsecuritypolicy.policy/istio-operator-psp-host created
podsecuritypolicy.policy/istio-operator-psp-restricted created
Note: If you don't want to use Kustomize, you can deploy the operator through whichever documented method you prefer, and then you can create your own PSP resources tailored to your specific needs.

Deploy Istio with CNI

Create the Istio CR with the Istio CNI enabled. The Istio operator will create the Istio mesh based on this CR. 
$ kubectl -n istio-system apply -f https://raw.githubusercontent.com/banzaicloud/istio-operator/release-1.6/config/samples/istio_v1beta1_istio_cni_gke.yaml
istio.istio.banzaicloud.io/istio-sample created
After a while, your Istio mesh should be up and running: 
$ kubectl get po -n=istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-8kxsm                    2/2     Running   0          3m3s
istio-cni-node-jx7k6                    2/2     Running   0          3m2s
istio-cni-node-mqgkn                    2/2     Running   0          3m3s
istio-cni-node-pdz5q                    2/2     Running   0          3m3s
istio-cni-node-zp98z                    2/2     Running   0          3m2s
istio-ingressgateway-5c5b6c7fdd-bqwn7   1/1     Running   0          2m32s
istio-operator-controller-manager-0     1/1     Running   0          11m
istiod-76fb86d68f-tf5rv                 1/1     Running   0          3m4s

Validating restricted PSPs

Let's make sure that our PSP works by modifying a flag in the Istio CR and trying to run the istio-ingressgateway pod as root: 
kubectl patch istio -n istio-system istio-sample --type=json -p='[{"op": "replace", "path": "/spec/gateways/ingress/runAsRoot", "value":true}]'
After a few seconds you should see that the new istio-ingressgateway pod is unable start: 
$ kubectl get po -n=istio-system -l app=istio-ingressgateway
NAME                                    READY   STATUS                       RESTARTS   AGE
istio-ingressgateway-5c5b6c7fdd-bqwn7   1/1     Running                      0          3m32s
istio-ingressgateway-5d979f4dd4-jqlgq   0/1     CreateContainerConfigError   0          106s
Let's see why: 
$ kubectl describe po -n=istio-system -l app=istio-ingressgateway
Warning  Failed     6s (x10 over 118s)  kubelet, gke-psp-demo-default-pool-7528dec2-kww1  Error: container has runAsNonRoot and image will run as root
The PSP admission controller will not allow the pod to start as root because of the restricted PSP resource we've already created with Kustomize. Let's change the runAsRoot flag back to false so that we can proceed with our demonstration: 
kubectl patch istio -n istio-system istio-sample --type=json -p='[{"op": "replace", "path": "/spec/gateways/ingress/runAsRoot", "value":false}]'

Deploy BookInfo

For BookInfo pods we need to bind a PSP resource such that the PSP admission controller will allow us to create them. Copy the following resources to a file named bookinfo-psp.yaml: 
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: istio
spec:
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
    - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio
rules:
  - apiGroups:
      - policy
    resourceNames:
      - istio
    resources:
      - podsecuritypolicies
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio
subjects:
  - kind: ServiceAccount
    name: bookinfo-productpage
    namespace: default
  - kind: ServiceAccount
    name: bookinfo-details
    namespace: default
  - kind: ServiceAccount
    name: bookinfo-ratings
    namespace: default
  - kind: ServiceAccount
    name: bookinfo-reviews
    namespace: default
Apply the resources based on that file: 
kubectl apply -f bookinfo-psp.yaml
podsecuritypolicy.policy/istio created
clusterrole.rbac.authorization.k8s.io/istio created
clusterrolebinding.rbac.authorization.k8s.io/istio-user created
Run the following commands to deploy the BookInfo application, expose it through an ingress gateway and open its UI in a browser: 
kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n default apply -f https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/networking/bookinfo-gateway.yaml
INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
open http://$INGRESS_HOST/productpage
After the pods start successfully, the application should be up and running: book Details At this point, your mesh is functioning with the minimum necessary privileges, enforced by your freshly created PSPs.

Cleanup

To delete the BookInfo application, the Istio control plane and the Istio operator from your cluster, run the following commands: 
kubectl -n default delete -f https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl -n default delete -f https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl -n default delete -f bookinfo-psp.yaml
kubectl -n istio-system delete -f https://raw.githubusercontent.com/banzaicloud/istio-operator/release-1.6/config/samples/istio_v1beta1_istio_cni_gke.yaml
kubectl delete -k .

Takeaway

As you can see, with the open source Banzai Cloud Istio operator, it's possible to setup an Istio mesh with the bare minimum privileges necessary to run Istio's control plane components, and which enforces those constraints by using restricted PSPs.
Subscribe card background
Subscribe
Subscribe to
the Shift!

Get emerging insights on emerging technology straight to your inbox.

Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach

Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.

thumbnail
Download
I

Related articles

Featured home blog
Icon
Insights

Inject secrets into Kubernetes pods in a continuous way

Kubernetes
Featured home blog
Icon
Insights

Kubernetes and multi-cloud security

Kubernetes Security
Featured home blog
Icon
Insights

Accordion - A cloud native framework to enable fast SDLC for SaaS and on-prem projects

Kubernetes
Subscribe
Subscribe
 to
the Shift
!
Get
emerging insights
on emerging technology straight to your inbox.

The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.

Outshift Background