8 Kubernetes Security tips for cloud engineers

How important is Kubernetes security?

38% of 600+ DevOps and engineers surveyed recently by Red Hat think it isn’t taken seriously enough, or that security investment is inadequate.

So before we start, let’s look into how critical Kubernetes security can get…

• 21% of survey respondents said a security incident led to employee termination
• 25% said the organization was fined
• 37% identified revenue/customer loss as a result of a container and Kubernetes security incident
• 67% said Kubernetes security concerns had slowed down deployment
• 90% experienced at least one security incident in the last 12 months

Suffice to say: it’s a problem. Supply chain attacks, for example, are increasing rapidly, especially after the SolarWinds attack — one of the biggest cybersecurity breaches of the 21st century — and the discovery of Log4Shell and Spring4Shell vulnerability.

We’ve already covered Kubernetes security at length here on our blog. But what are the latest tips for keeping your clusters safe? We took inspiration from Lacework’s recent white paper, and augmented its advice with some Scaleway K8s expertise. We hope it helps!

1. Encrypt your data

It’s a standard step for data protection everywhere, including with Kubernetes security! So it’s critical to check that your cloud service provider (CSP) includes this option by default, as most do:

How can you check? The CSP’s CSI (Container Storage Interface) “encrypted: “true”” line confirms its encryption of persistent volumes.

As we explained here, this is particularly critical when it comes to user data, as its “critical” and “valuable” status means it’s the first information type concerned by data regulations such as GDPR.

2. Strict access control

The next key step in Kubernetes security is determining who can do what. This can be managed in several ways.

RBAC (Role-based access control), your K8s cluster’s built-in method for determining whether a given user is allowed to perform certain actions, such as modifying configuration or accessing data. Users both need to access the cluster (authentication), then have the authorization to make changes, which means two levels of security are built in in this case. Use ClusterRoles to define roles across an entire cluster, and apply the rule of least privilege, whereby users’ accounts only have the privileges they need to perform their function.

IAM, or Identity Access Management, is recommended for larger organizations, and/or for those more comfortable inside the cloud, as it allows admins to fine-tune who can access which parts of their cloud infrastructure. Find out more about IAM here.

Open source tools such as Auth0 can also be used, so developers can connect any application written in any language or stack, and define the external identity providers, as well as the integrations they want to use… whilst using authentication for deployment, of course :) Discover more open source K8s tools here…

Open ID Connect/OIDC is one of the most foolproof ways to manage cluster access. Identity providers include Google, Facebook, and GitHub, meaning users’ existing IDs can be used, providing them with identity tokens which can be cryptographically verified by an X509 certificate. Find out more about OIDC, and K8s authentication and authorization, in this blogpost.

Whatever the tools, other security basics that should always apply are, of course: deleting users that have left your company, as well as inactive users; and ensuring your infrastructure’s security and network policies are correctly configured.

3. Monitor endpoints

It’s not that uncommon that public endpoints are accidentally published or exposed; even adding an external load balancer to your Kubernetes configuration can inadvertently give authentication rights to unwanted users. Indeed, this time last year, cybersecurity firm Cyble found over 900,000 Kubernetes instances exposed across the internet, according to InfoSecurity Magazine. Whence huge global potential for Kubernetes security incidents.

Early 2022, for example, security researchers from Apiiro discovered a vulnerability in the open-source continuous delivery platform ArgoCD, that let attackers access and exfiltrate sensitive information like passwords and API keys from clusters. Similarly notorious breaches have also occurred in recent years at Weight Watchers, Honda, Tesla and Universal, to name just a few.

Best practices to avoid such breaches include restricting access to carefully-selected developers, using IAM or RBAC (cf. above); manually approving all first external connections; creating a network map; and whitelisting key IP addresses.

4. Anticipate potential IaC errors

Using a Kubernetes-specific IaC (infrastructure as code) service like HashiCorp Terraform as the basis of your infrastructure allows you to automate deployments and re-launch them as many times as needed. However, with IaC, even the slightest human mistakes can have wide-reaching consequences, as potential errors can find themselves in production on thousands of different systems, provoking security issues and unwanted dependencies.

Ways around this Kubernetes security hazard range from common sense to super smart. Obviously, developers should be responsible for constantly inspecting their own code. If your organization doesn’t already have a solid peer review process for its code, it should. And if help is needed, numerous IaC scanning tools, including automated ones, exist to do so. They should notably prevent developers from pushing non-secure code, and therefore reduce the risk of costly errors ending up in production. Finally, only those who really need to should be able to access the repository.

5. Master container configuration at conception

When building a container, it’s essential to package all components so that your application runs properly. CI/CD should be considered as an external user in its own right, not as something that simply works with your cluster. As such, its minimal access requirements are a Container Registry (where you store the containers’ images), and access to the different namespaces of your cluster. Most importantly, it should be easy to remove access keys if they are compromised.

The solution is careful code craftsmanship. So be sure to master your creation, rather than improvising!

6. Secure and isolate nodes

One fundamental IT security principle, that naturally applies with Kubernetes security, is limiting the potentially attackable surface. This notably means isolating your K8s nodes in a private network, to avoid unnecessary exposure to the public internet.

Here’s what this looks like with Scaleway’s Kapsule K8s service (which is now available on private networks):

As explained here, this type of structure has several advantages:

• All of your CSP’s different resources (Instances, Load Balancers, Managed Databases) can communicate securely with each other, whilst limiting attack surface
• Less manual configuration work is required
• Your infrastructure’s best practice compliance is assured — a must for enterprise customers
• You’ll benefit from lower latency.

Another key measure is restraining nodes’ access to the SSH (Secure Shell) server. Some CSPs may offer this access by default, in which case you should know how to disable SSH access on your nodes (here’s how to do it with Scaleway).

7. Monitor network connections

Another Kubernetes security basic is monitoring your clusters’ activity once they’re running. You should scan both internal and external traffic for any abnormal activities. These can include an excessive number of API calls that are blocked by the internet provider, or by external threats to the network.

Such threats can result in attacks which, if successful, allow for unauthorized access to the cluster. Once hackers have access to your cluster, they might install cryptocurrency applications in your containers or Amazon S3 buckets… and you’ll only find out when you get your (excessively high) cloud bill at the end of the month.

K8s itself offers no built-in tools to scan your network for such activity. You should instead rely on standard network security tools, which should be able to detect abnormal occurrences, or specialized tools such as Falco, which describes itself as a “Kubernetes threat detection engine.”

You could also try a tool like Sysdig, which allows you to monitor your Kubernetes infrastructure and applications in real-time, detect security threats and vulnerabilities, and enforce compliance policies (more open source K8s tools here).

8. Monitor K8S audit logs

Kubernetes is de facto highly automated. This means you should pay special attention to all manual activity, like connections to containers, which must remain minimal. And when such activity does happen, it needs to be noted in an audit log, within the K8s control plane. Given this can result in millions of logs per day, the only ones that require close inspection are activities like kubectl exec or kubectl attach.

This inspection is a good way to monitor traffic connections (cf. previous point) and identify any unusual patterns that may turn out to be inappropriate behavior, or even threats. Audit tools come in useful here: Scaleway’s Cockpit, for example, will integrate audit logs in coming months.

Congratulations! You’ve made it to the end, which means your clusters’ security is optimized. Just bear in mind that there’s no such thing as perfect Kubernetes security, which means you’ll constantly have to keep checking over the above points. You got this!