K8s security - Episode 3: Security as a coding philosophy

In our previous article, we wrote about the many ways of protecting our Kubernetes clusters, namespaces, and pods. It seems obvious that applying all these rules and limitations as an afterthought to an existing production environment would be tricky. It would make things a lot easier if security was thought through beforehand, and considered “by design” in our architecture and development decisions.

So, this time we are going to think about our applications and software and their security before the production system, right from the start of the design process.

Interactions between software applications are key to product security. As we are able to isolate components depending on their communication with each other, we can take a step back and think about how we want to design.

Customers and internal users

When building a product from scratch, the main focus should always be on our end-user. They are the reason why we are developing a product or a piece of software.

We start by defining our market and the user typology we want to target. Sometimes, we find ourselves with very disparate profiles, usage patterns, needs, and expectations. We need to clarify these roles and their similarities, because where they meet is where an MVP (Minimum Viable Product) starts.

However, we need to consider more than just the end-user of our product. In Information Technology, it is rarely about only one application. We often have specific needs for managing the information we are dealing with, necessitating a whole range of interconnected applications. For instance, for our product we might need a data pipeline, analytics tools, dashboards, algorithms, and more. This software may not be directly visible to our standard end-user, but is actually where we are creating real value for our product.

We can call these types of software "internal applications", and assess that we also have corresponding internal users, such as data scientists, data engineers, devops, or even service providers. So, as well as the end-user, we also need to map out all these other stakeholders in our product and business. Once this first step is complete, and we have defined the “Who”, we can then move on to elaborating the “What”.

Functions / Applications

As we define more and more roles, more and more features become available to each of them. Clearly associating roles, functions, authorizations, accesses and permissions beforehand is a difficult task, but it is crucial if we want to to build a secure application.

It is by identifying who should have access to what part of our software that we define our layers of security. We can decide the most appropriate security measures for categories of users or single users.

Restricting access by design forces us to organize the way we are thinking about our code and our applications.

Since an example is worth a thousand words, let’s imagine a Software as a Service (SaaS) application, rendering data analysis results to end-users in the marketing business. The SaaS company's analysts would need access to their data pipeline application, to the raw data, and to a database to store transformed and enriched data. They can be represented by a group of users, requiring only the following permissions:

• read-only access to the raw data database
• read/write access to the output database
• full access to the data pipeline logs and application

This category of users will not need access to the production database of the final product (SaaS), to the product production environment, and maybe not even to the product itself.

The definition of the different roles in a software product does not only concern developers. People managing the production environment should also pay attention and get involved in the process, as it will have a major impact on the way the final environment is built and secured.

So architecture, development and end-users are not the exclusive concern of Ops any longer. If it impacts security, if it impacts the software you work on, it concerns everyone, and everyone should be invested.

Containers / Pods

We define each different application and piece of software naturally, as we map our different stakeholders and the actions they need to perform.

If some of these applications need to share a network without being accessible by anyone, they can be thought of as a part of the same pod. They can communicate with each other using localhost (the local network inside the pod), without exposing any port to the outside world (i.e. containers running in the pod cannot be reached from outside).

Other applications may need to interact in a different way, and might be coded as separated components, thus pointing to microservices and serverless use cases.

This coding strategy has multiple advantages in terms of software and security when used in a Kubernetes environment:

• The failure of a component does not imply the failure of the whole product
• A security breach in a small component is easier to fix, and does not expose all the components of the product
• Permissions and privileges can be managed on a per-component basis
• Resource limits can be applied on a component level
• A component can autoscale independently of other components

In a Kubernetes cluster, we will tend to identify these components and package them into separated containers and separated pods (unless there is a benefit to them sharing a localhost network).

Even though components are separated into different pods, we need to remember that all pods in a Kubernetes cluster communicate through the same Overlay Network. This network allows all pods to interact with each other, and is necessary so that the cluster can know whether all replicas of the same pod are ready or not (even if they are on different nodes).

The Overlay Network, by definition and for Kubernetes to operate correctly, cannot be modified or isolated.

If we want a specific user to have access to multiple pods, but not all of them, it might make sense to add isolation on a higher level of abstraction.

Namespaces

When talking about security layers, we mainly mean accesses and privileges. This takes us back to the initial simple question: Who needs access to what?

If we consider multiple teams working on the same Kubernetes cluster, we can isolate them using namespaces. This limits resource usage, so a group of users cannot monopolize all the resources of the cluster, and avoid mistakes when applying a new manifest on the cluster.

---

Conclusion

Software design is all about the end-user we are targeting. When we draw schemas of databases and architectures, we are already starting the design process, we just need to go a little bit further to include security concerns.

Security often is the victim of a technological race against a market or a competitor. When starting to develop a product from scratch, there is so much to do that it is hard to see the benefit of taking these few extra steps to include security. But this short term vision can only get us so far. When a business is growing and accelerating, that is when it needs to be able to act quickly, without needing to spend time on securing its production environment.

Unfortunately, this is usually when security issues start, and when you need to build up the security onion, layer after layer.

Time spent on security is never wasted, but it is better spent right at the start of the process.

Security should not be seen as a constraint, but as an opportunity to learn, improve, and we should be willing to do it by default. It will never be a waste of time.