[Chaoskube] Resiliency Testing in Kubernetes

Chaoskube is a tool used to introduce chaos engineering principles into Kubernetes clusters. It randomly terminates pods in a cluster to test the resilience and reliability of the system. By simulating unexpected failures, Chaoskube helps ensure that applications and services are robust and can recover gracefully.

Key Features of Chaoskube

1. Random Pod Termination:

• Chaoskube randomly selects and deletes pods within a Kubernetes cluster at regular intervals.

2. Namespace and Label Filtering:

• You can restrict which pods Chaoskube targets by specifying namespaces or labels.

3. Exclusion Rules:

• Specific pods, namespaces, or labels can be excluded from termination to prevent disruption to critical components.

4. Time Window Scheduling:

• Allows chaos experiments to run only during specific timeframes, avoiding disruptions during critical business hours.

5. Configurable Chaos:

• Parameters like interval and grace period can be customized to control the frequency and behavior of pod termination.

6. Dry Run Mode:

• Chaoskube can simulate chaos without actually deleting pods, allowing safe testing.

How Chaoskube Works

1. Pod Selection:

• Chaoskube queries the Kubernetes API to list all pods in the cluster.

• Filters are applied based on namespaces, labels, or exclusion rules.

2. Random Termination:

• A pod is selected at random from the filtered list and terminated using Kubernetes’ delete API.

3. Chaos Frequency:

• The interval for termination is configurable (e.g., every 30 seconds or 5 minutes).

Use Cases

1. Resilience Testing:

• Ensure that your applications can handle unexpected pod failures and recover automatically.

2. Load Balancer Testing:

• Verify that load balancers redistribute traffic effectively when a pod goes down.

3. Fault Tolerance Validation:

• Test the robustness of failover mechanisms and redundancy strategies.

4. Continuous Chaos:

• Integrate Chaoskube into CI/CD pipelines for continuous resilience testing.

Installation and Usage

1. Install Chaoskube

• Deploy Chaoskube in your Kubernetes cluster using Helm or a YAML manifest.

2. Example YAML Configuration

apiVersion: apps/v1

kind: Deployment

metadata:

  name: chaoskube

  namespace: chaos-testing

spec:

  replicas: 1

  selector:

    matchLabels:

      app: chaoskube

  template:

    metadata:

      labels:

        app: chaoskube

    spec:

      containers:

      - name: chaoskube

        image: linki/chaoskube:latest

        args:

        - --interval=10s

        - --namespace=default

        - --labels=app=example

        - --dry-run=false

3. Key Parameters

• --interval: Time between pod terminations (e.g., 10s, 1m).

• --namespace: Target specific namespaces.

• --labels: Target pods with specific labels.

• --dry-run: Simulate chaos without actually deleting pods. 

Best Practices

1. Start Small:

• Begin with a dry-run mode to understand the impact of chaos experiments.

2. Use Exclusions:

• Exclude critical pods, namespaces, or labels to avoid disruptions to essential services.

3. Monitor and Observe:

• Use monitoring tools like Prometheus and Grafana to observe the system’s behavior during chaos experiments.

4. Time Constraints:

• Schedule chaos experiments during non-critical hours to minimize business impact.

5. Gradual Increase:

• Gradually increase the frequency and scope of chaos experiments as your system matures.

Benefits of Using Chaoskube

• Improved Resilience:

• Identify weaknesses in your system and improve recovery mechanisms.

• Proactive Failure Handling:

• Prepare for real-world failures by simulating them in a controlled environment.

• Continuous Improvement:

• Build confidence in the reliability of your applications and infrastructure.

Conclusion

Chaoskube is a lightweight and effective tool for introducing chaos engineering into Kubernetes environments. By simulating pod failures, it helps teams build more resilient systems capable of handling real-world disruptions.

Docker vs Kubernetes

Kubernetes and Docker are not competing technologies. In fact, they actually complement one another to get the best out of both.

Quick difference between them as below

Docker

• Containers, isolated environment for application
• Automated building and deploying applications - CI
• Container platform for configuring, building and distributing containers

Kubernetes

• Infrastructure for managing multiple containers
• Automated scheduling and management of application containers
• Ecosystem of managing a cluster of Docker containers

Kubernetes Architecture explained

In this session, we're gonna look at two types of nodes that kubernetes operates on one is master and another one is slave and we're gonna see what is the difference between those and which role each one of them has inside of the cluster and we're going to go through the basic concepts of how kubernetes does what it does and how the cluster is self-managed and self-healing etc.

Worker Nodes:

• 3 Node Processes
• Container Runtime
• Kubelet
• Kube Proxy
• Each node has multiple pods on it
• 3 processes must be installed on every Node
• Worker Nodes do the actual work
• First process that needs to run on every node is the container runtime because application pods have containers running inside, a container runtime needs to be installed on every node.
• Process that actually schedules those and containers underneath is kubelet which is a process of kubernetes itself unlike container runtime that has interface with both container runtime and machine (worker node itself) because at the end of the day kubelet is responsible for taking that configuration and actually running apod or starting a pod with a container inside and then assigning resources from that node to the container like CPU, RAM and storage resources.
• Usually kubernetes cluster is made up of multiple nodes which also must have container runtime and kubelet services installed and you can have hundreds of worker nodes which will run run other pods and containers and repplicas of the existing pods like my-app and database as an example and the way communication between them works is using services which is sort of loadbalancer which basically catches the requests directed to the pod or application like database for example and then forward it to respective pod and third process that is responsible for forwarding the requests from services to pods is actually kube proxy.
• Kube proxy must be installed on every node and kube proxy has actually intelligent forwarding logic inside that makes sure that the communication also works in a performant way with low overhead for example if an application my-app replica is making a request database instead of service just randomly forwarding the request to any replica, it will actually forward it to the replicat that is running on the same node as the pod that initiated the reuqest thus this way avoiding the network calls overhead of sending the request to another machine.
• To summarise 3 node processes must be installed on every node in order for kubernetes cluster to function properly.

Master Nodes + Master Processes:

• We discussed above regarding worker nodes and processes in details but another question comes in mind so, how do you interact with this cluster?
• How to:
• schedule pod?
• monitor?
• re-schedule/re-start pod?
• join a new node?

• Answer to above ask is all these managing processes are done Master Nodes
• There are 4 processes that run on every master node that control the cluster state and the worker nodes as well.
• API Server
• Scheduler
• Controller Manager
• etcd - the cluster brain
• API Server
• When you as a user want to deploy a new application in a kubernetes cluster, you interact with the APi Server using some client, it could be a UI like kubertest dashboard, could be commandline tool like kubelet etc. API server is like cluster gateway which gets the initial request which updates into the cluster or even the queries from the cluster and it also acts as a gatekeeper for authentication to make sure only authenticated and authorised requests get through to the cluster. 
• That means whenever you want to schedule new pods, deploy new application, create new server or any other component, you have to talk to the API server on the master node and it validates your reuqtest and if everything fine then it forward your request to other processes.
• Also if you want to query the status os your deployment or the cluster health etc, you make a request to the API server and it gives you the response which is good for security because you have one entry point into the cluster.

• Scheduler
• As mentioned API server sends request to scheduler and scheduler has intelligent way to decide in which node pod has to put based on how much resources new pod needs and how much resources availables in the given nodes etc.

• Controller Manager
• Another crucial component because what happens when pods die on any node, there must be a way to detect that nodes died and then reschedule those pods asap.
• Controller manager detects cluster state changes like crashing of pods for example when pods die, controller manager detects that and try to recover the cluster state asap and for that it makes a request to the scheduler to reschedule those dead pods in the same cycle what we discussed during scheduler part discussion.

• etcd
• etcd is the cluster brain
• key-value store
• Cluster changes get stored in the key value store
• why we call it as cluster brain because all of those mechianism with scheduler and controller manager works because of this data, for example
• How scheduler knows what resources are available on each worker node.
• How does controller manager know that a cluster state changed in some way for example pods died or that kubelet restarted new pods upon the request of a scheduler.
• or when you make a query request to API server about the cluster health or for example you application deployment state where as API server get all this information from so all of this information is stored in etcd cluster.
• Actual application data is not stored in this one, only cluster state data.

Happy Learning !! :) 

Kubernetes daily use commands

 Here is a quick view on k8 commands which are ideally useful for daily interaction work

• kubectl config get-contexts
• display list of contexts
• kubectx <env-name>
• to switch the env
• kubectl get nodes
• Get all nodes
• kubectl get namespaces
• Get all namespaces for a environment
• kubectl -n services get pod <pod-name>
• Get specific pod details
• kubectl -n services get pods
• Get all pods
• kubectl -n services delete pod <pod-name>
• Delete specific pod name
• -o wide
• To get data in more details like which node etc
• -o yaml
• To open specific file to see the details
• kubectl -n services describe pod <pod-name>
• To see description of the pod
• kubectl -n services exec -it  <pod-name> /bin/bash
• Logging into the pod to see details at code level
• kubectl -n services get hpa <hpd-pod-name>
• To check hpa details if hpa enabled
• kubectl -n services scale deploy <pod-name> --replicas=3
• To manual scale
• kubectl -n services get pods -o wide | grep ½
• Get all pods which are in unhealthy state
• kubectl -n services get deployment <pod-name> -o yaml
• To check min/max replicas or any other data point w.r.t deployment
• kubectl -n services get deployment <pod-name> -o yaml
• To edit min/max replicas or any resources
• kubectl -n services rollout restart deployment <po-name>
• To restart a service
• kubectl -n services logs <pod-name> -c install -f
• This could be used if our pod is in Init stage. Gives us the keys that are missing from config(Search for nil after running this command)
• kubectl -n configuration get pods
• Consul and Vault status
• stern -n configuration <name>
• Check logs of vault
• stern -n services <pod-name> -c <app-name> -t --since 1m
• Prints logs of all the pods of last 1 min
• stern -n services geolayers-api-primary -c geolayers-api -t --since 1m | grep '<text>' 
• Prints log only for the text that is in grep
• kubectl describe canary <pod-name> -n services
• To check canary deployment of specific pod
• kubectl get canary -n services
• To check all pods for which canary enabled

Kubectl commands complete cheatsheet : https://kubernetes.io/docs/reference/kubectl/cheatsheet/

Happy Learning !! :)

Kubernetes Basic Components

 Here is a quick view of few basic components of K8

Pod:

• Smallest unit of K8
• Abstraction over container
• Usually one application per Pod
• Each Pod gets his own IP address
• New IP address on re-creation

Services:

• An abstract way to load balance across the pods and expose an application deployed on a set of Pods.
• Permanent IP address

ConfigMap:

• External configuration of your application like DBs config etc

Secret:

• Used to store secret data like DBs username/pwd etc which can't be store in plain text in ConfigMap
• base64 encoded

Kubernetes Basics

 Official definition of Kubernetes

Open source container orchestration tool developed by Google and helps you manage containerized applications in different deployment environments like physical machines, virtual machines or cloud environment or even hybrid deployment environments.

The need for a container orchestration tool

• Trend from Monolith to Microservices
• Increased usage of containers
• Demand for a proper way of managing those hundreds of containers

What features do orchestration tools offer?

• High Availability or no downtime
• Scalability or high performance
• Disaster recovery - backup and restore

Kubernetes Basic Architecture

Made up with master node and couple of worker nodes where each node has kubelet process running on it. Kubelet is basically a kubertnetes process that makes it possible for cluster to talk to each other and actually executes some tasks on those nodes like running application processes. Each worker node has docker containers of different applications deployed on it. On worker nodes your applications are running.

Master node runs several K8 processes

• API Server which also a container and an entrypoint to K8 cluster.
• Controller Manager keeps track of whats happening in the cluster, may be if container died and needs a restart etc.
• Scheduler ensures Pods placement.
• etcd (key value storage) which basically holds at anytime current state of K8 cluster. It has all the configuration data inside and all the status data of each node and each container inside of that node and backup restore actually made of this etcd snapshot.

Kubernetes Basics Concepts

Pod is a smallest unit which you as a user will configure and interact with and basically a wrapper of container. In a worker node you gonna have multiple pods. Each pod is it's own self contained server with it's own IP address. Pods are recreated frequently and it gets new IP address on creation hence another component of K8 called service is used along with pod.