Kubernetes Manifests / Custom Resource / Objects

Kubernetes Manifests are physical YAML file which are used to create Kubernetes Objects.

While native Kubernetes gives you built-in kinds like (Service, Deployment, or Pod).
Kubebuilder provides the tooling to define entirely new kinds tailored to your applications (such as kind: Database, kind: RedisCluster, or kind: AppDeployment).

Kubernets Object
The Object is the actual active entity running inside the cluster after you submit the manifest using a command like "kubectl apply -f file.yaml"

Required Manifests
- Only deployment.yaml and service.yaml are mandatory to deploy and access your container.
- Rest are optional tools used to handle scaling, security, configuration, and monitoring.

Predefined Kubernets Manifest Types / Custom Resource

1. deployment.yaml (Mandatory)

apiVersion: apps/v1               #(Required) Which version of the Kubernetes API you're using to create this object
kind: Deployment                  #(Required) What kind of object you want to create
metadata:                         #(Required) Data that helps uniquely identify the object
  name: Kafka
spec:                             #(Required) What state you desire for the object
  replicas: 3                     # Run 3 identical copies of your app
  selector:
    matchLabels:
      app: test                   # POD Label
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app-container
        image: redpandadata/redpanda:v24.2.2    # Kafka Contianer Image.
        ports:
        - containerPort: 8080      # Internal port your app listens on
$ kubectl apply -f test.yaml

In kubernetes docker image is not build, Image is only pulled(from registry) and pod is created.

Deployment
Create a pod when kubectl apply -f test.yaml is executed.
Deployment is self healing if pod crashes kubernetes Controller automatically provisions a brand-new Pod to replace it.
Zero downtime updates: if we want to update the Redpanda version, we just change the image name in the YAML and apply it.

2. service.yaml (Mandatory)

apiVersion: v1
kind: Service
metadata:
  namespace: test
spec:
  selector:
    app: Kafka
  ports:
  - protocol: TCP
    port: 80                      # External port exposed to users
    targetPort: 8080              # Internal port inside your container
    nodePort: 30080               # Fixed external NodePort
  type: ClusterIP                 # Internal routing (change to LoadBalancer for public)

Service
In Kubernetes, Pods are temporary. They are created, destroyed, and replaced constantly during updates or node failures.
if pod gets deleted and new pod is created, how new pod would be reachable?
A Service provides a static/permanent IP address and a permanent DNS name (kafka.test.svc.cluster.local) to pod
Now other apps will talk to service
Every time a new Pod is created, it gets a dynamic, unpredictable IP address.

10.244.a.b:9092
       ↓
       ↓ Pod gets deleted
       ↓
10.244.c.d :9092

DNS: kafka.test.svc.cluster.local:9092 remains same, where other apps will reach

graph LR
    %% Define styles
    classDef client fill:#f9f,stroke:#333,stroke-width:2px;
    classDef svc fill:#85C1E9,stroke:#333,stroke-width:2px;
    classDef deployment fill:#D2B4DE,stroke:#333,stroke-width:2px;
    classDef pod fill:#2ECC71,stroke:#333,stroke-width:2px;
    classDef dns fill:#F5B041,stroke:#333,stroke-width:2px;

    %% Components
    Client[Client App-X Pod
Anywhere in Cluster]:::client
    
    subgraph DNS [CoreDNS System]
        DNS_Record["kafka.test.svc.cluster.local"]:::dns
    end

    subgraph K8s_Namespace [test Namespace]
        
        Service[Service: kafka
Type: ClusterIP
IP: 10.96.x.x STATIC IP]:::svc
        
        subgraph Deployment_Layer [Deployment: kafka]
            Pod[Pod: kafka-7f8d...
Labels: app=kafka
IP: 10.244.x.x DYNAMIC IP]:::pod
            Container[Redpanda Container
Port: 9092]:::pod
        end
        
    end

    %% Connections
    Client -->|1. Resolves Name| DNS_Record
    DNS_Record -->|2. Returns Static Service IP| Client
    Client ==> |3. Sends traffic to 10.96.x.x:9092| Service
    Service ==> |4. Routes traffic via Selector app=kafka| Pod
    Pod --> Container

Every service is assigned a CLUSTER-IP, via which pod behind service can be reached.
CLUSTER-IP: Kubernetes assigns an internal virtual IP (the ClusterIP). when any pod inside the cluster sends a packet to that ClusterIP, the packet gets NATed and load-balanced to one of the backend pods


$ kubectl get services -o wide
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORTS(s)        AGE   SELECTOR
pod1-service      NodePort    10.43.200.121   <none>     8080:30102/TCP   4h5m  app=pod1

NodePort is build on Services Read about NodePort 1st.


A ClusterIP service is created (e.g., 10.96.1.20:80)
kube-proxy adds iptables rules on each node:  <NodeIP>:30080 → 10.96.1.20:80 → PodIP:8080
Packet Flow: 

External client → NodeIP:30080 → ClusterIP:80 → PodIP:8080

That’s why kubectl get svc still shows a ClusterIP — it always exists; NodePort is built on top of it.

3. ServiceAccount

When someone creates a pod using kubectl apply -f deloy.yaml, kubectl creates a JWT token for this pod and places token inside (/var/run/secrets/...)
After pod bringup if application running inside pod wants communicate with API Server for example to run some command kubectl get pods -A, then pod will present the created JWT token and API server will validate it Credentials used by service running inside pod to communicate with API server.
This JWT token is associated with serviceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: server-ingest                 <<<<<<<<< JWT Token would be associated with This
  namespace: default
  labels:
    app: server-ingest
    tier: ingestion

sequenceDiagram
    actor user as User
    box "Worker Node"
        participant kctl as Kubectl
        participant App as Pod
Executable
    end
    box "Control Plane"
        participant API as Kubernetes API Server
        participant RBAC as RBAC Engine
    end

    user ->> kctl: kubectl apply -f deploy.yaml
    kctl ->> App: Create pod
    kctl ->> App: Place JWT
(/var/run/secrets/...)
    note over App: JWT in 
(/var/run/secrets/...)
    note over App: Pod started
 kubectl list pod -A
    App ->> API: HTTP GET
Authorization Token: JWT
kubectl list pod -A
    API->>API: Verify JWT signature &
extract ServiceAccount (app)
    API->>RBAC: Check RoleBindings for
 identity permissions
    RBAC-->>API: Allow / Deny action
    API-->>App: Return API Response (e.g., list of pods)

4. kind: PodDisruptionBudget (PDB)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webApp
spec:
  minAvailable: 4       <<<< Atleast 4 pods of web-store running
  selector:
    matchLabels:
      app: web-store

Limits the number of pods of an application that can be down at the same time during voluntary disruptions or during mainteinance.
Think of it as an insurance policy that guarantees your application always has enough running pods to handle traffic, even when cluster administrators are messing with the infrastructure.
PDB information is stored in Control Plane on etcd

sequenceDiagram
    autonumber

    box Worker Node
        actor Admin as Cluster Administrator
    end
    box Control Plane
        participant API as API Server
        participant DB as etcd (Database)
        participant Ctrl as ControllerManager
has
Disruption Controller
    end

    Note over Admin: kubectl apply 
-f test.yaml (minAvailable: 4)
    Admin->>API: HTTP POST 
/apis/policy/v1/.../poddisruptionbudgets
    API->>DB: Store spec
    DB-->>API: Acknowledge save
    API-->>Admin: 201 Created / 200 OK

    Note over Admin: kubectl apply 
-f test.yaml (pods= 2)
    Admin->>Ctrl: Change pods to 2
    Ctrl->>API: Get minPods
    API->>DB: Get PDB status for this Pod
    DB-->>API: minRequired=4
    API->>Ctrl: minRequired=4
    Ctrl-->>Admin: Error: Cannot evict pod (violates PDB)

5. hpa.yaml (Horizontal Pod Autoscaler) (Optional)

# hap.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70    # Add pods if average CPU exceeds 70%
# kubectl apply -f hpa.yaml

Scales the number of Pod replicas up or down automatically based on real-time CPU or memory usage metrics.

How pods are bringup automatically at load?
When kubectl apply -f hpa.yaml is executed it reaches API server(control plane)
- The API Server checks syntax, permissions, and validates that the target app exists.
- Creation: The HorizontalPodAutoscaler (HPA) resource is officially registered in your cluster.
- Check Controller. Loop inside the Controller Manager wakes up and starts monitoring app's CPU usage every 15 seconds.

sequenceDiagram
autonumber

    box rgba(0, 204, 102, 0.1) Worker Node
        participant MS as Metrics Server Pod
        participant Node as Kubelet Agent
        actor Admin as Administrator
    end

    box rgba(0, 128, 255, 0.1) Control Plane
        participant API as API Server
        participant DB as etcd Database
        participant HPA as HPA Controller
        participant RS as ReplicaSet Controller
    end

    Note over Admin: kubectl apply -f hpa.yaml
    Admin->>API: kubectl command travels to Api server
HTTP POST /apis/autoscaling/v2/...
    API->>DB: Save HPA spec
(min:2, max:10, target:70%)
    DB-->>API: Confirm save
    API-->>Admin: 201 Created

    Note over MS, Node: Ongoing Metric Collection
Track CPU usage via cgroups
    MS->>Node: give metrics
    Node-->>MS: Return CPU usage data

    loop Every 15 seconds
        HPA->>API: Query CPU metrics for app
        API->>MS: HTTP Get /metrics
        MS-->>API: Return Average CPU=90%
        API-->>HPA: Average CPU=90%
    end

    Note over HPA: CPU 90% > 70%
    HPA->>API: Update Deployment scale (replicas = 4)
    API->DB: Get Replica count
    
    RS->>API: Create new Pod
    API->>DB: Save new Pod object details
    API->>Node: Kubelet schedule & run pod
    Note over Node: Spin up new container replica

kind: Ingress

An Ingress is a Kubernetes API object that manages external HTTP/HTTPS access to services inside a cluster.
It acts as an application-layer (Layer 7) load balancer and reverse proxy, allowing you to route traffic based on URL paths (e.g., /api, /login) or domain names (e.g., ingest.local) to different internal Kubernetes Services.
An Ingress resource requires an Ingress Controller (such as NGINX, Traefik, or HAProxy) running in the cluster to actually fulfill the rules and handle the incoming traffic.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: server-ingest
  namespace: ns
  labels:
    app: test
    tier: ingestion
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  rules:
    - host: ingest.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: server-ingest
                port:
                  number: 8080
# kubectl apply -f ingress.yaml

ingressClassName: Tells Kubernetes that this Ingress resource should be processed and managed specifically by the NGINX Ingress controller.
annotations: Custom configurations specific to the NGINX controller. Here, they extend the read and send timeouts to 3,600 seconds (1 hour), which is useful for long-running ingestion streams.
paths: Directs any traffic matching the prefix / to the Kubernetes Service named server-ingest on port 8080.

Client
  Host: ingest.local  GET /
        ↓
nginx Ingress Controller (80/443)
        ↓
Service server-ingest:8080
        ↓
Pod (container port)

graph LR
    classDef client fill:#f9f,stroke:#333,stroke-width:2px;
    classDef ing fill:#F5B041,stroke:#333,stroke-width:2px;
    classDef svc fill:#85C1E9,stroke:#333,stroke-width:2px;
    classDef pod fill:#2ECC71,stroke:#333,stroke-width:2px;

    C[Web Client
hit REST Endpoint/]:::client
    IC[Ingress Controller
class: nginx 8080]:::ing
    S[Service
server-ingest:8080]:::svc
    P[Pod]:::pod

    C -->|HTTP Host + path| IC
    IC -->|proxy| S
    S --> P

How Ingress relates to NetworkPolicy

	Ingress	NetworkPolicy
Network Layer	Layer 7 (Application / HTTP/HTTPS)	Layer 3 & 4 (IP addresses, ports, protocols like TCP/UDP)
Traffic Direction	(Outside to Inside Cluster) Inbound traffic coming from outside the cluster into internal Services.	(Inside Cluster. Pod to Pod) Internal traffic between pods (Ingress and Egress relative to a pod) within the cluster.
Prerequisite	Requires an Ingress Controller (e.g., NGINX).	Requires a CNI plugin that supports NetworkPolicies (e.g., Calico, Cilium, Flannel with support).

Kind: NetworkPolicy

RBAC decides who may call the API server.
NetworkPolicy decides which Pods may open a TCP/UDP connection to which other Pods in cluster or which need to send data to outside world. This is a firewall for pod
policyTypes says which direction this YAML is a firewall for:

policyTypes	Meaning
`Ingress`	Rules for traffic coming in to the selected Pods (who may call me)
`Egress`	Rules for traffic going out from the selected Pods (who I may call)

Why use Egress: a stolen or buggy app inside a Pod cannot scan the cluster, dump data to a random IP, or call the public internet — it can only reach what you listed (for example the database).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: shop-can-only-talk-to-db
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: shop              # firewall applies to these Pods
  policyTypes:
  - Egress                   # this YAML is about OUTGOING packets
  egress:
  - to:                      # allow: shop → db only
    - podSelector:
        matchLabels:
          app: db
    ports:
    - protocol: TCP
      port: 5432
# kubectl apply -f networkpolicy.yaml

Very simple story
Pods labeled app: shop run the store frontend.
They should call Postgres (app: db, port 5432) and nothing else.

podSelector = whose outgoing traffic we filter.
policyTypes: [Egress] = we care about shop’s outbound connections, not who calls shop.
egress.to = the only destination still allowed.

After apply:

shop  --TCP 5432-->  db     ALLOW
shop  --any port-->  internet  DROP
shop  --any port-->  other Pods DROP

CNI must support NetworkPolicy (Calico, Cilium, …). Without that plugin the YAML is stored but not enforced.

4. namespace.yaml (Optional)

Creates an isolated virtual workspace folder inside your cluster to organize resources and separate environments (e.g., production vs staging)


apiVersion: v1
kind: Namespace
metadata:
  name: production-env

NodePool

What is Karpenter?
NodePool is Karpenter Configuration


apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        # Tells Karpenter to only provision Linux AMD64 machines
        - key: "kubernetes.io/os"
          operator: In
          values: ["linux"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        # Allow both cheap Spot instances and reliable On-Demand instances
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        name: default # Links to cloud-specific settings (like AWS subnets/security groups)
  limits:
    cpu: 1000 # Maximum total CPU cores Karpenter is allowed to spin up

5. configmap-config.yaml & configmap-policy.yaml (Optional)

Stores non-sensitive settings (like configuration files, environment paths, or feature flags) so you do not have to bake them into your Docker image.


apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_HOST: "database.internal.net"
  LOG_LEVEL: "debug"

6. secret-ca.yaml (Optional)

Stores sensitive information securely (like SSL certificates, API tokens, or passwords) using Base64 encryption, preventing secrets from being written in plain text.


apiVersion: v1
kind: Secret
metadata:
  name: secret-ca
type: Opaque
data:
  ca.crt: dGhpcyBpcyBhIGZha2U=     # Base64 encoded string of your certificate

7. servicemonitor.yaml (Optional) (Requires Prometheus installed)

This is a Custom Resource (CRD) from the Prometheus system. It tells Prometheus how and where to scrape performance metrics from your application service.


apiVersion: ://coreos.com
kind: ServiceMonitor
metadata:
  name: app-monitor
spec:
  selector:
    matchLabels:
      app: web-app
  endpoints:
  - port: metrics-port
    interval: 15s

8. ClusterRole (Optional)

Defines set of permissions or access control rules for resources across an entire Kubernetes cluster. it applies to all namespaces in the cluster.


$ test.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:                       //Metadata of clusterRole
    name: my-cluster-role
    annotations:                  //Annotations: any number of key-value pairs, and can be used to provide additional context
    my-annotation: "example"
    namespace:     "test"
rules:
- apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]

9. Job

This object runs a specific task to completion. will create 1 or more pods and execute continously until job completes.


apiVersion: batch/v1
kind: Job
metadata:
    name: pi                #Name of Job
spec:
    template:
    spec:
        serviceAccountName: "Test"    // Name of ServiceAccount that should be used by the pod that is created to run the Job
        containers:                   //Container configuration for job
        - name: pi                    // Container name to be created by this Job
        image: perl:5.34.0
        env:                        //environment variables to set for the container.
            - name: DATABASE_HOST     //this env variable is set using a SecretKeyRef
            valueFrom:
                secretKeyRef:
                name: {{ .Release.Name }}-test-db
                key: host
        restartPolicy: Never
    backoffLimit: 4

10. RBAC (Role-based Access Control)

refers to the authorization mechanism that allows one Kubernetes service or workload to access another service or resource within a cluster based on predefined roles and permissions(eg: configmaps, secrets etc). The RBAC API declares 4 kinds of Kubernetes object

a. Role & RoleBindings

Defines who (subjects) can perform actions/verbs(create, get, update etc) on which resources(eg: pods, deployments, services). Roles specify the permissions, and RoleBindings associate these roles with service accounts, users, or groups


apiVersion: rbac.authorization.k8s.io/v1    //API version of RBAC being defined
kind: Role
rules:
    - apiGroups:                //Rule1: Grant Permission to create Tokenreviews is granted in group(authentication.k8s.io)
        - authentication.k8s.io
    verbs:
        - create
    resources:
        - tokenreviews

    - apiGroups:                //Rule2: Grant Permission to get jobs in group(batch)
        - batch
    verbs:
        - get
    resources:
        - jobs

    - apiGroups: ["coordination.k8s.io"]    //Rule3: Grant Permission to perform actions in group(coordination.k8s.io)
    resources: ["leases"]
    verbs: ["get", "watch", "list", "delete", "update", "create", "patch"]

b. RoleBinding

Grants the permissions defined in a role to Subjects. Subjects can be user or set of users. Example: user:jane can read pods in default namespace


apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: read-pods
    namespace: default
subjects:
- kind: User
    name: jane            #can read pods in default namespace
    apiGroup: rbac.authorization.k8s.io
roleRef:                                  # "roleRef" specifies the binding to a Role / ClusterRole
    kind: Role                              # this must be Role or ClusterRole
    name: pod-reader                        # You need to already have a Role named "pod-reader" in that namespace.
    apiGroup: rbac.authorization.k8s.io

c. CapabilityMapping 1. Give capabilities to a process running within linux container, Eg(process to modify n/w config, mouting file system, accessing h/w devices etc)
2. TAMS capability mapping When mapping-a is enabled, service can call method1,2. When mapping-b is enabled, service can call method3,4.