Logging inside Container

Problem? I have a small go application, which dumps logs on console, But when this application runs inside container on AWS, How logs can be seen?
Solution? Loki + Promtail + Grafana

Solution = Loki + Promtail + Grafana

For small applications, We can deploy this pipeline

[Go Application] → (stdout/stderr) → [Docker Logs] → [Promtail] → [Loki] ← [Grafana]
        

Go Application: writes logs to stdout/stderr as it currently does
Docker/Container Runtime: Captures these stdout/stderr streams
Promtail:
  Runs as a sidecar container or daemonset
  Scrapes the container logs from the Docker log files
  Adds labels to identify the source (container name, pod name, etc.)
  Sends logs to Loki
Loki: Receives, stores logs from Promtail. Indexes the logs
Grafana: Connects to Loki & gets logs when user want to view the logs. Provides UI for querying and visualizing logs

Deployment on local docker using docker-compose

docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword # Create user with username(myuser), password(mypassword)
      POSTGRES_DB: mydb     # Create database named mydb
    ports:
      - "5432:5432"
    volumes:
      - db_data:/var/lib/postgresql/data
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql # For auto-creating tables

  app:
    build:
      context: .
      dockerfile: Dockerfile
    environment:
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: myuser
      DB_PASSWORD: mypassword
      DB_NAME: mydb
    ports:
      - "8080:8080"
    depends_on:
      - db

  loki:
    image: grafana/loki:3.4.1
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/mnt/config/loki-config.yaml
    command: -config.file=/mnt/config/loki-config.yaml
    networks:
      - app-network

  promtail:
    image: grafana/promtail:3.4.1
    container_name: promtail
    volumes:
      - ./promtail-config.yaml:/mnt/config/promtail-config.yaml
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/mnt/config/promtail-config.yaml
    depends_on:
      - loki
    networks:
      - app-network

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      - ./grafana-datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
    depends_on:
      - loki
    networks:
      - app-network

volumes:
  # db_data volume is a named Docker volume used to persist PostgreSQL data outside the container
  # When you first start the database, this volume is empty. PostgreSQL will initialize it with its data files
  db_data:
networks:
  app-network:
    driver: bridge
        

promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: 15s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
    pipeline_stages:
      - docker: {}
        

loki-config.yml

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
        

Grafanna will run on localhost:3000
grafana-datasource.yml

apiVersion: 1
datasources:
  - name: Loki
    type: loki
    url: http://loki:3100
    isDefault: true
        

Run Containers

// Start docker desktop (Since docker cli will connect with docker engine)

> docker-compose up -d
[+] Running 7/7
 ✔ Network go-server_app-network  Created                                                                          0.1s 
 ✔ Network go-server_default      Created                                                                          0.1s 
 ✔ Container go-server-db-1       Started                                                                          3.6s 
 ✔ Container loki                 Started                                                                          3.6s 
 ✔ Container grafana              Started                                                                          4.1s 
 ✔ Container promtail             Started                                                                          3.9s 
 ✔ Container go-server-app-1      Started                                                                          3.6s

// List all containers
> docker ps                         
CONTAINER ID   IMAGE                    COMMAND                  CREATED              STATUS              PORTS         
           NAMES
6725da48c337   go-server-app            "./main"                 About a minute ago   Up About a minute   0.0.0.0:8080->8080/tcp   go-server-app-1
650e2ca190fd   grafana/promtail:3.4.1   "/usr/bin/promtail -…"   About a minute ago   Up About a minute                 
           promtail
78c2aaeda9f8   grafana/grafana:latest   "/run.sh"                About a minute ago   Up About a minute   0.0.0.0:3000->3000/tcp   grafana
48cba656a368   postgres:16              "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:5432->5432/tcp   go-server-db-1
        
// Go inside postgres container & list tables
> docker exec -it 48cba656a368 bash
root@48cba656a368:/# psql -U myuser mydb
psql (16.9 (Debian 16.9-1.pgdg120+1))
Type "help" for help.

mydb=# \dt
           List of relations
 Schema |     Name     | Type  | Owner
--------+--------------+-------+--------
 public | employee     | table | myuser
 public | followup     | table | myuser
 public | leads        | table | myuser
 public | password     | table | myuser
 public | requirements | table | myuser
 public | tenants      | table | myuser
(6 rows)

Kibana dashboard

http://localhost:3000   [admin/admin]

        

Deployment inside AWS

ECS Instance 1

Promtail runs as sidecar container

[ECS Cluster]
├── App Task
│   ├── Go App Container
│   └── Promtail Sidecar Container
└── Database Task (or better yet, AWS RDS)
    └── PostgreSQL Container (with EFS volume for persistence)
        

ECS Instance-2

[ECS Service]
├── Loki Container (or separate service)
└── Grafana Container (or separate service)