Distributed Configuration Enforcement Engine (Ceph Infra Client)

Distributed Ceph Infra client

Chef Software developed Chef Infra, an Infrastructure-as-Code (IaC) platform that automates following on remote laptop/servers/clients:
- Download artifacts (software to be installed. eg: nginx1.28)
- Verify checksums/signatures
- Execute installers or configuration changes
- Restart services if necessary
- Verify final state
- Report compliance

Requirements

Functional

1. Agent periodically pulls policy from control plane.
2. Support millions of nodes.
3. Download only changed policy versions.
4. Support versioning and rollback.

Non Functional

1. Massive Scalability. 10+ million nodes
2. Low Resource Consumption

Architecture

1. Administrator creates a policy: Install NGINX 1.28

2. Policy Engine performs these independent actions:
a. Stores the desired state in the Policy Repository.
b. Publishes lightweight metadata (latest version, artifact URL, checksum, signature).
c. Uploads the platform-specific installers to the Artifact Repository.

3. The Chef Infra Client wakes up periodically (with jitter) and only asks the Metadata Service:
a. "Has the desired state changed?"

4. If the metadata indicates a newer policy, the client:
a. Compares the desired state with the current local state.
b. Determines whether an upgrade is actually required (idempotency).
c. Only if an upgrade is needed does the client download the installer directly from the CDN/Artifact Repository, verify its integrity, and invoke the native package manager to install NGINX 1.28

flowchart LR

subgraph ControlPlane["Chef Control Plane"]

Admin["Administrator"]

PolicyUI["Policy Management UI"]

PolicyEngine["Policy Engine"]

PolicyRepo["Policy Repository
Desired State

Install:
NGINX 1.28
Service=Running"]

Metadata["Metadata Service

Latest Policy = v128

Artifact URL

Checksum

Signature"]

ArtifactRepo["Artifact Repository

nginx-1.28.rpm

nginx-1.28.deb

nginx-1.28.msi

nginx-1.28.pkg"]

LB["Global Load Balancer"]

CDN["CDN / Edge Cache"]

Admin -->|"Create Policy"| PolicyUI

PolicyUI --> PolicyEngine

PolicyEngine -->|"Store Desired State"| PolicyRepo

PolicyEngine -->|"Publish Metadata"| Metadata

PolicyEngine -->|"Upload Artifacts"| ArtifactRepo

Metadata --> LB

ArtifactRepo --> LB

LB --> CDN

end


subgraph Endpoint["Managed Laptop"]

Scheduler["Scheduler
(Periodic + Jitter)"]

Chef["Chef Client (Rust)"]

Cache["Local Metadata Cache"]

State["Current State Collector"]

Diff["Diff Engine"]

Downloader["Artifact Downloader"]

Verifier["Checksum / Signature Verification"]

Executor["Execution Engine"]

Pkg["Native Package Manager"]

Scheduler --> Chef

Chef --> Cache

Chef -->|"GET Latest Metadata"| Metadata

Metadata -->|"Policy v128
Artifact URL"| Chef

Chef --> State

State --> Diff

Cache --> Diff

Diff -->|"Upgrade Needed"| Downloader

Downloader -->|"HTTP GET Artifact"| CDN

CDN -->|"Stream NGINX Package"| Downloader

Downloader --> Verifier

Verifier --> Executor

Executor --> Pkg

Pkg -->|"Install NGINX 1.28"| Laptop["Local Operating System"]

end

Sequence Diagram (Download nginx1.28)

Ceph Infra Client (Installed on Customer Laptop)

What? client is much more than a downloader. It is the configuration enforcement engine running on every managed node.
Responsibilites

Responsiblity	Description
Scheduler	Periodically wakes up (with configurable interval and random jitter) to start a Chef run.
Authentication & Secure Communication	Authenticates with the Chef Server using client certificates/keys, establishes TLS connections, and securely fetches policies.
Policy Fetcher	Retrieves the latest policy metadata, cookbook versions, and execution instructions from the control plane.
Local Cache Manager	Stores downloaded cookbooks, policies, artifacts, and metadata locally to reduce network traffic and support offline or retry scenarios.
Policy Parser / Compiler	Parses the downloaded policy (cookbooks, recipes, resources) into an internal execution model.
Current State Collector	Discovers the machine's current state including OS, installed packages, running services, files, users, permissions, certificates, and Windows Registry entries.
Diff / Idempotency Engine	Compares the desired state with the current state and determines exactly what changes are required while preventing unnecessary operations.
Dependency & Execution Planner	Resolves dependencies between resources and computes the correct execution order (for example, install package → update configuration → restart service).
Artifact Downloader	Downloads required binaries, packages, templates, and other artifacts from an artifact repository or CDN. Supports retries and resumable downloads.
Integrity & Security Verifier	Verifies SHA-256 checksums, digital signatures, and artifact integrity before execution to prevent tampering.
Execution Engine	Applies configuration changes by invoking native operating system facilities such as APT, RPM, MSI, PowerShell, systemd, launchd, filesystem operations, and user management APIs.
Resource Throttling	Limits CPU, memory, disk I/O, network bandwidth, and concurrent operations to avoid disrupting production workloads.
Failure Handling & Rollback	Handles transient failures, retries operations, records partial progress, and supports rollback or recovery where applicable.
Compliance Verification	Re-validates the system after execution to confirm that the desired state has been successfully achieved.
Reporting & Telemetry	Sends execution status, compliance reports, metrics, logs, execution duration, and errors back to the Chef Server for monitoring and auditing.
Audit Logging	Maintains detailed local logs for troubleshooting, compliance, diagnostics, and forensic analysis.

sequenceDiagram
autonumber

actor Admin
box LightCyan Customer Premises
  participant UI
  participant Compiler
  participant Metadata
  participant ArtifactRepo
end
  participant CDN

box LightYellow Ceph Infra Client(Installed on Laptop)
  participant Scheduler
  participant ChefClient
  participant Cache
  participant StateCollector
  participant DiffEngine
  participant Planner
  participant Downloader
  participant PackageManager
  participant LocalSystem
  participant Audit
end

Admin->>UI: Create Policy
Install nginx 1.28
Enable service
Copy nginx.conf

UI->>Compiler: Compile Cookbook

Compiler->>ArtifactRepo: Store installers
Windows MSI
Linux RPM
Linux DEB
macOS PKG

Compiler->>Metadata: Publish Policy v128
Artifact URLs
Checksums
Digital Signatures

Note over Scheduler: Every 30 min ± Random Jitter

Scheduler->>ChefClient: Wake Up

ChefClient->>Cache: Read last applied version

Cache-->>ChefClient: Current = v127

ChefClient->>Metadata: GET Latest Policy Metadata

Metadata-->>ChefClient: Latest=v128
Artifact URL
Checksum
Signature
Execution Rules

ChefClient->>StateCollector: Collect Current Machine State

StateCollector-->>ChefClient: OS=Ubuntu
Arch=x64
nginx=1.26
Service=Running
Config=Old

ChefClient->>DiffEngine: Compare Desired vs Current

DiffEngine-->>ChefClient: Upgrade Required

ChefClient->>Planner: Build Execution Plan

Planner-->>ChefClient: 1.Install nginx 1.28
2.Copy config
3.Restart only if config changes
4.Verify version

ChefClient->>Downloader: Download Artifact

Downloader->>CDN: HTTP GET linux-amd64.rpm

CDN-->>Downloader: Stream RPM Package

Downloader-->>ChefClient: Download Complete

ChefClient->>ChefClient: Verify SHA256 Checksum

ChefClient->>ChefClient: Verify Digital Signature

ChefClient->>PackageManager: Install RPM

PackageManager->>LocalSystem: Upgrade nginx

LocalSystem-->>PackageManager: Success

ChefClient->>LocalSystem: Copy nginx.conf

ChefClient->>LocalSystem: Restart Service

ChefClient->>StateCollector: Verify Final State

StateCollector-->>ChefClient: nginx=1.28
Service=Running
Config=Current

ChefClient->>Cache: Store Policy Version=v128

ChefClient->>Audit: POST Compliance Report
Policy=v128
Status=Success
Duration
MachineID

Audit-->>ChefClient: ACK

ChefClient-->>Scheduler: Sleep Until Next Run