Multi-Cloud Pricing Engine

The Multi-Cloud Pricing Engine is a high-performance orchestration system that unifies pricing data from AWS, Azure, and GCP. It provides a canonical API for cost resolution and acts as a financial context server for AI agents via the Model Context Protocol (MCP).

Getting Started

Prerequisites

Docker Engine (v20.10+)
Docker Compose
4GB+ RAM recommended for large-scale data syncing

Installation

Pull the Docker Images The engine is distributed as production-ready containers on Docker Hub.
bash
```
docker pull gauravmishra0/cost-engine:stable-api
docker pull gauravmishra0/cost-engine:stable-mcp
```

Prepare Configuration Create a docker-compose.yml file. This file controls the API, MCP Server, and Database.

yaml

services:
  pricing-api:
    image: gauravmishra0/cost-engine:stable-api
    container_name: pricing-api
    ports:
      - "9090:9090"
    environment:
      - DB_HOST=postgres
      - DB_PORT=5432
      - DB_USER=postgres
      - DB_PASSWORD=secure_password
      - DB_NAME=pricing
      - LOG_LEVEL=info
      - INGEST_BATCH_SIZE=2000
      - STARTUP_SYNC=false          # Auto-sync on container start
      - GCP_API_KEY=${GCP_API_KEY}  # Required for GCP ingestion
      # Cron scheduler configuration
      - SYNC_ENABLED=true           # Enables daily multi-cloud sync
      - SYNC_SCHEDULE=@daily        # Interval: @daily, @hourly, or cron format
      # Cloud-specific regions (comma-separated)
      - AWS_REGIONS=us-east-1,us-west-2
      - AZURE_REGIONS=eastus,westus2
      - GCP_REGIONS=us-central1,us-east1
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  mcp-server:
    image: gauravmishra0/cost-engine:stable-mcp
    container_name: cost-engine-mcp
    depends_on:
      - pricing-api
    environment:
      - API_URL=http://pricing-api:9090
    stdin_open: true
    tty: true
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    container_name: cost-engine-db
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=secure_password
      - POSTGRES_DB=pricing
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres_data:

networks:
  default:
    name: cost-engine-network

Start the Engine Launch the stack in detached mode.
bash
```
docker compose up -d
```
Verify Health Ensure the API is running.
bash
```
curl http://localhost:9090/v1/health
```

Automated Ingestion Scheduler

The engine features a built-in Cron Scheduler designed to keep your pricing data synchronized across all supported clouds (AWS, Azure, GCP).

How to Configure

To create a scheduler, simply define the scheduling environment variables in your Docker configuration. No code changes or external cron jobs are required.

SYNC_ENABLED: Set to true to activate the background worker.
AWS_REGIONS: Comma-separated list of AWS regions (e.g., us-east-1,us-west-2).
AZURE_REGIONS: Comma-separated list of Azure regions (e.g., eastus,westus2).
GCP_REGIONS: Comma-separated list of GCP regions (e.g., us-central1,us-east1).
SYNC_SCHEDULE: Accepts standard Cron syntax.
- @daily: Run once at midnight.
- @hourly: Run once at the top of every hour.
- 0 30 * * *: Run every day at 30 minutes past midnight.

Behavior

When triggered, the scheduler performs the following operations sequentially:

AWS Sync: Fetches pricing for 48 supported services across all major categories (EC2, RDS, S3, Lambda, EKS, CloudFront, etc.) in configured regions.
Azure Sync: Pulls retail pricing updates for all available Azure services in configured regions (no filtering).
GCP Sync: Updates SKU catalog for 36 supported services across all major categories (Compute Engine, Cloud SQL, BigQuery, GKE, Cloud Run, etc.) in configured regions.

Note: Each sync operation runs independently. Services are synced automatically - no service selection required.

AI Agent Integration (MCP)

This engine is designed to give AI agents access to real-time cloud pricing. To connect Claude Desktop or other MCP clients:

Open your claude_desktop_config.json.
Add the following server configuration:

json

{
  "mcpServers": {
    "cost-engine": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "--network", "cost-engine-network",
        "-e", "API_URL=http://pricing-api:9090",
        "gauravmishra0/cost-engine:stable-mcp",
        "/app/mcp"
      ]
    }
  }
}

Or if using docker-compose, the MCP server is already running and can be accessed via stdio.

AWS Service Discovery Modes

The engine supports three modes for AWS service selection, giving you full control over which services to sync:

Mode 1: Curated (Default - Recommended)

Use Case: Production deployments, daily sync jobs
Configuration: AWS_DISCOVERY_MODE=curated (default if not specified)
Performance: ~15 minutes for single region, 150MB RAM peak
Coverage: 48 production-tested AWS services with known pricing data

Services include all major categories:

Compute: EC2, Lambda, ECS, EKS, Batch, Lightsail
Storage: S3, EBS, EFS, FSx, Backup, Glacier
Databases: RDS, DynamoDB, ElastiCache, Redshift, Neptune, DocumentDB, MemoryDB, Keyspaces, Timestream, QLDB
Networking: CloudFront, VPC, ELB, Route53, Direct Connect, API Gateway
Analytics: Athena, EMR, Kinesis, Glue, MSK, OpenSearch
ML/AI: SageMaker, Rekognition, Comprehend, Transcribe, Bedrock
Containers: ECR, App Runner
Integration: SNS, SQS, EventBridge
Security: Secrets Manager, WAF

Why use this: Avoids wasting time on 170+ AWS services that return 404 or have no pricing data. All 48 services are verified to have active pricing.

Mode 2: Dynamic Discovery

Use Case: Weekly full scans, new service discovery
Configuration: AWS_DISCOVERY_MODE=dynamic
Performance: ~2-4 hours for single region, 500MB RAM peak per concurrent service
Coverage: Auto-discovers all 200+ AWS services from pricing API index

How it works:

Fetches AWS offers index (2MB, lists all services)
Attempts to sync each discovered service
Automatically filters by region during streaming
170+ services will return 404 or empty (AWS doesn't have pricing for many listed services)

Why use this: Automatically detects new AWS services without code updates. Good for periodic comprehensive scans to catch newly launched services.

Trade-offs:

✅ Zero maintenance - new services detected automatically
✅ No code changes when AWS launches new products
❌ 90+ minutes wasted on HTTP 404s and empty responses
❌ 8x slower than curated mode

Mode 3: Custom Service List

Use Case: Project-specific needs, cost-optimized syncing
Configuration: AWS_DISCOVERY_MODE=custom + AWS_SERVICES=AmazonEC2,AmazonRDS,AmazonS3
Performance: Depends on selected services (~3-5 min per service)
Coverage: Exactly what you specify (comma-separated)

Why use this: When you only need specific services (e.g., only EC2 + RDS for compute-heavy workload). Fastest option for targeted deployments.

Configuration Examples

Docker Compose:

yaml

services:
  pricing-api:
    environment:
      # Option 1: Curated (default)
      - AWS_DISCOVERY_MODE=curated
      
      # Option 2: Dynamic discovery
      # - AWS_DISCOVERY_MODE=dynamic
      
      # Option 3: Custom services
      # - AWS_DISCOVERY_MODE=custom
      # - AWS_SERVICES=AmazonEC2,AmazonRDS,AmazonS3,AWSLambda,AmazonDynamoDB

Recommendation: Use curated for production and daily syncs. Run dynamic mode weekly/monthly in a separate job to discover new services, then add valuable ones to your curated list.

GCP Service Discovery Modes

The engine supports the same three modes for GCP service selection:

Mode 1: Curated (Default - Recommended)

Configuration: GCP_DISCOVERY_MODE=curated
Performance: ~5 minutes for single region, 100MB RAM peak
Coverage: 36 production-tested GCP services

Services include:

Compute: Compute Engine, Kubernetes Engine, Cloud Run, Cloud Functions, App Engine, Cloud GPUs
Storage & Databases: Cloud SQL, Cloud Storage, Bigtable, Spanner, Firestore, Memorystore (Redis/Memcached), Filestore, Persistent Disk, Cloud Backup
Networking: Cloud CDN, Load Balancing, VPN, Interconnect, Armor, DNS
Analytics: BigQuery, Dataflow, Dataproc, Pub/Sub, Composer, Data Fusion
ML & AI: Vertex AI, Vision API, Natural Language API, Translation API
Security: Secret Manager, KMS, Monitoring, Logging

Mode 2: Dynamic Discovery

Configuration: GCP_DISCOVERY_MODE=dynamic
Performance: ~20-30 minutes for single region, 150MB RAM peak
Coverage: Auto-discovers all 100+ GCP services from Cloud Billing API

Trade-offs:

✅ Detects new GCP services automatically
❌ 4-6x slower due to 100+ API calls
❌ Rate limit risk with high request volume

Mode 3: Custom Service List

Configuration: GCP_DISCOVERY_MODE=custom + GCP_SERVICES=Compute Engine,Cloud SQL,BigQuery
Performance: ~1-2 min per service
Coverage: Exactly what you specify (comma-separated display names)

Azure Optimization

Azure already uses an optimal approach:

Single paginated API for all services (no per-service calls needed)
Already streaming and filtering by region during ingestion
No service discovery modes needed - it's already efficient!

Azure syncs all available services automatically with minimal overhead (~5-7 minutes per region).

Cross-Cloud Performance Optimizations

Beyond service discovery, the engine includes several performance optimizations applicable to all clouds:

1. Parallel Processing

Environment Variable: MAX_CONCURRENT_SYNCS=3 (default)
What it does: Controls how many regions/services can sync simultaneously
Tuning:

1: Sequential (safest, slowest, ~300MB RAM)
3: Balanced (default, ~500MB RAM)
5-10: Aggressive (fast but 1GB+ RAM, may hit rate limits)

When to increase:

Large multi-region deployments
Powerful servers (8GB+ RAM)
When time is more critical than memory

When to decrease:

Low-memory environments (< 2GB)
Rate limit concerns
Single-region deployments

2. Batch Size Optimization

Environment Variable: INGEST_BATCH_SIZE=2000 (default)
What it does: Number of price records to accumulate before database insert
Tuning:

500-1000: Low memory, slower inserts
2000-3000: Balanced (recommended)
5000+: High memory, fastest inserts

Trade-off: Larger batches = fewer DB roundtrips but more RAM usage

3. Query Response Caching (Future Optimization)

Environment Variables:

ENABLE_QUERY_CACHE=false (default, not yet implemented)
CACHE_TTL_MINUTES=60

What it does: Caches identical cost queries to avoid repeated database lookups
Status: Configuration exists, implementation coming in v2.2

4. Streaming Buffer Size

Environment Variable: INGEST_STREAM_BUFFER_KB=64 (default)
What it does: Buffer size for JSON stream parsing (AWS primarily)
Tuning:

32-64: Low memory
128-256: Better performance
512+: Maximum throughput (AWS bulk files only)

5. Database Connection Pooling

Environment Variables:

DB_MAX_OPEN_CONNS=25 (default)
DB_MAX_IDLE_CONNS=5 (default)

What it does: Controls PostgreSQL connection pool size
Tuning for high concurrency:

Increase MAX_OPEN_CONNS to 50-100 for parallel syncs
Match MAX_CONCURRENT_SYNCS to avoid connection exhaustion

Configuration Example (Optimized for Speed)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=curated       # Fast, 48 services
      - GCP_DISCOVERY_MODE=curated       # Fast, 36 services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=5           # 5 parallel syncs
      - INGEST_BATCH_SIZE=3000           # Larger batches
      - INGEST_STREAM_BUFFER_KB=128      # Larger buffer
      - DB_MAX_OPEN_CONNS=50             # Match concurrency
      
      # Memory requirements: ~1.5-2GB

Configuration Example (Optimized for Memory)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=custom
      - AWS_SERVICES=AmazonEC2,AmazonRDS,AmazonS3  # Only 3 services
      - GCP_DISCOVERY_MODE=custom
      - GCP_SERVICES=Compute Engine,Cloud SQL      # Only 2 services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=1           # Sequential only
      - INGEST_BATCH_SIZE=1000           # Smaller batches
      - INGEST_STREAM_BUFFER_KB=32       # Smaller buffer
      
      # Memory requirements: ~300MB

Configuration Example (Comprehensive Discovery)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=dynamic       # All 200+ services
      - GCP_DISCOVERY_MODE=dynamic       # All 100+ services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=3           # Moderate parallelism
      - INGEST_BATCH_SIZE=2000           # Standard batches
      
      # Expected: 2-4 hours, 2-3GB RAM, comprehensive coverage

API Reference

1. Ingestion and Syncing

Manually trigger updates for specific providers.

Sync AWS Data

Syncs 48 supported services covering compute (EC2, ECS, EKS, Lambda), databases (RDS, DynamoDB, ElastiCache, Redshift), storage (S3, EFS, FSx), networking (VPC, CloudFront, Route53), analytics (Athena, EMR, Kinesis), ML/AI (SageMaker, Rekognition), and more. Input:

bash

curl -X POST http://localhost:9090/v1/ingest/aws/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["us-east-1"]}'

Output:

json

{
  "results": [
    {
      "provider": "aws",
      "status": "success",
      "candidates_stored": 15420,
      "message": "Sync completed successfully"
    }
  ]
}

Sync Azure Data

Syncs all available Azure services (Virtual Machines, Storage, SQL Database, App Service, etc.) - no service filtering applied. Input:

bash

curl -X POST http://localhost:9090/v1/ingest/azure/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["eastus"]}'

Output:

json

{
  "results": [
    {
      "provider": "azure",
      "status": "success",
      "candidates_stored": 8500,
      "message": "Sync completed successfully"
    }
  ]
}

Sync GCP Data

Syncs 36 supported services covering compute (Compute Engine, GKE, Cloud Run, Cloud Functions), storage & databases (Cloud SQL, Cloud Storage, Bigtable, Spanner, Firestore), analytics (BigQuery, Dataflow, Dataproc, Pub/Sub), networking (Cloud CDN, Load Balancing, Cloud VPN), ML/AI (Vertex AI, Vision API), and more. Input:

bash

curl -X POST http://localhost:9090/v1/ingest/gcp/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["us-central1"]}'

Output:

json

{
  "results": [
    {
      "provider": "gcp",
      "status": "success",
      "candidates_stored": 12000,
      "message": "Sync completed successfully"
    }
  ]
}

2. Metadata Discovery

Explore available data.

List Regions

Input:

bash

curl "http://localhost:9090/v1/metadata/regions?provider=aws"

Output:

json

{
  "provider": "aws",
  "regions": ["us-east-1", "us-west-2", "eu-central-1"],
  "count": 3
}

List Services

Input:

bash

curl "http://localhost:9090/v1/metadata/services?provider=azure&region=eastus"

Output:

json

{
  "provider": "azure",
  "region": "eastus",
  "services": ["Virtual Machines", "SQL Database", "Storage"],
  "count": 3
}

3. Cost Resolution

Calculate costs using the engine's resolution logic.

Single Resource Lookup

Input:

bash

curl -X POST http://localhost:9090/v1/cost/resolve \
     -H "Content-Type: application/json" \
     -d '{
       "provider": "aws",
       "region": "us-east-1",
       "service": "compute",
       "resource_type": "vm",
       "instance_type": "t3.medium",
       "os": "linux",
       "billing_model": "hourly",
       "hours": 730,
       "quantity": 1
     }'

Output:

json

{
  "provider": "aws",
  "region": "us-east-1",
  "instance_type": "t3.medium",
  "unit_price": 0.0416,
  "total_cost": 30.368,
  "currency": "USD",
  "effective_date": "2024-01-15T00:00:00Z"
}

Batch Resolution

Input:

bash

curl -X POST http://localhost:9090/v1/cost/resolve/batch \
     -H "Content-Type: application/json" \
     -d '{
       "requests": [
         {
           "provider": "aws", 
           "region": "us-east-1", 
           "instance_type": "t3.medium", 
           "service": "compute", 
           "resource_type": "vm", 
           "os": "linux", 
           "billing_model": "hourly", 
           "hours": 730, 
           "quantity": 10
         }
       ]
     }'

Output:

json

{
  "results": [
    {
      "provider": "aws",
      "instance_type": "t3.medium",
      "total_cost": 303.68,
      "currency": "USD"
    }
  ]
}

Developer Guide

Architecture Overview

The cost engine follows a microservices architecture with two primary components:

plain

┌─────────────────┐         ┌──────────────────┐
│   MCP Server    │────────▶│   Pricing API    │
│  (stdio/JSON)   │         │   (HTTP/REST)    │
└─────────────────┘         └──────────────────┘
                                     │
                                     ▼
                            ┌──────────────────┐
                            │   PostgreSQL     │
                            │  (Versioned DB)  │
                            └──────────────────┘

Component Responsibilities

1. Pricing API (cmd/api/main.go)

HTTP REST API on port 9090
Cloud provider sync orchestration (AWS, Azure, GCP)
Cost resolution engine
Background cron scheduler
Swagger documentation at /swagger/index.html

2. MCP Server (cmd/mcp/main.go)

Model Context Protocol server
Stdio-based JSON-RPC communication
Proxies requests to Pricing API
Designed for AI agent integration (Claude Desktop, etc.)

3. PostgreSQL Database

Immutable pricing versions
Time-based effective date resolution
Auto-creates schema on first connection

Database Schema

The engine uses a versioned, immutable pricing model:

Core Tables

pricing_versions - Pricing snapshots

sql

CREATE TABLE pricing_versions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    version_date DATE NOT NULL UNIQUE,          -- YYYY-MM-DD
    fetched_at TIMESTAMP NOT NULL,              -- Actual fetch time
    ingested_rows BIGINT,                       -- Number of prices
    status VARCHAR(50) NOT NULL,                -- in_progress | active | failed
    error_msg TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

prices - Individual pricing records

sql

CREATE TABLE prices (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    pricing_version_id UUID NOT NULL REFERENCES pricing_versions(id) ON DELETE CASCADE,
    provider VARCHAR(50) NOT NULL,              -- aws | azure | gcp
    service VARCHAR(100) NOT NULL,              -- compute | serverless | storage
    resource_type VARCHAR(100) NOT NULL,        -- vm | lambda | disk
    billing_model VARCHAR(50) NOT NULL,         -- hourly | usage-based
    region VARCHAR(100) NOT NULL,               -- us-east-1 | eastus | etc.
    instance_type VARCHAR(255) NOT NULL,        -- t3.medium | Standard_D2s_v3
    sku VARCHAR(255) NOT NULL,                  -- Cloud provider SKU
    unit VARCHAR(100) NOT NULL,                 -- hour | gb-second | request
    unit_price NUMERIC(18, 10) NOT NULL,        -- Price per unit
    currency VARCHAR(10) NOT NULL DEFAULT 'USD',
    attributes JSONB,                           -- Additional metadata
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

-- Performance indexes
CREATE INDEX idx_prices_provider_service ON prices(provider, service);
CREATE INDEX idx_prices_region_resource ON prices(region, resource_type);
CREATE INDEX idx_prices_instance_type ON prices(instance_type);

service_metadata - Available services catalog

sql

CREATE TABLE service_metadata (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    provider VARCHAR(50) NOT NULL,
    service VARCHAR(100) NOT NULL,
    resource_type VARCHAR(100) NOT NULL,
    billing_model VARCHAR(50) NOT NULL,
    regions TEXT[] NOT NULL,                    -- Array of supported regions
    last_updated TIMESTAMP NOT NULL,
    UNIQUE(provider, service, resource_type, billing_model)
);

Development Setup

Local Development (without Docker)

Install Dependencies

bash

# Go 1.24+
go mod download

# PostgreSQL 16+
brew install postgresql@16  # macOS
sudo apt install postgresql-16  # Ubuntu

# Swagger CLI (for API docs)
go install github.com/swaggo/swag/cmd/swag@latest

Setup Database

bash

# Create database
createdb pricing

# Schema is auto-created on first connection
# Or manually apply:
psql -U postgres -d pricing -f internal/database/schema.go

Environment Configuration

bash

# Create .env file
cat > .env <<EOF
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=
DB_NAME=pricing
LOG_LEVEL=debug
INGEST_BATCH_SIZE=2000
GCP_API_KEY=your_key_here
EOF

Build and Run

bash

# Build API
go build -o bin/api cmd/api/main.go

# Build MCP
go build -o bin/mcp cmd/mcp/main.go

# Run API
./bin/api

# Run MCP (in another terminal)
./bin/mcp

Generate Swagger Docs

bash

swag init -g cmd/api/main.go -o docs
# Visit http://localhost:9090/swagger/index.html

API Error Handling

All API endpoints return structured error responses:

Success Response (200/201)

json

{
  "results": [{"status": "success", ...}]
}

Error Response (4xx/5xx)

json

{
  "error": "error_code",
  "message": "Human-readable error description",
  "details": {
    "field": "additional_context"
  }
}

Common Error Codes

| Code | HTTP Status | Description | Solution | |------|-------------|-------------|----------| | invalid_request | 400 | Missing required field or invalid JSON | Check request body matches API spec | | price_not_found | 404 | No pricing data for given parameters | Sync the region first using /ingest/{provider}/sync | | region_not_supported | 400 | Region not available for provider | Check /metadata/regions for valid regions | | database_locked | 409 | Sync already in progress | Wait for current sync to complete | | internal_error | 500 | Unexpected server error | Check logs with docker logs pricing-api | | provider_api_error | 502 | Cloud provider API failure | Retry after a few minutes |

Testing

Unit Tests

bash

# Run all tests
go test ./...

# Run with coverage
go test -cover ./...

# Test specific package
go test ./internal/pricing/...

# Verbose output
go test -v ./internal/ingestion/...

Integration Tests

bash

# Start test database
docker run -d --name pricing-test-db \
  -e POSTGRES_PASSWORD=test \
  -p 5433:5432 postgres:16-alpine

# Run integration tests
DB_PORT=5433 go test -tags=integration ./...

# Cleanup
docker rm -f pricing-test-db

API Testing with curl

bash

# Health check
curl http://localhost:9090/v1/health

# Sync AWS (returns immediately, runs in background)
curl -X POST http://localhost:9090/v1/ingest/aws/sync \
  -H "Content-Type: application/json" \
  -d '{"regions":["us-east-1"]}'

# Check sync status (wait 30 seconds, then query)
sleep 30
psql -U postgres -d pricing -c \
  "SELECT version_date, status, ingested_rows FROM pricing_versions ORDER BY created_at DESC LIMIT 1;"

# Query pricing
curl -X POST http://localhost:9090/v1/cost/resolve \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "aws",
    "region": "us-east-1",
    "service": "compute",
    "resource_type": "vm",
    "instance_type": "t3.medium",
    "billing_model": "hourly",
    "os": "linux",
    "hours": 730
  }' | jq .

MCP Testing

bash

# Test MCP directly with JSON-RPC
echo '{
  "jsonrpc":"2.0",
  "id":1,
  "method":"tools/call",
  "params":{
    "name":"resolve_cost",
    "arguments":{
      "provider":"aws",
      "region":"us-east-1",
      "service":"compute",
      "resource_type":"vm",
      "instance_type":"t3.medium",
      "billing_model":"hourly"
    }
  }
}' | docker exec -i cost-engine-mcp /app/mcp

Performance Considerations

Memory Usage

AWS EC2 Full Sync: ~150MB peak (streaming JSON decoder)
Azure Full Sync: ~100MB peak
GCP Full Sync: ~120MB peak
Idle State: ~50MB

Sync Duration (typical)

AWS us-east-1: 45-90 minutes (48 services with 1000+ SKUs each)
Azure eastus: 5-8 minutes (all services, single API)
GCP us-central1: 15-25 minutes (36 services)

Database Growth

Per Region: ~10-15K pricing records
Storage: ~500KB per region (compressed)
Indexes: ~2x data size

Optimization Tips

Batch Size: Adjust INGEST_BATCH_SIZE (default: 2000)
- Lower = Less memory, slower inserts
- Higher = More memory, faster inserts
Connection Pool: Configure PostgreSQL max_connections
yaml
```
postgres:
  command: postgres -c max_connections=200
```

Parallel Sync: Run multiple region syncs in parallel

bash

# Sync multiple regions simultaneously
for region in us-east-1 us-west-2 eu-central-1; do
  curl -X POST http://localhost:9090/v1/ingest/aws/sync \
    -H "Content-Type: application/json" \
    -d "{\"regions\":[\"$region\"]}" &
done
wait

Debugging

Enable Debug Logging

yaml

environment:
  - LOG_LEVEL=debug  # info | debug | warn | error

Common Debug Scenarios

1. Sync failing silently

bash

# Check API logs
docker logs pricing-api --tail 100 | grep -i error

# Check database status
psql -U postgres -d pricing -c \
  "SELECT * FROM pricing_versions WHERE status='failed' ORDER BY created_at DESC LIMIT 5;"

2. MCP not responding

bash

# Verify MCP container is running
docker ps | grep mcp

# Check MCP logs
docker logs cost-engine-mcp --tail 50

# Test MCP connectivity
docker exec cost-engine-mcp curl -f http://pricing-api:9090/v1/health

3. Pricing data missing

bash

# List available regions
curl "http://localhost:9090/v1/metadata/regions?provider=aws" | jq .

# Check if region is synced
psql -U postgres -d pricing -c \
  "SELECT DISTINCT region FROM prices WHERE provider='aws';"

# Count prices per region
psql -U postgres -d pricing -c \
  "SELECT region, COUNT(*) FROM prices GROUP BY region ORDER BY count DESC;"

4. Database connection issues

bash

# Test from container
docker exec pricing-api ping -c 3 postgres

# Check PostgreSQL logs
docker logs cost-engine-db --tail 50

# Verify credentials
docker exec pricing-api env | grep DB_

Advanced Configuration

Custom Cron Schedules

yaml

# Every 6 hours
SYNC_SCHEDULE: "0 */6 * * *"

# Every Monday at 3 AM
SYNC_SCHEDULE: "0 3 * * 1"

# Every 15 minutes (not recommended - rate limiting)
SYNC_SCHEDULE: "*/15 * * * *"

Multi-Region Optimization

yaml

# Prioritize frequently-used regions
AWS_REGIONS: "us-east-1,us-west-2,eu-west-1"
AZURE_REGIONS: "eastus,westus2,westeurope"
GCP_REGIONS: "us-central1,europe-west1"

Startup Sync

yaml

# Auto-sync on container start (useful for cold starts)
STARTUP_SYNC: "true"

Cloud Provider Specifics

AWS

API: AWS Pricing API (bulk JSON)
Rate Limits: 10 requests/second
Regions: 20+ regions supported
Supported Services: 48 services including:
- Compute: EC2, ECS, EKS, Lambda, Batch, Lightsail
- Databases: RDS, DynamoDB, ElastiCache, Redshift, Neptune, DocumentDB, Keyspaces
- Storage: S3, EFS, FSx, S3 Glacier, Storage Gateway, Backup
- Networking: VPC, CloudFront, Route53, Global Accelerator, API Gateway
- Analytics: Athena, EMR, Kinesis, OpenSearch, Glue, QuickSight
- ML/AI: SageMaker, Rekognition, Comprehend, Transcribe
- Integration: SQS, SNS, MQ, Step Functions, EventBridge
- Developer Tools: CodeBuild, CodeDeploy, CodePipeline
- Security: Secrets Manager, KMS, Support Business
Update Frequency: Daily (AWS updates overnight)

Azure

API: Azure Retail Prices API (paginated)
Rate Limits: 100 requests/minute
Regions: 60+ regions supported
Supported Services: All Azure services (Virtual Machines, Storage, SQL Database, App Service, Cosmos DB, etc.)
Update Frequency: Real-time

GCP

API: Cloud Billing API (requires API key)
Rate Limits: 60 requests/minute
Regions: 30+ regions supported
Supported Services: 36 services including:
- Compute: Compute Engine, Cloud Run, GKE, Cloud Functions, App Engine
- Storage & Databases: Cloud SQL, Cloud Storage, Filestore, Bigtable, Spanner, Firestore, Memorystore
- Analytics & Big Data: BigQuery, Dataflow, Dataproc, Pub/Sub, Datastream
- Networking: Cloud CDN, Load Balancing, Cloud VPN, Cloud NAT, Cloud DNS, Cloud Armor
- AI/ML: Vertex AI, ML Engine, Vision API, Natural Language API
- Developer Tools: Cloud Build, Container Registry, Artifact Registry
- Operations: Cloud Logging, Cloud Monitoring, Cloud Trace
- Security: Cloud KMS, Secret Manager
Update Frequency: Daily

Contributing

Code Structure

plain

cost-engine/
├── cmd/
│   ├── api/          # HTTP API entrypoint
│   └── mcp/          # MCP server entrypoint
├── internal/
│   ├── api/
│   │   ├── router.go           # HTTP routes
│   │   └── handlers/           # Request handlers
│   ├── ingestion/              # Cloud provider adapters
│   │   ├── aws.go
│   │   ├── azure.go
│   │   └── gcp.go
│   ├── pricing/                # Cost resolution engine
│   │   ├── calculator.go
│   │   ├── eligibility.go
│   │   └── selector.go
│   ├── database/               # Data layer
│   │   ├── schema.go
│   │   └── db.go
│   └── domain/                 # Core models
├── docs/                       # Swagger docs (auto-generated)
├── migrations/                 # Database migrations
└── pkg/
    └── mcp/                    # MCP protocol implementation

Adding a New Cloud Provider

Create ingestion adapter in internal/ingestion/newcloud.go

type NewCloudAdapter struct {
    client *http.Client
}

func (a *NewCloudAdapter) FetchPricing(region string) ([]domain.PriceCandidate, error) {
    // Implement provider-specific API calls
}

Add handler in internal/api/handlers/newcloud_sync.go

func NewCloudSyncHandler(repo *database.Repository, log *logger.Logger) {
    // Implement sync endpoint
}

Register route in internal/api/router.go

api.POST("/ingest/newcloud/sync", handlers.NewCloudSync)

Update cron scheduler in cmd/api/main.go

// Add to scheduler job
newCloudHandler := handlers.NewNewCloudSyncHandler(repo, structLog)
newCloudErr := newCloudHandler.SyncProgrammatic(syncCtx, newCloudRegions)

Troubleshooting

Common Issues:

connection refused: Ensure Docker container is running and port 9090 is mapped correctly.
price_not_found: Verify that the specific region has been synced.
database_locked: The engine allows only one sync operation per provider at a time.

For additional support, please open an issue in the repository.

Production Deployment Guide

This section is for customers integrating the pricing engine into their own application stack.

1. Sidecar Deployment (Kubernetes)

For low-latency access, deploy the engine as a sidecar container in your application pod.

yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    # Your Main Application
    - name: my-app
      image: my-company/app:latest
      env:
        - name: COST_ENGINE_URL
          value: "http://localhost:9090"

    # Multi-Cloud Pricing Engine Sidecar
    - name: pricing-api
      image: gauravmishra0/cost-engine:stable-api
      ports:
        - containerPort: 9090
      env:
        - name: DB_HOST
          value: "postgres-service"
        - name: DB_USER
          value: "postgres"
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secrets
              key: password
        - name: DB_NAME
          value: "pricing"
        - name: SYNC_ENABLED
          value: "true"
        - name: SYNC_SCHEDULE
          value: "@daily"
        - name: AWS_REGIONS
          value: "us-east-1,us-west-2"
        - name: AZURE_REGIONS
          value: "eastus,westus2"
        - name: GCP_REGIONS
          value: "us-central1"
        - name: GCP_API_KEY
          valueFrom:
            secretKeyRef:
              name: gcp-secrets
              key: api-key

2. Standalone Service (Docker Compose)

For centralized access across multiple microservices, deploy as a standalone service.

bash

# 1. Create a dedicated network
docker network create pricing-net

# 2. Run Database
docker run -d --name pricing-db --net pricing-net \
  -e POSTGRES_PASSWORD=prod_pass \
  postgres:16-alpine

# 3. Run Engine
docker run -d --name pricing-api --net pricing-net \
  -p 9090:9090 \
  -e DB_HOST=pricing-db \
  -e DB_USER=postgres \
  -e DB_PASSWORD=prod_pass \
  -e DB_NAME=pricing \
  -e SYNC_ENABLED=true \
  -e SYNC_SCHEDULE="0 3 * * *" \
  -e AWS_REGIONS="us-east-1,us-west-2" \
  -e AZURE_REGIONS="eastus" \
  -e GCP_REGIONS="us-central1" \
  -e GCP_API_KEY=your_gcp_key \
  gauravmishra0/cost-engine:stable-api

3. Sizing Recommendations

Small Scale (Under 10k requests/day): 1 vCPU, 512MB RAM (Sufficient for background ingestion of hourly datasets)
Large Scale (Over 1M requests/day): 2 vCPU, 4GB RAM (Recommended to handle memory spikes during full-catalog syncs)
Storage: 50GB SSD persistent volume recommended for the PostgreSQL backend.

Multi-Cloud Pricing Engine

Getting Started

Prerequisites

Docker Engine (v20.10+)
Docker Compose
4GB+ RAM recommended for large-scale data syncing

Installation

Pull the Docker Images The engine is distributed as production-ready containers on Docker Hub.
bash
```
docker pull gauravmishra0/cost-engine:stable-api
docker pull gauravmishra0/cost-engine:stable-mcp
```

Prepare Configuration Create a docker-compose.yml file. This file controls the API, MCP Server, and Database.

yaml

services:
  pricing-api:
    image: gauravmishra0/cost-engine:stable-api
    container_name: pricing-api
    ports:
      - "9090:9090"
    environment:
      - DB_HOST=postgres
      - DB_PORT=5432
      - DB_USER=postgres
      - DB_PASSWORD=secure_password
      - DB_NAME=pricing
      - LOG_LEVEL=info
      - INGEST_BATCH_SIZE=2000
      - STARTUP_SYNC=false          # Auto-sync on container start
      - GCP_API_KEY=${GCP_API_KEY}  # Required for GCP ingestion
      # Cron scheduler configuration
      - SYNC_ENABLED=true           # Enables daily multi-cloud sync
      - SYNC_SCHEDULE=@daily        # Interval: @daily, @hourly, or cron format
      # Cloud-specific regions (comma-separated)
      - AWS_REGIONS=us-east-1,us-west-2
      - AZURE_REGIONS=eastus,westus2
      - GCP_REGIONS=us-central1,us-east1
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  mcp-server:
    image: gauravmishra0/cost-engine:stable-mcp
    container_name: cost-engine-mcp
    depends_on:
      - pricing-api
    environment:
      - API_URL=http://pricing-api:9090
    stdin_open: true
    tty: true
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    container_name: cost-engine-db
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=secure_password
      - POSTGRES_DB=pricing
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  postgres_data:

networks:
  default:
    name: cost-engine-network

Start the Engine Launch the stack in detached mode.
bash
```
docker compose up -d
```
Verify Health Ensure the API is running.
bash
```
curl http://localhost:9090/v1/health
```

Automated Ingestion Scheduler

The engine features a built-in Cron Scheduler designed to keep your pricing data synchronized across all supported clouds (AWS, Azure, GCP).

How to Configure

To create a scheduler, simply define the scheduling environment variables in your Docker configuration. No code changes or external cron jobs are required.

SYNC_ENABLED: Set to true to activate the background worker.
AWS_REGIONS: Comma-separated list of AWS regions (e.g., us-east-1,us-west-2).
AZURE_REGIONS: Comma-separated list of Azure regions (e.g., eastus,westus2).
GCP_REGIONS: Comma-separated list of GCP regions (e.g., us-central1,us-east1).
SYNC_SCHEDULE: Accepts standard Cron syntax.
- @daily: Run once at midnight.
- @hourly: Run once at the top of every hour.
- 0 30 * * *: Run every day at 30 minutes past midnight.

Behavior

When triggered, the scheduler performs the following operations sequentially:

AWS Sync: Fetches pricing for 48 supported services across all major categories (EC2, RDS, S3, Lambda, EKS, CloudFront, etc.) in configured regions.
Azure Sync: Pulls retail pricing updates for all available Azure services in configured regions (no filtering).
GCP Sync: Updates SKU catalog for 36 supported services across all major categories (Compute Engine, Cloud SQL, BigQuery, GKE, Cloud Run, etc.) in configured regions.

Note: Each sync operation runs independently. Services are synced automatically - no service selection required.

AI Agent Integration (MCP)

This engine is designed to give AI agents access to real-time cloud pricing. To connect Claude Desktop or other MCP clients:

Open your claude_desktop_config.json.
Add the following server configuration:

json

{
  "mcpServers": {
    "cost-engine": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "--network", "cost-engine-network",
        "-e", "API_URL=http://pricing-api:9090",
        "gauravmishra0/cost-engine:stable-mcp",
        "/app/mcp"
      ]
    }
  }
}

Or if using docker-compose, the MCP server is already running and can be accessed via stdio.

AWS Service Discovery Modes

The engine supports three modes for AWS service selection, giving you full control over which services to sync:

Mode 1: Curated (Default - Recommended)

Services include all major categories:

Compute: EC2, Lambda, ECS, EKS, Batch, Lightsail
Storage: S3, EBS, EFS, FSx, Backup, Glacier
Databases: RDS, DynamoDB, ElastiCache, Redshift, Neptune, DocumentDB, MemoryDB, Keyspaces, Timestream, QLDB
Networking: CloudFront, VPC, ELB, Route53, Direct Connect, API Gateway
Analytics: Athena, EMR, Kinesis, Glue, MSK, OpenSearch
ML/AI: SageMaker, Rekognition, Comprehend, Transcribe, Bedrock
Containers: ECR, App Runner
Integration: SNS, SQS, EventBridge
Security: Secrets Manager, WAF

Why use this: Avoids wasting time on 170+ AWS services that return 404 or have no pricing data. All 48 services are verified to have active pricing.

Mode 2: Dynamic Discovery

How it works:

Fetches AWS offers index (2MB, lists all services)
Attempts to sync each discovered service
Automatically filters by region during streaming
170+ services will return 404 or empty (AWS doesn't have pricing for many listed services)

Why use this: Automatically detects new AWS services without code updates. Good for periodic comprehensive scans to catch newly launched services.

Trade-offs:

✅ Zero maintenance - new services detected automatically
✅ No code changes when AWS launches new products
❌ 90+ minutes wasted on HTTP 404s and empty responses
❌ 8x slower than curated mode

Mode 3: Custom Service List

Why use this: When you only need specific services (e.g., only EC2 + RDS for compute-heavy workload). Fastest option for targeted deployments.

Configuration Examples

Docker Compose:

yaml

services:
  pricing-api:
    environment:
      # Option 1: Curated (default)
      - AWS_DISCOVERY_MODE=curated
      
      # Option 2: Dynamic discovery
      # - AWS_DISCOVERY_MODE=dynamic
      
      # Option 3: Custom services
      # - AWS_DISCOVERY_MODE=custom
      # - AWS_SERVICES=AmazonEC2,AmazonRDS,AmazonS3,AWSLambda,AmazonDynamoDB

Recommendation: Use curated for production and daily syncs. Run dynamic mode weekly/monthly in a separate job to discover new services, then add valuable ones to your curated list.

GCP Service Discovery Modes

The engine supports the same three modes for GCP service selection:

Mode 1: Curated (Default - Recommended)

Configuration: GCP_DISCOVERY_MODE=curated
Performance: ~5 minutes for single region, 100MB RAM peak
Coverage: 36 production-tested GCP services

Services include:

Compute: Compute Engine, Kubernetes Engine, Cloud Run, Cloud Functions, App Engine, Cloud GPUs
Storage & Databases: Cloud SQL, Cloud Storage, Bigtable, Spanner, Firestore, Memorystore (Redis/Memcached), Filestore, Persistent Disk, Cloud Backup
Networking: Cloud CDN, Load Balancing, VPN, Interconnect, Armor, DNS
Analytics: BigQuery, Dataflow, Dataproc, Pub/Sub, Composer, Data Fusion
ML & AI: Vertex AI, Vision API, Natural Language API, Translation API
Security: Secret Manager, KMS, Monitoring, Logging

Mode 2: Dynamic Discovery

Configuration: GCP_DISCOVERY_MODE=dynamic
Performance: ~20-30 minutes for single region, 150MB RAM peak
Coverage: Auto-discovers all 100+ GCP services from Cloud Billing API

Trade-offs:

✅ Detects new GCP services automatically
❌ 4-6x slower due to 100+ API calls
❌ Rate limit risk with high request volume

Mode 3: Custom Service List

Azure Optimization

Azure already uses an optimal approach:

Single paginated API for all services (no per-service calls needed)
Already streaming and filtering by region during ingestion
No service discovery modes needed - it's already efficient!

Azure syncs all available services automatically with minimal overhead (~5-7 minutes per region).

Cross-Cloud Performance Optimizations

Beyond service discovery, the engine includes several performance optimizations applicable to all clouds:

1. Parallel Processing

Environment Variable: MAX_CONCURRENT_SYNCS=3 (default)
What it does: Controls how many regions/services can sync simultaneously
Tuning:

1: Sequential (safest, slowest, ~300MB RAM)
3: Balanced (default, ~500MB RAM)
5-10: Aggressive (fast but 1GB+ RAM, may hit rate limits)

When to increase:

Large multi-region deployments
Powerful servers (8GB+ RAM)
When time is more critical than memory

When to decrease:

Low-memory environments (< 2GB)
Rate limit concerns
Single-region deployments

2. Batch Size Optimization

Environment Variable: INGEST_BATCH_SIZE=2000 (default)
What it does: Number of price records to accumulate before database insert
Tuning:

500-1000: Low memory, slower inserts
2000-3000: Balanced (recommended)
5000+: High memory, fastest inserts

Trade-off: Larger batches = fewer DB roundtrips but more RAM usage

3. Query Response Caching (Future Optimization)

Environment Variables:

ENABLE_QUERY_CACHE=false (default, not yet implemented)
CACHE_TTL_MINUTES=60

What it does: Caches identical cost queries to avoid repeated database lookups
Status: Configuration exists, implementation coming in v2.2

4. Streaming Buffer Size

Environment Variable: INGEST_STREAM_BUFFER_KB=64 (default)
What it does: Buffer size for JSON stream parsing (AWS primarily)
Tuning:

32-64: Low memory
128-256: Better performance
512+: Maximum throughput (AWS bulk files only)

5. Database Connection Pooling

Environment Variables:

DB_MAX_OPEN_CONNS=25 (default)
DB_MAX_IDLE_CONNS=5 (default)

What it does: Controls PostgreSQL connection pool size
Tuning for high concurrency:

Increase MAX_OPEN_CONNS to 50-100 for parallel syncs
Match MAX_CONCURRENT_SYNCS to avoid connection exhaustion

Configuration Example (Optimized for Speed)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=curated       # Fast, 48 services
      - GCP_DISCOVERY_MODE=curated       # Fast, 36 services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=5           # 5 parallel syncs
      - INGEST_BATCH_SIZE=3000           # Larger batches
      - INGEST_STREAM_BUFFER_KB=128      # Larger buffer
      - DB_MAX_OPEN_CONNS=50             # Match concurrency
      
      # Memory requirements: ~1.5-2GB

Configuration Example (Optimized for Memory)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=custom
      - AWS_SERVICES=AmazonEC2,AmazonRDS,AmazonS3  # Only 3 services
      - GCP_DISCOVERY_MODE=custom
      - GCP_SERVICES=Compute Engine,Cloud SQL      # Only 2 services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=1           # Sequential only
      - INGEST_BATCH_SIZE=1000           # Smaller batches
      - INGEST_STREAM_BUFFER_KB=32       # Smaller buffer
      
      # Memory requirements: ~300MB

Configuration Example (Comprehensive Discovery)

yaml

services:
  pricing-api:
    environment:
      # Service Discovery
      - AWS_DISCOVERY_MODE=dynamic       # All 200+ services
      - GCP_DISCOVERY_MODE=dynamic       # All 100+ services
      
      # Performance Tuning
      - MAX_CONCURRENT_SYNCS=3           # Moderate parallelism
      - INGEST_BATCH_SIZE=2000           # Standard batches
      
      # Expected: 2-4 hours, 2-3GB RAM, comprehensive coverage

API Reference

1. Ingestion and Syncing

Manually trigger updates for specific providers.

Sync AWS Data

bash

curl -X POST http://localhost:9090/v1/ingest/aws/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["us-east-1"]}'

Output:

json

{
  "results": [
    {
      "provider": "aws",
      "status": "success",
      "candidates_stored": 15420,
      "message": "Sync completed successfully"
    }
  ]
}

Sync Azure Data

Syncs all available Azure services (Virtual Machines, Storage, SQL Database, App Service, etc.) - no service filtering applied. Input:

bash

curl -X POST http://localhost:9090/v1/ingest/azure/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["eastus"]}'

Output:

json

{
  "results": [
    {
      "provider": "azure",
      "status": "success",
      "candidates_stored": 8500,
      "message": "Sync completed successfully"
    }
  ]
}

Sync GCP Data

bash

curl -X POST http://localhost:9090/v1/ingest/gcp/sync \
     -H "Content-Type: application/json" \
     -d '{"regions": ["us-central1"]}'

Output:

json

{
  "results": [
    {
      "provider": "gcp",
      "status": "success",
      "candidates_stored": 12000,
      "message": "Sync completed successfully"
    }
  ]
}

2. Metadata Discovery

Explore available data.

List Regions

Input:

bash

curl "http://localhost:9090/v1/metadata/regions?provider=aws"

Output:

json

{
  "provider": "aws",
  "regions": ["us-east-1", "us-west-2", "eu-central-1"],
  "count": 3
}

List Services

Input:

bash

curl "http://localhost:9090/v1/metadata/services?provider=azure&region=eastus"

Output:

json

{
  "provider": "azure",
  "region": "eastus",
  "services": ["Virtual Machines", "SQL Database", "Storage"],
  "count": 3
}

3. Cost Resolution

Calculate costs using the engine's resolution logic.

Single Resource Lookup

Input:

bash

curl -X POST http://localhost:9090/v1/cost/resolve \
     -H "Content-Type: application/json" \
     -d '{
       "provider": "aws",
       "region": "us-east-1",
       "service": "compute",
       "resource_type": "vm",
       "instance_type": "t3.medium",
       "os": "linux",
       "billing_model": "hourly",
       "hours": 730,
       "quantity": 1
     }'

Output:

json

{
  "provider": "aws",
  "region": "us-east-1",
  "instance_type": "t3.medium",
  "unit_price": 0.0416,
  "total_cost": 30.368,
  "currency": "USD",
  "effective_date": "2024-01-15T00:00:00Z"
}

Batch Resolution

Input:

bash

curl -X POST http://localhost:9090/v1/cost/resolve/batch \
     -H "Content-Type: application/json" \
     -d '{
       "requests": [
         {
           "provider": "aws", 
           "region": "us-east-1", 
           "instance_type": "t3.medium", 
           "service": "compute", 
           "resource_type": "vm", 
           "os": "linux", 
           "billing_model": "hourly", 
           "hours": 730, 
           "quantity": 10
         }
       ]
     }'

Output:

json

{
  "results": [
    {
      "provider": "aws",
      "instance_type": "t3.medium",
      "total_cost": 303.68,
      "currency": "USD"
    }
  ]
}

Developer Guide

Architecture Overview

The cost engine follows a microservices architecture with two primary components:

plain

┌─────────────────┐         ┌──────────────────┐
│   MCP Server    │────────▶│   Pricing API    │
│  (stdio/JSON)   │         │   (HTTP/REST)    │
└─────────────────┘         └──────────────────┘
                                     │
                                     ▼
                            ┌──────────────────┐
                            │   PostgreSQL     │
                            │  (Versioned DB)  │
                            └──────────────────┘

Component Responsibilities

1. Pricing API (cmd/api/main.go)

HTTP REST API on port 9090
Cloud provider sync orchestration (AWS, Azure, GCP)
Cost resolution engine
Background cron scheduler
Swagger documentation at /swagger/index.html

2. MCP Server (cmd/mcp/main.go)

Model Context Protocol server
Stdio-based JSON-RPC communication
Proxies requests to Pricing API
Designed for AI agent integration (Claude Desktop, etc.)

3. PostgreSQL Database

Immutable pricing versions
Time-based effective date resolution
Auto-creates schema on first connection

Database Schema

The engine uses a versioned, immutable pricing model:

Core Tables

pricing_versions - Pricing snapshots

sql

CREATE TABLE pricing_versions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    version_date DATE NOT NULL UNIQUE,          -- YYYY-MM-DD
    fetched_at TIMESTAMP NOT NULL,              -- Actual fetch time
    ingested_rows BIGINT,                       -- Number of prices
    status VARCHAR(50) NOT NULL,                -- in_progress | active | failed
    error_msg TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

prices - Individual pricing records

sql

CREATE TABLE prices (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    pricing_version_id UUID NOT NULL REFERENCES pricing_versions(id) ON DELETE CASCADE,
    provider VARCHAR(50) NOT NULL,              -- aws | azure | gcp
    service VARCHAR(100) NOT NULL,              -- compute | serverless | storage
    resource_type VARCHAR(100) NOT NULL,        -- vm | lambda | disk
    billing_model VARCHAR(50) NOT NULL,         -- hourly | usage-based
    region VARCHAR(100) NOT NULL,               -- us-east-1 | eastus | etc.
    instance_type VARCHAR(255) NOT NULL,        -- t3.medium | Standard_D2s_v3
    sku VARCHAR(255) NOT NULL,                  -- Cloud provider SKU
    unit VARCHAR(100) NOT NULL,                 -- hour | gb-second | request
    unit_price NUMERIC(18, 10) NOT NULL,        -- Price per unit
    currency VARCHAR(10) NOT NULL DEFAULT 'USD',
    attributes JSONB,                           -- Additional metadata
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

-- Performance indexes
CREATE INDEX idx_prices_provider_service ON prices(provider, service);
CREATE INDEX idx_prices_region_resource ON prices(region, resource_type);
CREATE INDEX idx_prices_instance_type ON prices(instance_type);

service_metadata - Available services catalog

sql

CREATE TABLE service_metadata (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    provider VARCHAR(50) NOT NULL,
    service VARCHAR(100) NOT NULL,
    resource_type VARCHAR(100) NOT NULL,
    billing_model VARCHAR(50) NOT NULL,
    regions TEXT[] NOT NULL,                    -- Array of supported regions
    last_updated TIMESTAMP NOT NULL,
    UNIQUE(provider, service, resource_type, billing_model)
);

Development Setup

Local Development (without Docker)

Install Dependencies

bash

# Go 1.24+
go mod download

# PostgreSQL 16+
brew install postgresql@16  # macOS
sudo apt install postgresql-16  # Ubuntu

# Swagger CLI (for API docs)
go install github.com/swaggo/swag/cmd/swag@latest

Setup Database

bash

# Create database
createdb pricing

# Schema is auto-created on first connection
# Or manually apply:
psql -U postgres -d pricing -f internal/database/schema.go

Environment Configuration

bash

# Create .env file
cat > .env <<EOF
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=
DB_NAME=pricing
LOG_LEVEL=debug
INGEST_BATCH_SIZE=2000
GCP_API_KEY=your_key_here
EOF

Build and Run

bash

# Build API
go build -o bin/api cmd/api/main.go

# Build MCP
go build -o bin/mcp cmd/mcp/main.go

# Run API
./bin/api

# Run MCP (in another terminal)
./bin/mcp

Generate Swagger Docs

bash

swag init -g cmd/api/main.go -o docs
# Visit http://localhost:9090/swagger/index.html

API Error Handling

All API endpoints return structured error responses:

Success Response (200/201)

json

{
  "results": [{"status": "success", ...}]
}

Error Response (4xx/5xx)

json

{
  "error": "error_code",
  "message": "Human-readable error description",
  "details": {
    "field": "additional_context"
  }
}

Common Error Codes

Testing

Unit Tests

bash

# Run all tests
go test ./...

# Run with coverage
go test -cover ./...

# Test specific package
go test ./internal/pricing/...

# Verbose output
go test -v ./internal/ingestion/...

Integration Tests

bash

# Start test database
docker run -d --name pricing-test-db \
  -e POSTGRES_PASSWORD=test \
  -p 5433:5432 postgres:16-alpine

# Run integration tests
DB_PORT=5433 go test -tags=integration ./...

# Cleanup
docker rm -f pricing-test-db

API Testing with curl

bash

# Health check
curl http://localhost:9090/v1/health

# Sync AWS (returns immediately, runs in background)
curl -X POST http://localhost:9090/v1/ingest/aws/sync \
  -H "Content-Type: application/json" \
  -d '{"regions":["us-east-1"]}'

# Check sync status (wait 30 seconds, then query)
sleep 30
psql -U postgres -d pricing -c \
  "SELECT version_date, status, ingested_rows FROM pricing_versions ORDER BY created_at DESC LIMIT 1;"

# Query pricing
curl -X POST http://localhost:9090/v1/cost/resolve \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "aws",
    "region": "us-east-1",
    "service": "compute",
    "resource_type": "vm",
    "instance_type": "t3.medium",
    "billing_model": "hourly",
    "os": "linux",
    "hours": 730
  }' | jq .

MCP Testing

bash

# Test MCP directly with JSON-RPC
echo '{
  "jsonrpc":"2.0",
  "id":1,
  "method":"tools/call",
  "params":{
    "name":"resolve_cost",
    "arguments":{
      "provider":"aws",
      "region":"us-east-1",
      "service":"compute",
      "resource_type":"vm",
      "instance_type":"t3.medium",
      "billing_model":"hourly"
    }
  }
}' | docker exec -i cost-engine-mcp /app/mcp

Performance Considerations

Memory Usage

AWS EC2 Full Sync: ~150MB peak (streaming JSON decoder)
Azure Full Sync: ~100MB peak
GCP Full Sync: ~120MB peak
Idle State: ~50MB

Sync Duration (typical)

AWS us-east-1: 45-90 minutes (48 services with 1000+ SKUs each)
Azure eastus: 5-8 minutes (all services, single API)
GCP us-central1: 15-25 minutes (36 services)

Database Growth

Per Region: ~10-15K pricing records
Storage: ~500KB per region (compressed)
Indexes: ~2x data size

Optimization Tips

Batch Size: Adjust INGEST_BATCH_SIZE (default: 2000)
- Lower = Less memory, slower inserts
- Higher = More memory, faster inserts
Connection Pool: Configure PostgreSQL max_connections
yaml
```
postgres:
  command: postgres -c max_connections=200
```

Parallel Sync: Run multiple region syncs in parallel

bash

# Sync multiple regions simultaneously
for region in us-east-1 us-west-2 eu-central-1; do
  curl -X POST http://localhost:9090/v1/ingest/aws/sync \
    -H "Content-Type: application/json" \
    -d "{\"regions\":[\"$region\"]}" &
done
wait

Debugging

Enable Debug Logging

yaml

environment:
  - LOG_LEVEL=debug  # info | debug | warn | error

Common Debug Scenarios

1. Sync failing silently

bash

# Check API logs
docker logs pricing-api --tail 100 | grep -i error

# Check database status
psql -U postgres -d pricing -c \
  "SELECT * FROM pricing_versions WHERE status='failed' ORDER BY created_at DESC LIMIT 5;"

2. MCP not responding

bash

# Verify MCP container is running
docker ps | grep mcp

# Check MCP logs
docker logs cost-engine-mcp --tail 50

# Test MCP connectivity
docker exec cost-engine-mcp curl -f http://pricing-api:9090/v1/health

3. Pricing data missing

bash

# List available regions
curl "http://localhost:9090/v1/metadata/regions?provider=aws" | jq .

# Check if region is synced
psql -U postgres -d pricing -c \
  "SELECT DISTINCT region FROM prices WHERE provider='aws';"

# Count prices per region
psql -U postgres -d pricing -c \
  "SELECT region, COUNT(*) FROM prices GROUP BY region ORDER BY count DESC;"

4. Database connection issues

bash

# Test from container
docker exec pricing-api ping -c 3 postgres

# Check PostgreSQL logs
docker logs cost-engine-db --tail 50

# Verify credentials
docker exec pricing-api env | grep DB_

Advanced Configuration

Custom Cron Schedules

yaml

# Every 6 hours
SYNC_SCHEDULE: "0 */6 * * *"

# Every Monday at 3 AM
SYNC_SCHEDULE: "0 3 * * 1"

# Every 15 minutes (not recommended - rate limiting)
SYNC_SCHEDULE: "*/15 * * * *"

Multi-Region Optimization

yaml

# Prioritize frequently-used regions
AWS_REGIONS: "us-east-1,us-west-2,eu-west-1"
AZURE_REGIONS: "eastus,westus2,westeurope"
GCP_REGIONS: "us-central1,europe-west1"

Startup Sync

yaml

# Auto-sync on container start (useful for cold starts)
STARTUP_SYNC: "true"

Cloud Provider Specifics

AWS

API: AWS Pricing API (bulk JSON)
Rate Limits: 10 requests/second
Regions: 20+ regions supported
Supported Services: 48 services including:
- Compute: EC2, ECS, EKS, Lambda, Batch, Lightsail
- Databases: RDS, DynamoDB, ElastiCache, Redshift, Neptune, DocumentDB, Keyspaces
- Storage: S3, EFS, FSx, S3 Glacier, Storage Gateway, Backup
- Networking: VPC, CloudFront, Route53, Global Accelerator, API Gateway
- Analytics: Athena, EMR, Kinesis, OpenSearch, Glue, QuickSight
- ML/AI: SageMaker, Rekognition, Comprehend, Transcribe
- Integration: SQS, SNS, MQ, Step Functions, EventBridge
- Developer Tools: CodeBuild, CodeDeploy, CodePipeline
- Security: Secrets Manager, KMS, Support Business
Update Frequency: Daily (AWS updates overnight)

Azure

API: Azure Retail Prices API (paginated)
Rate Limits: 100 requests/minute
Regions: 60+ regions supported
Supported Services: All Azure services (Virtual Machines, Storage, SQL Database, App Service, Cosmos DB, etc.)
Update Frequency: Real-time

GCP

API: Cloud Billing API (requires API key)
Rate Limits: 60 requests/minute
Regions: 30+ regions supported
Supported Services: 36 services including:
- Compute: Compute Engine, Cloud Run, GKE, Cloud Functions, App Engine
- Storage & Databases: Cloud SQL, Cloud Storage, Filestore, Bigtable, Spanner, Firestore, Memorystore
- Analytics & Big Data: BigQuery, Dataflow, Dataproc, Pub/Sub, Datastream
- Networking: Cloud CDN, Load Balancing, Cloud VPN, Cloud NAT, Cloud DNS, Cloud Armor
- AI/ML: Vertex AI, ML Engine, Vision API, Natural Language API
- Developer Tools: Cloud Build, Container Registry, Artifact Registry
- Operations: Cloud Logging, Cloud Monitoring, Cloud Trace
- Security: Cloud KMS, Secret Manager
Update Frequency: Daily

Contributing

Code Structure

plain

cost-engine/
├── cmd/
│   ├── api/          # HTTP API entrypoint
│   └── mcp/          # MCP server entrypoint
├── internal/
│   ├── api/
│   │   ├── router.go           # HTTP routes
│   │   └── handlers/           # Request handlers
│   ├── ingestion/              # Cloud provider adapters
│   │   ├── aws.go
│   │   ├── azure.go
│   │   └── gcp.go
│   ├── pricing/                # Cost resolution engine
│   │   ├── calculator.go
│   │   ├── eligibility.go
│   │   └── selector.go
│   ├── database/               # Data layer
│   │   ├── schema.go
│   │   └── db.go
│   └── domain/                 # Core models
├── docs/                       # Swagger docs (auto-generated)
├── migrations/                 # Database migrations
└── pkg/
    └── mcp/                    # MCP protocol implementation

Adding a New Cloud Provider

Create ingestion adapter in internal/ingestion/newcloud.go

type NewCloudAdapter struct {
    client *http.Client
}

func (a *NewCloudAdapter) FetchPricing(region string) ([]domain.PriceCandidate, error) {
    // Implement provider-specific API calls
}

Add handler in internal/api/handlers/newcloud_sync.go

func NewCloudSyncHandler(repo *database.Repository, log *logger.Logger) {
    // Implement sync endpoint
}

Register route in internal/api/router.go

api.POST("/ingest/newcloud/sync", handlers.NewCloudSync)

Update cron scheduler in cmd/api/main.go

// Add to scheduler job
newCloudHandler := handlers.NewNewCloudSyncHandler(repo, structLog)
newCloudErr := newCloudHandler.SyncProgrammatic(syncCtx, newCloudRegions)

Troubleshooting

Common Issues:

connection refused: Ensure Docker container is running and port 9090 is mapped correctly.
price_not_found: Verify that the specific region has been synced.
database_locked: The engine allows only one sync operation per provider at a time.

For additional support, please open an issue in the repository.

Production Deployment Guide

This section is for customers integrating the pricing engine into their own application stack.

1. Sidecar Deployment (Kubernetes)

For low-latency access, deploy the engine as a sidecar container in your application pod.

yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  containers:
    # Your Main Application
    - name: my-app
      image: my-company/app:latest
      env:
        - name: COST_ENGINE_URL
          value: "http://localhost:9090"

    # Multi-Cloud Pricing Engine Sidecar
    - name: pricing-api
      image: gauravmishra0/cost-engine:stable-api
      ports:
        - containerPort: 9090
      env:
        - name: DB_HOST
          value: "postgres-service"
        - name: DB_USER
          value: "postgres"
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secrets
              key: password
        - name: DB_NAME
          value: "pricing"
        - name: SYNC_ENABLED
          value: "true"
        - name: SYNC_SCHEDULE
          value: "@daily"
        - name: AWS_REGIONS
          value: "us-east-1,us-west-2"
        - name: AZURE_REGIONS
          value: "eastus,westus2"
        - name: GCP_REGIONS
          value: "us-central1"
        - name: GCP_API_KEY
          valueFrom:
            secretKeyRef:
              name: gcp-secrets
              key: api-key

2. Standalone Service (Docker Compose)

For centralized access across multiple microservices, deploy as a standalone service.

bash

# 1. Create a dedicated network
docker network create pricing-net

# 2. Run Database
docker run -d --name pricing-db --net pricing-net \
  -e POSTGRES_PASSWORD=prod_pass \
  postgres:16-alpine

# 3. Run Engine
docker run -d --name pricing-api --net pricing-net \
  -p 9090:9090 \
  -e DB_HOST=pricing-db \
  -e DB_USER=postgres \
  -e DB_PASSWORD=prod_pass \
  -e DB_NAME=pricing \
  -e SYNC_ENABLED=true \
  -e SYNC_SCHEDULE="0 3 * * *" \
  -e AWS_REGIONS="us-east-1,us-west-2" \
  -e AZURE_REGIONS="eastus" \
  -e GCP_REGIONS="us-central1" \
  -e GCP_API_KEY=your_gcp_key \
  gauravmishra0/cost-engine:stable-api

3. Sizing Recommendations

Small Scale (Under 10k requests/day): 1 vCPU, 512MB RAM (Sufficient for background ingestion of hourly datasets)
Large Scale (Over 1M requests/day): 2 vCPU, 4GB RAM (Recommended to handle memory spikes during full-catalog syncs)
Storage: 50GB SSD persistent volume recommended for the PostgreSQL backend.

Multi-Cloud Pricing Engine Documentation

Multi-Cloud Pricing Engine

Getting Started

Prerequisites

Installation

Automated Ingestion Scheduler

How to Configure

Behavior

AI Agent Integration (MCP)

AWS Service Discovery Modes

Mode 1: Curated (Default - Recommended)

Mode 2: Dynamic Discovery

Mode 3: Custom Service List

Configuration Examples

GCP Service Discovery Modes

Mode 1: Curated (Default - Recommended)

Mode 2: Dynamic Discovery

Mode 3: Custom Service List

Azure Optimization

Cross-Cloud Performance Optimizations

1. Parallel Processing

2. Batch Size Optimization

3. Query Response Caching (Future Optimization)

4. Streaming Buffer Size

5. Database Connection Pooling

Configuration Example (Optimized for Speed)

Configuration Example (Optimized for Memory)

Configuration Example (Comprehensive Discovery)

API Reference

1. Ingestion and Syncing

Sync AWS Data

Sync Azure Data

Sync GCP Data

2. Metadata Discovery

List Regions

List Services

3. Cost Resolution

Single Resource Lookup

Batch Resolution

Developer Guide

Architecture Overview

Component Responsibilities

Database Schema

Core Tables

Development Setup

Local Development (without Docker)

API Error Handling

Common Error Codes

Testing

Unit Tests

Integration Tests

API Testing with curl

MCP Testing

Performance Considerations

Memory Usage

Sync Duration (typical)

Database Growth

Optimization Tips

Debugging

Enable Debug Logging

Common Debug Scenarios

Advanced Configuration

Custom Cron Schedules

Multi-Region Optimization

Startup Sync

Cloud Provider Specifics

AWS

Azure

GCP

Contributing

Code Structure

Adding a New Cloud Provider

Troubleshooting

Production Deployment Guide

1. Sidecar Deployment (Kubernetes)

2. Standalone Service (Docker Compose)

3. Sizing Recommendations

Multi-Cloud Pricing Engine Documentation

Multi-Cloud Pricing Engine

Getting Started