Self-Hosted Observability Stack

This project runs a reusable observability platform for all of your projects.

It uses:

• OpenTelemetry Collector as the single ingestion endpoint for every app.
• OpenObserve for logs, metrics, traces, dashboards, alerts, and general APM.
• Phoenix for LLM, RAG, agent, prompt, dataset, and evaluation observability.
• PostgreSQL for durable Phoenix storage.
• Caddy as the reverse proxy and TLS endpoint for browser access and OTLP HTTP ingestion.
• Local and S3 backup scripts for disaster recovery.

The important design decision is that your applications talk to OpenTelemetry, not to OpenObserve or Phoenix directly. This keeps your projects portable if you later replace or add backends.

Architecture

Application services
  -> OpenTelemetry SDK / auto-instrumentation
  -> OTLP HTTP or gRPC
  -> OpenTelemetry Collector
      -> OpenObserve for logs, metrics, traces
      -> Phoenix for LLM traces

Browser users
  -> Caddy HTTPS reverse proxy
      -> OpenObserve UI
      -> Phoenix UI

Default exposed endpoints (DuckDNS gives one hostname, so Caddy routes by path):

Purpose	Endpoint
OpenObserve UI	https://observability.duckdns.org/
Phoenix UI	https://observability.duckdns.org/phoenix
OTLP HTTP ingest	https://observability.duckdns.org/v1/traces, /v1/metrics, /v1/logs
Raw OTLP HTTP ingest	http://host:4318, bound to localhost by default
Raw OTLP gRPC ingest	http://host:4317, bound to localhost by default

For production, always send telemetry to the https://observability.duckdns.org/v1/* paths through Caddy, not raw 4318.

Files

File	Purpose
docker-compose.yml	Runs OpenObserve, Phoenix, PostgreSQL, Collector, and Caddy.
config/otel-collector.yaml	Receives OTLP and exports telemetry to OpenObserve and Phoenix.
caddy/Caddyfile	Routes public HTTPS hostnames to the internal services.
.env.example	Template for credentials, domains, stream names, and backup settings.
scripts/backup-local.sh	Creates local backups of OpenObserve data, Phoenix PostgreSQL, config, and .env.
scripts/backup-s3.sh	Creates a local backup, then syncs it to S3 with server-side encryption.
scripts/restore-local.sh	Restores OpenObserve data and Phoenix PostgreSQL from a local backup folder.
USER_MANAGEMENT.md	Guide for creating and managing users in OpenObserve and Phoenix.
AWS_SETUP.md	Infrastructure and configuration steps for AWS deployment.

Initial Setup

1. Copy the environment template:

   cp .env.example .env

2. Edit .env and replace every change-me and replace value.

3. Generate the OpenObserve auth header:

   printf '%s' 'admin@example.com:your-openobserve-password' | base64

   Put the result into .env like this:

   OPENOBSERVE_AUTH_HEADER=Basic pasted_base64_value

4. Generate the Collector ingest password hash:

   docker run --rm httpd:2.4-alpine htpasswd -nbB otel_ingest 'your-ingest-password'

   Put the entire output into .env, but escape every $ as $$ because Docker Compose treats $ as interpolation syntax:

   OTEL_COLLECTOR_HTPASSWD=otel_ingest:$$2y$$05$$...

5. Generate a Phoenix secret:

   openssl rand -hex 32

   Put it into:

   PHOENIX_SECRET=generated_value

6. Start the stack:

   docker compose up -d

7. Open Phoenix, log in as the initial admin, then create a system API key:

   Email: admin@localhost
   Password: PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD from .env
   Phoenix UI -> Settings -> API Keys -> System API Key
   

8. Add that key to .env:

   PHOENIX_API_KEY=your_system_api_key

9. Restart the Collector:

   docker compose up -d otel-collector

Until PHOENIX_API_KEY is set, OpenObserve can receive traces, logs, and metrics, but Phoenix export will fail authentication.

Local Development

For local-only development without DNS:

1. Set this in .env:

   PUBLIC_DOMAIN=observability.local:8080

2. Add this to /etc/hosts on this machine:

   127.0.0.1 observability.local
   

3. For other devices on the LAN, add the same name to their hosts files, but point it to this machine's LAN IP:

   192.168.x.x observability.local
   

4. Use these local endpoints:

   OpenObserve: http://observability.local:8080
   Phoenix: http://observability.local:8080/phoenix
   Collector HTTP: http://observability.local:8080/v1/traces
   Collector HTTP: http://localhost:4318
   Collector gRPC: http://localhost:4317
   

For a public deployment, set HTTP_BIND=80 and HTTPS_BIND=443.

The Compose file intentionally does not expose OpenObserve or Phoenix directly. Caddy is the browser entry point. If you want raw local UI ports during development, temporarily add ports to the relevant services.

Hosting This on a Home Laptop Behind DuckDNS

This stack is designed to run on a machine you keep at home (e.g. a laptop) while remote services, such as an app on AWS, send telemetry to it over the internet.

1. Register observability at duckdns.org and copy your token.

2. In .env, set:

   PUBLIC_DOMAIN=observability.duckdns.org
   ACME_EMAIL=you@example.com
   HTTP_BIND=80
   HTTPS_BIND=443
   DUCKDNS_SUBDOMAIN=observability
   DUCKDNS_TOKEN=your-duckdns-token

3. On your home router, forward external ports 80/tcp and 443/tcp to this machine's LAN IP. These are the only ports that need to be reachable from the internet.

4. Run docker compose up -d. The duckdns service keeps observability.duckdns.org pointed at your current public IP (important on a residential connection where the IP can change), and Caddy requests a Let's Encrypt certificate for that hostname automatically once DNS resolves and ports 80/443 are reachable from the internet.

5. From the AWS side, point the app's OTLP exporter at https://observability.duckdns.org (see "Sending Telemetry From Projects" below). No inbound rule is needed on the AWS side — it's an outbound-only connection to your home IP.

Public Server Deployment (EC2 or similar)

Recommended shape for a small team:

• Ubuntu LTS.
• Docker Engine and Docker Compose plugin.
• At least 2 vCPU, 4 GB RAM for light use.
• 8 GB RAM or more if you ingest many logs/traces.
• A persistent volume mounted where this repo lives.
• Firewall/security group allowing:
  • 22/tcp from your IP only.
  • 80/tcp and 443/tcp from users/apps that need access.
  • Do not expose 5080, 6006, 5432, or raw 4317/4318 publicly.

DNS (or DuckDNS, as above) should point observability.duckdns.org at this host's public IP.

Caddy will request a TLS certificate automatically once the domain resolves correctly and ports 80/443 are reachable.

Sending Telemetry From Projects

Every app should send to the Collector, not directly to OpenObserve or Phoenix.

For OTLP HTTP:

OTEL_SERVICE_NAME=my-service
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://observability.duckdns.org
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic base64_of_otel_ingest_colon_password
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

For local raw HTTP:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic base64_of_otel_ingest_colon_password

The Basic header for applications is not the htpasswd hash. It is the base64 value of:

otel_ingest:your-ingest-password

Generate it with:

printf '%s' 'otel_ingest:your-ingest-password' | base64

Trace ID Correlation

The Collector forwards the same trace to both backends. That means a trace ID created by your app should be searchable in both OpenObserve and Phoenix.

Use the same OpenTelemetry trace context across normal application spans and LLM spans:

• Do not create separate root traces for LLM calls if they are part of a request.
• Let OpenInference, LangChain, LlamaIndex, OpenAI SDK instrumentation, or your manual spans run under the active request context.
• Include service.name, deployment.environment, service.version, and important domain identifiers like tenant.id, user.id, request.id, or conversation.id where allowed by your privacy policy.

In OpenObserve, use the trace view for request-level debugging and the logs view for log-to-trace correlation. In Phoenix, use the same trace ID to inspect LLM call inputs, outputs, tool calls, retrieval steps, and evaluations.

Security Model

Browser Access

OpenObserve has its own login controlled by:

OPENOBSERVE_ROOT_EMAIL
OPENOBSERVE_ROOT_PASSWORD
OPENOBSERVE_RETENTION_DAYS

Phoenix auth is enabled with:

PHOENIX_ENABLE_AUTH=True
PHOENIX_SECRET
PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD
PHOENIX_RETENTION_DAYS

For detailed instructions on adding more users, see USER_MANAGEMENT.md.

After the first Phoenix startup, changing PHOENIX_DEFAULT_ADMIN_INITIAL_PASSWORD does not reset the admin password. Change it inside Phoenix or follow Phoenix admin password reset procedures.

Ingestion Access

The Collector requires Basic auth before accepting OTLP data:

OTEL_COLLECTOR_HTPASSWD=otel_ingest:hashed_password

Applications send:

Authorization: Basic base64(otel_ingest:plain_password)

Do not expose OpenObserve ingestion credentials to applications. Only the Collector should know OPENOBSERVE_AUTH_HEADER.

Network Rules

Use these rules in production:

• Expose only Caddy on ports 80 and 443.
• Keep PostgreSQL internal to Docker.
• Keep OpenObserve and Phoenix internal to Docker.
• Keep raw Collector gRPC bound to localhost unless you explicitly need it remotely.
• Prefer OTLP HTTP over HTTPS through Caddy for remote services.
• Restrict SSH to your IP address.
• Use AWS Security Groups, OS firewall rules, or both.

Secrets

Do not commit .env. It contains:

• OpenObserve admin password.
• OpenObserve ingestion auth header.
• Phoenix PostgreSQL password.
• Phoenix JWT signing secret.
• Phoenix admin bootstrap password.
• Phoenix API key.
• Collector ingest hash.

For EC2, store a second encrypted copy in AWS Systems Manager Parameter Store, Secrets Manager, or an encrypted S3 location.

Data Storage

OpenObserve stores local data under:

./data/openobserve

Phoenix stores durable data in PostgreSQL under:

./data/phoenix-postgres

Caddy stores TLS certificates under:

./data/caddy

Do not delete ./data unless you are intentionally resetting the stack.

Retention

The stack defaults to a 30-day retention cap for both systems:

OPENOBSERVE_RETENTION_DAYS=30
PHOENIX_RETENTION_DAYS=30

OpenObserve uses its compaction retention policy to delete older stream data. Phoenix uses its default project retention policy so new projects inherit the 30-day cap unless you override them in the UI.

Backups

Backups include:

• OpenObserve local data directory.
• Phoenix PostgreSQL dump.
• Runtime config and .env.

Run a local backup:

./scripts/backup-local.sh

The backup is written to:

./backups/YYYYMMDDTHHMMSSZ

Run an S3 backup:

./scripts/backup-s3.sh

Configure S3 in .env:

AWS_S3_BACKUP_URI=s3://your-bucket/observability
AWS_PROFILE=default

The S3 script uses:

aws s3 sync ... --sse AES256

For stronger control, enable bucket versioning, default encryption with KMS, lifecycle retention, and restricted IAM permissions.

Minimum IAM permissions for backup upload:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/observability/*"
      ]
    }
  ]
}

For restore from S3, add s3:GetObject.

Backup Schedule

For a local server:

15 2 * * * cd /path/to/observability && ./scripts/backup-local.sh >> ./backups/backup.log 2>&1

For EC2 with S3:

15 2 * * * cd /path/to/observability && ./scripts/backup-s3.sh >> ./backups/backup.log 2>&1

Recommended retention:

• Local: 7 to 14 daily backups.
• S3: 30 to 90 daily backups.
• Monthly archive: 6 to 12 months if logs are business-critical.

Restore

Restore from a local backup:

./scripts/restore-local.sh ./backups/YYYYMMDDTHHMMSSZ

Restore the latest backup from S3 back onto the EBS-backed local directories:

./scripts/restore-s3.sh

Restore a specific backup folder from S3:

./scripts/restore-s3.sh YYYYMMDDTHHMMSSZ

The restore process:

1. Stops Collector, Phoenix, and OpenObserve.
2. Replaces data/openobserve from backup.
3. Recreates the Phoenix PostgreSQL database.
4. Restores the Phoenix dump.
5. Starts the stack again.

Always test restore on a separate machine or directory before trusting backups.

Operations

Start:

docker compose up -d

Stop:

docker compose down

Restart one service:

docker compose restart otel-collector

View logs:

docker compose logs -f otel-collector
docker compose logs -f openobserve
docker compose logs -f phoenix

Upgrade images:

./scripts/backup-local.sh
docker compose pull
docker compose up -d

Do not upgrade without a fresh backup.

Retention and Cost Control

Start conservative:

• Use sampling in applications for very high-volume traces.
• Avoid putting secrets, full prompts, full completions, access tokens, or private user data into spans.
• Use stable attributes with bounded cardinality.
• Avoid high-cardinality labels like raw URLs, full SQL queries, email addresses, or unbounded user text.
• Keep verbose debug logs out of production unless you are actively investigating.

Recommended attributes:

service.name
service.version
deployment.environment
http.route
http.method
http.status_code
db.system
messaging.system
llm.model_name
llm.provider
openinference.span.kind

Use OpenObserve dashboards and alerts for service health. Use Phoenix projects and datasets for LLM workflows.

LLM Observability Guidance

Phoenix is most useful when your LLM spans include:

• Model provider and model name.
• Prompt template or prompt identifier.
• Input/output token counts.
• Latency.
• Tool calls.
• Retrieval query.
• Retrieved document IDs and scores.
• Evaluation results.
• Error messages and retry counts.

Be careful with prompt and response payloads. If they may contain secrets, customer data, or personal data, redact them before export or only export metadata.

Production Checklist

Before exposing this stack:

• Replace all placeholder secrets in .env.
• Confirm OpenObserve login works.
• Confirm Phoenix login works.
• Create Phoenix system API key and restart Collector.
• Confirm app telemetry reaches OpenObserve.
• Confirm LLM traces reach Phoenix.
• Confirm Caddy HTTPS certificates are issued.
• Confirm Security Group exposes only 80/443 and restricted SSH.
• Run ./scripts/backup-local.sh.
• Run ./scripts/backup-s3.sh on EC2.
• Test restore in a separate directory or host.
• Document who has admin access.
• Rotate ingestion credentials if a developer leaves or a project is compromised.

References

• OpenObserve OTLP ingestion: https://openobserve.ai/docs/ingestion/logs/otlp/
• OpenObserve traces: https://openobserve.ai/docs/user-guide/data-exploration/traces/traces/
• Phoenix self-hosting: https://arize.com/docs/phoenix/self-hosting/deploying-phoenix
• Phoenix configuration: https://arize.com/docs/phoenix/self-hosting/configuration
• Phoenix authentication: https://arize.com/docs/phoenix/self-hosting/authentication
• OpenTelemetry Collector configuration: https://opentelemetry.io/docs/collector/configuration/