Caddy

caddy logo

Table of contents

Background

Caddy is a modern server that is deployable for modern web services. With the increasing complexity and style of deployment of many web services, examining the behavior of the entire system quickly becomes very complex, and gains from any updates, optimizations or load shedding become very hard to qualify or examine.

To fill this void, Caddy rightfully so deserves modern observability.

Modern observability on the web server means that any operations after a request hits the web server can be observed, measured and examined irrespective of the destination of the request, whatever centric serving it performs or the big picture system behavior.

Distributed Tracing and Monitoring is the mechanism by which we can gain insights into the behavior of a distributed system.

Tracing gives us timing and territorial information about the progression of a request. Using context propagation on the transport e.g HTTP, we can send over information between remote services and after completion examine the propagations on various backends, without any vendor lock-in or any single cloud lock-in.

With monitoring/metric collection, we can collect any quantifiable metrics such as: * Client and Server latency * Memory statistics * Runtime behavior

An added advantage of vendor agnostic distributed tracing is that we can then export these traces and metrics simultaneously to a plethora of backends that your site reliability engineers and generally other developers can examine on such as: * Instana * Prometheus/Grafana * AWS X-Ray * Zipkin * Jaeger * DataDog * Stackdriver Monitoring and Tracing and many more

However, the distributed tracing and monitoring framework should provide very low latency and optionality so that users of the server will not incur expensive overhead. Also the addition of observability the web server should not add a maintainence burden for your teams, the project maintainers either nor should they require sophisticated and specialized distributed systems and observability knowledge, nor should it require sophisicated infrastructure deployments.

Enabling Observability

caddy -observability "<sampler_rate>;exporter1:[config1Key=config1Value:config2Key=config2Value...][,]exporter2...]"
Git checkout

You can enable observability by checking out the instrumented branch of caddy

go get github.com/mholt/caddy/caddy
cd $GOPATH/src/github.com/mholt/caddy
git add orijtech git@github.com:orijtech/caddy.git && git fetch orijtech && git checkout instrument-with-opencensus && go get ./caddy
Syntax
observability   := SamplerRate;Exporters
SamplerRate     := float64 value
Exporters       := [ExporterConfig](,ExporterConfig)?
ExporterConfig  := Name:[Key-ValuePairs]*
Name            := collection of symbolic names for exporters
Key-Value Pair  := KeyToken=ValueToken
KeyToken        := string
ValueToken      := string
Variables Table
Exporter Name Key Type Notes Example
aws-xray AWS_REGION String The region that your project is located in AWS_REGION=us-west-2
aws-xray AWS_ACCESS_KEY_ID Your access key AWS_ACCESS_KEY_ID=keyID
aws-xray AWS_SECRET_ACCESS_KEY Your access key AWS_SECRET_ACCESS_KEY=secretKey
jaeger agent The URL of the Jaeger caddy -observability "jaeger:agent=localhost:6831"
jaeger collector The URL of the Jaeger caddy -observability "jaeger:collector=http://localhost:9411"
jaeger service-name The service name when inspected by Jaeger caddy -observability "jaeger:service-name=search_endpoint"
prometheus port The port that will be scraped from your prometheus.yml file caddy -observability "prometheus:port=9987"
stackdriver GOOGLE_APPLICATION_CREDENTIALS File path for value The credentials for your Google Cloud Platform project GOOGLE_APPLICATION_CREDENTIALS=~/creds.json caddy -observability "1;stackdriver:tracing=true"
stackdriver monitoring boolean for value A commandline option to toggle monitoring caddy -observability "stackdriver:monitoring=true"
stackdriver tracing boolean for value A commandline option to toggle tracing caddy -observability "stackdriver:tracing=true"
stackdriver project-id string for value A commandline option caddy -observability "stackdriver:project-id=census-demos"
zipkin local URL The URL of the local endpoint caddy -observability "zipkin:local=192.168.1.5:5454"
zipkin reporter URL The URL of the reporter endpoint caddy -observability "zipkin:reporter=http://localhost:9411/api/v2/spans"
zipkin service-name string The name of your service caddy -observability "service-name=server"
Examples
GOOGLE_APPLICATION_CREDENTIALS=./creds.json AWS_REGION=us-west-2 AWS_ACCESS_KEY_ID=keyId AWS_SECRET_ACCESS_KEY=secretKey \
  caddy -observability "0.9;zipkin,prometheus:port=8999,aws-xray,stackdriver:tracing=true:monitoring=true:project-id=census-demos,jaeger:agent=localhost:6831,service-name=search-endpoint"