HTTP Server

The dftracer_server provides a high-performance HTTP server for querying and streaming DFTracer trace data via REST API. It uses bloom filter indexing to accelerate event searches and supports filtering, aggregation, and visualization API endpoints.

Starting the Server

Basic startup:

dftracer_server -d /path/to/traces

The server scans the trace directory on startup, loads or builds bloom/checkpoint sidecar indexes (.idx files), and begins listening for HTTP requests on 127.0.0.1:8080. It also serves the interactive Trace Viewer (Web UI) web UI at / and /index.html.

Custom Configuration:

# Expose on all interfaces (see Security below), port 9000
dftracer_server -b 0.0.0.0 -p 9000 -d /path/to/traces

# Use separate index directory (useful for NFS or slow disks)
dftracer_server -d /path/to/traces --index-dir /var/cache/dftracer_indexes

# Use 16 worker threads for concurrent request handling
dftracer_server -d /path/to/traces --executor-threads 16

Security

The server binds to loopback (127.0.0.1) by default, so it is not reachable from other hosts unless you explicitly pass -b 0.0.0.0 (or another address). When exposing it beyond localhost, set an access token:

dftracer_server -d /path/to/traces -b 0.0.0.0 --token "$(openssl rand -hex 16)"

With --token set, every request must present the token, either as a ?token=<TOKEN> query parameter or an Authorization: Bearer <TOKEN> header; otherwise the server responds 401 Unauthorized. Responses carry an open CORS header (Access-Control-Allow-Origin: *) so browser and editor webview clients can query the API cross-origin.

REST API

All endpoints return JSON responses and support filtering via query parameters. The server uses HTTP/1.1 with keep-alive connections for efficient streaming.

Trace Data API

GET /api/v1/files

List all available trace files in the directory.

Response:

{
    "files": [
        {
            "path": "trace1.pfw.gz",
            "has_bloom_data": true,
            "has_checkpoint_index": true,
            "is_small": false
        }
    ],
    "count": 1
}

GET /api/v1/files/info

Get detailed metadata for a specific file.

Query Parameters:

  • file (string) - Path to the trace file (e.g., trace1.pfw.gz) [required]

Response:

{
    "path": "trace1.pfw.gz",
    "has_bloom_data": true,
    "has_checkpoint_index": true,
    "is_small": false,
    "size_mb": 45.2,
    "compressed_size": 47185920,
    "num_lines": 1234567,
    "num_checkpoints": 29,
    "uncompressed_size": 943718400
}

Note

num_lines, num_checkpoints, and uncompressed_size are only included for non-small (indexed) files.

GET /api/v1/events

Query events with optional filtering. Returns results as streaming NDJSON using HTTP/1.1 chunked transfer encoding.

Query Parameters:

  • file (string) - Specific trace file to query (default: all files)

  • limit (integer) - Maximum number of events to return, 0-100000 (default: 1000)

  • cat (string) - Category filter (comma-separated for multiple values)

  • name (string) - Event name filter (comma-separated for multiple values)

  • pid (integer) - Process ID filter

  • ts_min (double) - Minimum timestamp in microseconds

  • ts_max (double) - Maximum timestamp in microseconds

  • dur_min (double) - Minimum duration in microseconds

  • dur_max (double) - Maximum duration in microseconds

Response: Streaming NDJSON (one JSON object per line)

  • Content-Type: application/x-ndjson

  • Transfer-Encoding: chunked

  • X-Limit: Echoes the limit applied

{"name":"MPI_Send","ph":"X","ts":1234567890,"dur":12345,"pid":1,"tid":1,"args":{}}
{"name":"MPI_Recv","ph":"X","ts":1234567950,"dur":200,"pid":1,"tid":1,"args":{}}

Example:

# Get first 100 MPI_Send events
curl "http://localhost:8080/api/v1/events?file=trace1.pfw.gz&name=MPI_Send&limit=100"

# Get events with specific duration threshold
curl "http://localhost:8080/api/v1/events?file=trace1.pfw.gz&dur_min=1000&limit=50"

# Filter by category and time range
curl "http://localhost:8080/api/v1/events?file=trace1.pfw.gz&cat=POSIX&ts_min=1000000&ts_max=2000000"

GET /api/v1/events/stream

Stream events as NDJSON without a limit. Identical to /api/v1/events but limit defaults to 0 (unlimited). Useful for large result sets and real-time processing.

Query Parameters: Same as /api/v1/events (but limit defaults to 0)

Response: Streaming NDJSON with chunked transfer encoding (same format as /api/v1/events)

{"name":"MPI_Send","ph":"X","ts":1234567890,"dur":12345,"pid":1,"tid":1,"args":{}}
{"name":"MPI_Recv","ph":"X","ts":1234567950,"dur":200,"pid":1,"tid":1,"args":{}}

Example:

# Stream all events with duration > 5000 microseconds
curl "http://localhost:8080/api/v1/events/stream?file=trace1.pfw.gz&dur_min=5000"

GET /api/v1/stats

Retrieve aggregated statistics across all trace files. Results are cached in-memory by request path.

Query Parameters: None

Response:

{
    "file_count": 3,
    "total_events": 3704701,
    "skipped_small_files": 0,
    "files": [
        {
            "file_path": "trace1.pfw.gz",
            "success": true,
            "total_events": 1234567,
            "num_categories": 5,
            "num_unique_names": 42,
            "num_pid_tids": 8,
            "time_range": {
                "min_timestamp_us": 1000000,
                "max_timestamp_us": 5000000,
                "time_span_seconds": 4.0
            },
            "duration": {
                "count": 1234567,
                "sum_us": 98765432,
                "mean_us": 80.0,
                "stddev_us": 15.2,
                "min_us": 1,
                "max_us": 45000
            },
            "category_counts": {"POSIX": 500000, "APP": 734567},
            "name_counts": {"read": 250000, "write": 250000},
            "pid_tid_counts": {"1:1": 600000, "1:2": 634567}
        }
    ]
}

GET /api/v1/info

Get global metadata about all trace files (time bounds, file listing).

Response:

{
    "file_count": 3,
    "time_range": {
        "min_timestamp_us": 1000000,
        "max_timestamp_us": 10000000
    },
    "files": [
        {
            "path": "trace1.pfw.gz",
            "has_bloom_data": true,
            "has_checkpoint_index": true,
            "is_small": false,
            "min_timestamp_us": 1000000,
            "max_timestamp_us": 5000000
        },
        {
            "path": "trace2.pfw.gz",
            "has_bloom_data": true,
            "has_checkpoint_index": true,
            "is_small": false,
            "min_timestamp_us": 3000000,
            "max_timestamp_us": 7000000
        },
        {
            "path": "trace3.pfw.gz",
            "has_bloom_data": true,
            "has_checkpoint_index": true,
            "is_small": false,
            "min_timestamp_us": 5000000,
            "max_timestamp_us": 10000000
        }
    ]
}

Visualization API

GET /api/v1/viz/events

Query events optimized for visualization with time-range windowing, lane grouping, and summary aggregation. Returns events binned and aggregated for efficient rendering in trace viewers.

Query Parameters:

  • begin (double) - Time window start [required] (normalized if ts_normalize=1)

  • end (double) - Time window end [required] (normalized if ts_normalize=1)

  • summary (integer) - Summary level [required] (1=full detail, higher=aggregate shorter events)

  • file (string) - Specific trace file to query (default: all files)

  • ts_normalize (integer) - If 1 (default), normalize timestamps relative to global minimum; if 0, use raw timestamps

  • limit (integer) - Maximum events to return, 0=unlimited (default: 0)

  • pid (integer) - Process ID filter

  • tid (integer) - Thread ID filter

  • cat (string) - Category filter

  • lanes (JSON) - Lane filtering as URL-encoded JSON (see below)

  • filters (JSON) - Complex filtering as URL-encoded JSON array (see below)

Response:

{
    "events": [
        {
            "name": "MPI_Send",
            "ph": "X",
            "ts": 234567,
            "dur": 12345,
            "pid": 1,
            "tid": 1,
            "args": {}
        }
    ],
    "metadata": {
        "begin": 0,
        "end": 1000000,
        "count": 1,
        "limit": 0,
        "truncated": false,
        "ts_normalized": true,
        "global_min_timestamp_us": 1000000
    }
}

Timestamp Normalization:

When ts_normalize=1 (default), the client sends begin/end values relative to the global minimum timestamp. The server de-normalizes internally for queries and normalizes the response events. The global_min_timestamp_us field is included for client-side denormalization if needed.

Summary Level:

The summary level controls a minimum duration threshold: time_range / (viewport_width * summary_level) (viewport_width defaults to 1920). Events below this threshold are filtered out at higher summary levels.

Lane Filtering:

Filter events by lane using URL-encoded JSON:

# Filter by process ID
curl "http://localhost:8080/api/v1/viz/events?begin=0&end=1000000&summary=1&lanes=%5B%7B%22field%22%3A%22pid%22%2C%22value%22%3A%221%22%7D%5D"

The lanes parameter accepts a JSON array or object:

[{"field": "pid", "value": "1"}]

Complex Filtering:

The filters parameter accepts a URL-encoded JSON array with field/operator/value objects:

[
    {"field": "pid", "op": "=", "value": 1},
    {"field": "dur", "op": ">=", "value": 0}
]

Supported operators: =, >=, <=, >, <

Example:

# Get events for visualization in normalized time range [0, 1M]
curl "http://localhost:8080/api/v1/viz/events?begin=0&end=1000000&summary=1"

# Same query with raw timestamps and PID filter
curl "http://localhost:8080/api/v1/viz/events?begin=1000000&end=2000000&summary=1&ts_normalize=0&pid=1"

GET /api/v1/viz/density

Like /api/v1/viz/events, but instead of dropping sub-pixel events it buckets them per (pid, tid, pixel-column) into aggregated density blocks, so zoomed-out views still show where activity is. Returns full-size events (with args, for the detail panel) plus a density array of blocks. Same begin/end/summary parameters as /api/v1/viz/events.

curl "http://localhost:8080/api/v1/viz/density?begin=0&end=999999999&summary=2"

{
  "events": [],
  "density": [
    {"pid": 100, "tid": 100, "ts": 0, "dur": 24457,
     "count": 910, "total": 22044, "depth": 0}
  ]
}

GET /api/v1/viz/counters

Per-bucket read/write bytes and I/O operation counts over a time range, for the bandwidth / IOPS tracks. Parameters begin, end, summary; returns a buckets array. Aggregated server-side in parallel.

curl "http://localhost:8080/api/v1/viz/counters?begin=0&end=999999999&summary=1"

{"buckets": [{"ts": 0, "read_bytes": 4096, "write_bytes": 0,
              "read_ops": 1, "write_ops": 0}]}

GET /api/v1/viz/stats

Server-side per-name aggregation over a time range (the Analyze panel). Returns a names table with count, total, avg, min, max per operation name. Parameters begin, end, summary; whole-trace, unfiltered queries are answered from a prebuilt summary.

curl "http://localhost:8080/api/v1/viz/stats?begin=0&end=999999999&summary=1"

{
  "count": 100, "total_dur": 5000,
  "names": [{"name": "read", "count": 50, "total": 2500,
             "avg": 50, "min": 10, "max": 90}]
}

GET /api/v1/viz/histogram

Distribution of event durations matching the query in [begin, end]: exact percentiles (p50, p99, …) plus a log-spaced buckets histogram of the shape. Narrow to one operation by folding name == "..." into the query.

curl "http://localhost:8080/api/v1/viz/histogram?begin=0&end=999999999&summary=1"

{"min": 10, "max": 900, "p50": 150, "p99": 880, "buckets": []}

GET /api/v1/viz/proctree

Infers the process fork hierarchy and returns a nodes array (one per process) with parent links, spawn/first timestamps, resolved host, rank (from PR metadata), and per-process bytes / io_ops / io_busy. Drives lane ordering and grouping in the viewer. Respects ?file= for per-node trees on multi-node traces.

curl "http://localhost:8080/api/v1/viz/proctree"

{"nodes": [{"pid": 100, "parent": -1, "host": "node01", "rank": "0",
            "bytes": 16384, "io_ops": 4, "io_busy": 600.0}]}

GET /api/v1/viz/calltree

Merges events into a flamegraph tree from ts/dur containment; identical name-paths fold together. Each node carries inclusive total, exclusive self, count, and children. Parameters begin, end, summary; group=pid keeps each process’s tree separate.

curl "http://localhost:8080/api/v1/viz/calltree?begin=0&end=999999999&summary=1"

{
  "name": "root", "total": 5000, "self": 0, "count": 0,
  "children": [{"name": "read", "total": 2500, "self": 2500, "count": 50}]
}

GET /api/v1/viz/layers

Whole-trace reference data: the operation-name to category layers map (a property of the name, so fetched once), plus total_files (declared via FH metadata) and io_files (those an I/O event actually touched).

curl "http://localhost:8080/api/v1/viz/layers"

{"layers": {"read": "POSIX", "write": "POSIX"},
 "total_files": 2, "io_files": 2}

GET / , /api, and /api/openapi.json

GET / (and /index.html) serve the embedded Trace Viewer (Web UI) web UI. GET /api serves the interactive API explorer page, and GET /api/openapi.json returns the OpenAPI 3.1 specification for every endpoint above (generated from the server’s own route table), so external tools can consume it too. None of these pages contain trace data; they query the API.

Event Filtering

The trace data endpoints (/api/v1/events, /api/v1/events/stream) use structured query parameters for filtering:

Query Parameter Filters:

  • cat - Filter by category (comma-separated for multiple: cat=POSIX,STDIO)

  • name - Filter by event name (comma-separated for multiple: name=read,write)

  • pid - Filter by process ID

  • ts_min / ts_max - Filter by timestamp range (microseconds)

  • dur_min / dur_max - Filter by duration range (microseconds)

# Get POSIX read/write events with duration > 1000 us
curl "http://localhost:8080/api/v1/events?file=trace.pfw.gz&cat=POSIX&name=read,write&dur_min=1000"

# Get events in a time window
curl "http://localhost:8080/api/v1/events?file=trace.pfw.gz&ts_min=1000000&ts_max=2000000"

Visualization Filters:

The /api/v1/viz/events endpoint supports JSON-based filters for complex predicates:

[
    {"field": "pid", "op": "=", "value": 1},
    {"field": "dur", "op": ">=", "value": 500}
]

Supported operators: =, >=, <=, >, <

Indexing

Bloom Filters:

Trace files larger than 1 MB (compressed) are automatically indexed with bloom filters (.idx files) during server startup. Bloom filters accelerate event filtering by skipping chunks that cannot contain matching events.

Checkpoint Indexes:

Checkpoint indexes (.idx files) store byte offsets and decompression state, enabling efficient random access to events by line number or byte position.

Small Files:

Files smaller than 1 MB are streamed directly without sidecar indexes. The server detects this automatically based on compressed file size.

Index Persistence:

Sidecar index files are stored in the trace directory (or --index-dir if specified) and persist across server restarts. Rebuilding indexes on subsequent starts is avoided, improving startup time.

Error Handling

The server returns standard HTTP status codes:

  • 200 OK - Request succeeded

  • 400 Bad Request - Invalid query parameter or filter syntax

  • 404 Not Found - Requested file or endpoint does not exist

  • 500 Internal Server Error - Unexpected server error (check logs)

Error responses include a JSON error message:

{
    "error": "File not found: trace.pfw.gz"
}

Performance Considerations

Concurrency:

The server uses coroutine-based concurrency to handle multiple simultaneous requests efficiently. Adjust --executor-threads to match your CPU core count for best throughput.

Memory:

Event filtering streams through bloom indexes and partial reads, minimizing memory usage. Both /api/v1/events and /api/v1/events/stream use chunked transfer encoding with iovec scatter-gather I/O, streaming NDJSON results without buffering the full response in memory.

Client Receive Timeouts:

The streaming endpoints (/api/v1/events, /api/v1/events/stream, /api/v1/viz/events) use HTTP/1.1 chunked transfer encoding and can hold a connection open while the server is still scanning chunks before any bytes are emitted. Clients should set a receive timeout of at least 15 seconds (the timeout used by the bundled integration tests, raised from 2 s in earlier builds) to accommodate the worst-case index-warmup path; the server itself does not impose a global request timeout (with_global_timeout(0)).

Query Optimization:

  • Use narrow time ranges in /api/v1/viz/events queries

  • Apply filters (cat, name, pid, time/duration ranges) to reduce the number of events scanned

  • Use limit for pagination on /api/v1/events

  • Use higher summary levels in visualization queries to aggregate short-duration events

  • Consider lanes filtering for visualization queries to reduce network overhead