DFTracer: is a library for profiling I/O calls and application functions.

User Guide

• Overview
• Definitions
• Limitations
  • Spawning of processes
  • OS Compatibility
  • Write Buffering
• Build DFTracer
  • Create Python environment (Recommended)
  • Build DFTracer with pip (Recommended)
  • Build DFTracer with Spack
  • Build DFTracer with CMake
  • Build Variables
• Using Docker for Testing DFTracer
  • Overview
  • Using Pre-built Docker Images
  • Building from Source
  • Testing DFTracer in Docker
  • Advanced Usage
  • Multi-Platform Builds
  • VS Code Dev Container
  • Testing Workflows
  • Troubleshooting
  • Best Practices
  • Example Testing Session
  • Additional Resources
  • See Also
• Building applications with DFTracer Code Annotations
  • Building for Projects with Makefiles
• DFTracer APIs
  • Modes of DFTracer
  • Configurations of DFTracer
  • DFTracer C++ APIs
  • DFTracer C APIs
  • DFTracer C/C++ Function Profiling using GCC
  • DFTracer Python APIs
• Model tracing
  • DFTracer Python APIs
• DFTracer Format
  • Complete Events
  • Metadata Events
• AI/DL Logging

DFAnalyzer

• Build
  • From source (Recommended)
  • From pip
• Getting Started with DFAnalyzer
• Running DFAnalyzer with Dask Distributed
  • Getting Started
  • Starting a Dask Distributed Cluster
  • Use DFAnalyzer
  • Acessing the Dask Dashboard
  • Stopping Dask Distributed Workers
• Running Dask Distributed in a new system
  • Initializing Dask Configurations
  • Editing $HOME/.dftracer/configuration.yaml
  • Editing Dask Configurations depending on your System
  • Run DFAnalyzer
• ALCF Polaris
  • Logging in to Polaris Login Node
  • Initializing Dask Configurations
  • Changing $HOME/.dftracer/configuration.yaml
  • Changing polaris.yaml config
  • Executing scheduler
  • Forwarding the Port
  • Opening Notebook File
  • Stopping Dask Distributed Workers
  • Tips and Tricks
• Overlap Analysis
  • Overlap vs Non-overlap Analysis
  • Annotating using DFTracer
  • Analysis using DFAnalyzer
• Custom Columns Derived from DFTracer
  • Example: Custom Workflow Columns

Utilities

• DFTracer Utility scripts
  • Additional Utilities
• DFTracer Service
  • Overview
  • Single-node quick start
  • Multi-node notes
  • Run on multiple nodes with mpirun
  • Run on multiple nodes with flux run
  • Run on multiple nodes with srun
  • Troubleshooting
• Bash Utility scripts
  • Handling .pfw files
  • Handling .pfw.gz files
  • Extracting JSON data
  • Querying AI DFTracer
• Perfetto UI
  • Loading traces
  • Using SQL Query

Debugging

• Preload Mode
• App Mode
  • Python:

Reference

• C++ Example
  • Application Level Example:
  • LD_PRELOAD Example:
  • Hybrid Example:
• C Example
  • Application Level Example:
  • LD_PRELOAD Example:
  • Hybrid Example:
• Python Example
• Integrated Applications
• Example Chrome Tracing Plots
• Migration from DLIO Profiler to DFTracer
  • Installation
  • Application building migration
  • C++ API Changes
  • C API Changes
  • Python API changes
  • Application building migration
  • C++ API Changes
  • C API Changes
  • Python API changes
  • Analyzer Changes

Applications

• Pegasus Montage with DFTracer
  • Step 1: Install Condor
  • Step 2: Install Pegasus
  • Step 3: Install and compile Montage
  • Step 4: Get the montage-pegasus-v3
  • Step 5: Compile the pegasus-mpi-cluster from source
  • Step 6: Create a single “install” directory for all Pegasus software
  • Step 7: Prepare software for Pegasus-Montage
  • Step 8: Annotate Montage
• Pegasus 1000-Genome with DFTracer
  • Step 1: Activate Environment
  • Step 2: Get the 1000genome-workflow
  • Step 3: Prepare software for Pegasus-1000Genome
  • Step 4: Annotate 1000-Genome

Contributing

• Developer Guide
  • Publishing package
  • ALCF Polaris
• Style Guides
  • Coding Conventions
  • Commit Message Format
• Testing Guide
  • Unit Tests

Links:

• GitHub
• Python Binding Documentation
• Python Binding GitHub
• Utilities Documentation
• Utilities GitHub

Indices and tables

• Index

• Module Index

• Search Page