Migration from DLIO Profiler to DFTracer

This section provides information to DLIO Profiler users on how to migrate their work from DLIO profiler to DFTracer.

Installation

Application building migration

To migrate your Makefile projects from using DLIO Profiler to DFTracer, you will need to update your compilation flags, specifically CFLAGS or CXXFLAGS, and LDFLAGS. Replace the DLIO Profiler flags with DFTracer flags as shown below:

Modifying Makefile to use DFTracer
 1# DLIO Profiler Flags (old)
 2DLIO_CFLAGS = -I/usr/workspace/iopp/kogiou1/venvs/pegasus-env/lib/python3.9/site-packages/dlio_profiler/include
 3DLIO_LDFLAGS = -L/usr/workspace/iopp/kogiou1/venvs/pegasus-env/lib/python3.9/site-packages/dlio_profiler/lib64 -ldlio_profiler
 4CFLAGS += $(DLIO_CFLAGS)
 5LIBS += $(DLIO_LDFLAGS)
 6
 7# Replace with DFTracer Flags (new)
 8# Add DFTracer include and library paths
 9DFTRACER_CFLAGS = -I/path/to/dftracer/include
10DFTRACER_LDFLAGS = -L/path/to/dftracer/lib64 -ldftracer
11
12# Append to existing CFLAGS and LDFLAGS
13CFLAGS += $(DFTRACER_CFLAGS)
14LDFLAGS += $(DFTRACER_LDFLAGS)

C++ API Changes

This section guides you through the necessary changes to migrate your application-level tracing for C++ projects from DLIO Profiler to DFTracer. The transition requires updating API names and includes directives to use the DFTracer’s new API. Please see examples.rst for more information on how to use DFTracer APIs.

Updating Includes

Replace the old DLIO Profiler include header with the new DFTracer header. This change points your application to the new tracing API.

1  // Old include
2  #include <dlio_profiler/dlio_profiler.h>
3
4  // Replace with new include
5  #include <dftracer/dftracer.h>

Initializing

Initialization now uses the DFTracer API, which can seamlessly integrate into your existing codebase where DLIO Profiler was previously initialized.

1  // Old initialization
2  DLIO_PROFILER_CPP_INIT(log_file, data_dirs, process_id);
3
4  // Replace with new initialization
5  DFTRACER_CPP_INIT(log_file, data_dirs, process_id);

This will configure the DFTracer environment, setting up the log file, data directories, and process ID exactly like the DLIO Profiler did. To migrate these configurations from DLIO Profile to DFTracer please replace your old enviromental variable configurations as shown bellow.

Finalizing

The finalization process ensures that all tracing data are correctly finalized and saved. Replace the DLIO Profiler finalization call with the DFTracer finalization.

1  // Old finalization
2  DLIO_PROFILER_CPP_FINI();
3
4  // Replace with new finalization
5  DFTRACER_CPP_FINI();

This function call is crucial for ensuring that your profiling data is not corrupted and is properly written to the log file.

Function and Region Profiling

For function and code block profiling, replace the old DLIO Profiler functions with their DFTracer counterparts.

1  // Old function and region profiling
2  DLIO_PROFILER_CPP_FUNCTION();
3  DLIO_PROFILER_CPP_REGION_<START/END>(CUSTOM);
4
5  // Replace with new function and region profiling
6  DFTRACER_CPP_FUNCTION();
7  DFTRACER_CPP_REGION_<START/END>(CUSTOM);

C API Changes

This section guides you through the necessary changes to migrate your application-level tracing for C projects from DLIO Profiler to DFTracer. The transition requires updating API names and includes directives to use the DFTracer’s new API. Please see examples.rst for more information on how to use DFTracer APIs.

Updating Includes

To transition your C projects to DFTracer, begin by updating the include directive to point to the new DFTracer API.

1 // Old include
2 #include <dlio_profiler/dlio_profiler.h>
3
4 // Replace with new include
5 #include <dftracer/dftracer.h>

Initializing

For C applications, DFTracer initialization replaces the older DLIO Profiler calls.

1 // Old initialization
2 DLIO_PROFILER_C_INIT(log_file, data_dirs, process_id);
3
4 // Replace with new initialization
5 DFTRACER_C_INIT(log_file, data_dirs, process_id);

This command configures DFTracer with the necessary parameters for logging and directory monitoring, similarly to how DLIO Profiler was configured. To migrate these configurations from DLIO Profile to DFTracer please replace your old enviromental variable configurations as shown bellow.

1 # Old environment variable configurations for DLIO Profiler
2 DLIO_LOG_FILE=~/dlio_log
3 DLIO_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset
4 export DLIO_INIT=PRELOAD
5 export DLIO_ENABLE=1

1 # Updated environment variable configurations for DFTracer
2 DFTRACER_LOG_FILE=~/log_file  # Changes the log file path variable name
3 DFTRACER_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset  # Consistent data directory path
4 export DFTRACER_INIT=PRELOAD  # Standardizing to PRELOAD mode
5 export DFTRACER_ENABLE=1  # Enabling the profiler

Finalizing

Finalize the DFTracer setup to ensure all tracing data are correctly captured and saved.

1 // Old finalization
2 DLIO_PROFILER_C_FINI();
3
4 // Replace with new finalization
5 DFTRACER_C_FINI();

Function and Region Profiling

Transition function and region profiling in your C code to use DFTracer’s updated API methods.

1 // Old function and region profiling
2 DLIO_PROFILER_C_FUNCTION_START();
3 DLIO_PROFILER_C_FUNCTION_END();
4
5 // Replace with new function and region profiling
6 DFTRACER_C_FUNCTION_START();
7 DFTRACER_C_FUNCTION_END();

Python API changes

Application building migration

To migrate your Makefile projects from using DLIO Profiler to DFTracer, you will need to update your compilation flags, specifically CFLAGS or CXXFLAGS, and LDFLAGS. Replace the DLIO Profiler flags with DFTracer flags as shown below:

Modifying Makefile to use DFTracer
 1# DLIO Profiler Flags (old)
 2DLIO_CFLAGS = -I/usr/workspace/iopp/kogiou1/venvs/pegasus-env/lib/python3.9/site-packages/dlio_profiler/include
 3DLIO_LDFLAGS = -L/usr/workspace/iopp/kogiou1/venvs/pegasus-env/lib/python3.9/site-packages/dlio_profiler/lib64 -ldlio_profiler
 4CFLAGS += $(DLIO_CFLAGS)
 5LIBS += $(DLIO_LDFLAGS)
 6
 7# Replace with DFTracer Flags (new)
 8# Add DFTracer include and library paths
 9DFTRACER_CFLAGS = -I/path/to/dftracer/include
10DFTRACER_LDFLAGS = -L/path/to/dftracer/lib64 -ldftracer
11
12# Append to existing CFLAGS and LDFLAGS
13CFLAGS += $(DFTRACER_CFLAGS)
14LDFLAGS += $(DFTRACER_LDFLAGS)

C++ API Changes

This section guides you through the necessary changes to migrate your application-level tracing for C++ projects from DLIO Profiler to DFTracer. The transition requires updating API names and includes directives to use the DFTracer’s new API. Please see examples.rst for more information on how to use DFTracer APIs.

Updating Includes

Replace the old DLIO Profiler include header with the new DFTracer header. This change points your application to the new tracing API.

1  // Old include
2  #include <dlio_profiler/dlio_profiler.h>
3
4  // Replace with new include
5  #include <dftracer/dftracer.h>

Initializing

Initialization now uses the DFTracer API, which can seamlessly integrate into your existing codebase where DLIO Profiler was previously initialized.

1  // Old initialization
2  DLIO_PROFILER_CPP_INIT(log_file, data_dirs, process_id);
3
4  // Replace with new initialization
5  DFTRACER_CPP_INIT(log_file, data_dirs, process_id);

This will configure the DFTracer environment, setting up the log file, data directories, and process ID exactly like the DLIO Profiler did. To migrate these configurations from DLIO Profile to DFTracer please replace your old enviromental variable configurations as shown bellow.

1 # Old environment variable configurations for DLIO Profiler
2 DLIO_LOG_FILE=~/dlio_log
3 DLIO_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset
4 export DLIO_INIT=PRELOAD
5 export DLIO_ENABLE=1

1 # Updated environment variable configurations for DFTracer
2 DFTRACER_LOG_FILE=~/log_file  # Changes the log file path variable name
3 DFTRACER_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset  # Consistent data directory path
4 export DFTRACER_INIT=PRELOAD  # Standardizing to PRELOAD mode
5 export DFTRACER_ENABLE=1  # Enabling the profiler

Finalizing

The finalization process ensures that all tracing data are correctly finalized and saved. Replace the DLIO Profiler finalization call with the DFTracer finalization.

1  // Old finalization
2  DLIO_PROFILER_CPP_FINI();
3
4  // Replace with new finalization
5  DFTRACER_CPP_FINI();

This function call is crucial for ensuring that your profiling data is not corrupted and is properly written to the log file.

Function and Region Profiling

For function and code block profiling, replace the old DLIO Profiler functions with their DFTracer counterparts.

1  // Old function and region profiling
2  DLIO_PROFILER_CPP_FUNCTION();
3  DLIO_PROFILER_CPP_REGION_<START/END>(CUSTOM);
4
5  // Replace with new function and region profiling
6  DFTRACER_CPP_FUNCTION();
7  DFTRACER_CPP_REGION_<START/END>(CUSTOM);

C API Changes

This section guides you through the necessary changes to migrate your application-level tracing for C projects from DLIO Profiler to DFTracer. The transition requires updating API names and includes directives to use the DFTracer’s new API. Please see examples.rst for more information on how to use DFTracer APIs.

Updating Includes

To transition your C projects to DFTracer, begin by updating the include directive to point to the new DFTracer API.

1 // Old include
2 #include <dlio_profiler/dlio_profiler.h>
3
4 // Replace with new include
5 #include <dftracer/dftracer.h>

Initializing

For C applications, DFTracer initialization replaces the older DLIO Profiler calls.

1 // Old initialization
2 DLIO_PROFILER_C_INIT(log_file, data_dirs, process_id);
3
4 // Replace with new initialization
5 DFTRACER_C_INIT(log_file, data_dirs, process_id);

This command configures DFTracer with the necessary parameters for logging and directory monitoring, similarly to how DLIO Profiler was configured. To migrate these configurations from DLIO Profile to DFTracer please replace your old enviromental variable configurations as shown bellow.

1 # Old environment variable configurations for DLIO Profiler
2 DLIO_LOG_FILE=~/dlio_log
3 DLIO_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset
4 export DLIO_INIT=PRELOAD
5 export DLIO_ENABLE=1

1 # Updated environment variable configurations for DFTracer
2 DFTRACER_LOG_FILE=~/log_file  # Changes the log file path variable name
3 DFTRACER_DATA_DIR=/dev/shm/:/p/gpfs1/$USER/dataset  # Consistent data directory path
4 export DFTRACER_INIT=PRELOAD  # Standardizing to PRELOAD mode
5 export DFTRACER_ENABLE=1  # Enabling the profiler

Finalizing

Finalize the DFTracer setup to ensure all tracing data are correctly captured and saved.

1 // Old finalization
2 DLIO_PROFILER_C_FINI();
3
4 // Replace with new finalization
5 DFTRACER_C_FINI();

Function and Region Profiling

Transition function and region profiling in your C code to use DFTracer’s updated API methods.

1 // Old function and region profiling
2 DLIO_PROFILER_C_FUNCTION_START();
3 DLIO_PROFILER_C_FUNCTION_END();
4
5 // Replace with new function and region profiling
6 DFTRACER_C_FUNCTION_START();
7 DFTRACER_C_FUNCTION_END();

Python API changes

Analyzer Changes

Migration of the DLP Analyzer jupyter notebook to DFAnalyzer involves configuring the YAML for Dask and renaming the imports and function calls in jupyter notebook cells.

  1. cd to dftracer/dfanalyzer/dask/conf and run install_dask_env.sh to create configuration.yaml in ~/.dftracer.

  2. update the app and environment path in configuration.yaml.

  1. update app_root variable by updating path of new configuration.yaml.

  2. replace dlp_analyzer with dfanalyzer and update the imports form dfanalyzer.main

1  ...
2  import dfanalyzer
3  from dfanalyzer.main import DFAnalyzer,get_dft_configuration,update_dft_configuration,setup_logging,setup_dask_cluster, reset_dask_cluster, get_dft_configuration
4  ...

  1. update the dask_run_dir to use dfanalyzer instead of dlp_analyzer.

  2. rename update and get configuration functions by calling DFtracer equivalent functions.

1  ...
2  conf = update_dft_configuration(dask_scheduler=dask_scheduler, verbose=True,
3                             log_file=f"./dft_{os.getenv('USER')}.log", rebuild_index=False, time_approximate=False,
4                             host_pattern=r'lassen(\d+)', time_granularity=30e6, skip_hostname=True, conditions=condition_fn)
5  conf = get_dft_configuration()
6  ...