[OFFLOAD] Add support for more fine grained debug messages control #165416

adurang · 2025-10-28T15:14:40Z

This PR introduces new debug macros that allow a more fined control of which debug message to output and introduce C++ stream syntax for debug messages.

Changing existing messages (except a few that I changed for testing) will come in subsequent PRs.

I also think that we should make debug enabling OpenMP agnostic but, for now, I prioritized maintaing the current libomptarget behavior for now, and we might need more changes further down the line as we we decouple libomptarget.

adurang · 2025-10-28T15:14:58Z

@pbalcer for awareness

llvmbot · 2025-10-28T15:15:13Z

@llvm/pr-subscribers-offload

Author: Alex Duran (adurang)

Changes

This PR introduces new debug macros that allow a more fined control of which debug message to output and introduce C++ stream syntax for debug messages.

Changing existing messages (except a few that I changed for testing) will come in subsequent PRs.

I also think that we should make debug enabling OpenMP agnostic but, for now, I prioritized maintaing the current libomptarget behavior for now, and we might need more changes further down the line as we we decouple libomptarget.

Full diff: https://github.com/llvm/llvm-project/pull/165416.diff

4 Files Affected:

(modified) offload/include/Shared/Debug.h (+187)
(modified) offload/libomptarget/OffloadRTL.cpp (+1-1)
(modified) offload/libomptarget/PluginManager.cpp (+1-1)
(modified) offload/plugins-nextgen/host/src/rtl.cpp (+2)

diff --git a/offload/include/Shared/Debug.h b/offload/include/Shared/Debug.h index 7c3db8dbf119f..85e1c0e19445d 100644 --- a/offload/include/Shared/Debug.h +++ b/offload/include/Shared/Debug.h @@ -42,6 +42,8 @@ #include <mutex> #include <string> +#include "llvm/Support/circular_raw_ostream.h" + /// 32-Bit field data attributes controlling information presented to the user. enum OpenMPInfoType : uint32_t { // Print data arguments and attributes upon entering an OpenMP device kernel. @@ -198,4 +200,189 @@ inline uint32_t getDebugLevel() { } \ } while (false) +// New macros that will allow for more granular control over debugging output +// Each message can be classified by Component, Type and Level +// Component: The broad component of the offload runtime emitting the message. +// Type: A cross-component classification of messages +// Level: The verbosity level of the message +// +// The component is pulled from the TARGET_NAME macro, Type and Level can be +// defined for each debug message but by default they are "default" and "1" +// respectively. +// +// For liboffload and plugins, use OFFLOAD_DEBUG(...) +// For libomptarget, use OPENMP_DEBUG(...) +// Constructing messages should be done using C++ stream style syntax. +// +// Usage examples: +// OFFLOAD_DEBUG("type1", 2, "This is a level 2 message of type1"); +// OFFLOAD_DEBUG("Init", "This is a default level of the init type"); +// OPENMP_DEBUG("This is a level 1 message of the default type"); +// OFFLOAD_DEBUG("Init", 3, NumDevices << " were initialized\n"); +// OFFLOAD_DEBUG("Kernel", "Starting kernel " << KernelName << " on device " << +// DeviceId); +// +// Message output can be controlled by setting LIBOMPTARGET_DEBUG or +// LIBOFFLOAD_DEBUG environment variables. Their syntax is as follows: +// [integer]|all|<type1>[:<level1>][,<type2>[:<level2>],...] +// +// 0 : Disable all debug messages +// all : Enable all debug messages +// integer : Set the default level for all messages +// <type> : Enable only messages of the specified type and level (more than one +// can be specified). Components are also supported as +// types. +// <level> : Set the verbosity level for the specified type (default is 1) +// +// For very specific cases where more control is needed, use OFFLOAD_DEBUG_RAW +// or OFFLOAD_DEBUG_BASE. See below for details. + +namespace llvm::offload::debug { + +#ifdef OMPTARGET_DEBUG + +struct DebugFilter { + StringRef Type; + uint32_t Level; +}; + +struct DebugSettings { + bool Enabled = false; + uint32_t DefaultLevel = 1; + llvm::SmallVector<DebugFilter> Filters; +}; + +/// dbgs - Return a circular-buffered debug stream. +inline llvm::raw_ostream &dbgs() { + // Do one-time initialization in a thread-safe way. + static struct dbgstream { + llvm::circular_raw_ostream strm; + + dbgstream() : strm(llvm::errs(), "*** Debug Log Output ***\n", 0) {} + } thestrm; + + return thestrm.strm; +} + +inline DebugFilter parseDebugFilter(StringRef Filter) { + size_t Pos = Filter.find(':'); + if (Pos == StringRef::npos) + return {Filter, 1}; + + StringRef Type = Filter.slice(0, Pos); + uint32_t Level = 1; + if (Filter.slice(Pos + 1, Filter.size()).getAsInteger(10, Level)) + Level = 1; + + return {Type, Level}; +} + +inline DebugSettings &getDebugSettings() { + static DebugSettings Settings; + static std::once_flag Flag{}; + std::call_once(Flag, []() { + printf("Configuring debug settings\n"); + // Eventually, we probably should allow the upper layers to set + // debug settings directly according to their own env var or + // other methods. + // For now, mantain compatibility with existing libomptarget env var + // and add a liboffload independent one. + char *Env = getenv("LIBOMPTARGET_DEBUG"); + if (!Env) { + Env = getenv("LIBOFFLOAD_DEBUG"); + if (!Env) + return; + } + + StringRef EnvRef(Env); + if (EnvRef == "0") + return; + + Settings.Enabled = true; + if (EnvRef.equals_insensitive("all")) + return; + + if (!EnvRef.getAsInteger(10, Settings.DefaultLevel)) + return; + + Settings.DefaultLevel = 1; + + SmallVector<StringRef> DbgTypes; + EnvRef.split(DbgTypes, ',', -1, false); + + for (auto &DT : DbgTypes) + Settings.Filters.push_back(parseDebugFilter(DT)); + }); + + return Settings; +} + +inline bool isDebugEnabled() { return getDebugSettings().Enabled; } + +inline bool shouldPrintDebug(const char *Component, const char *Type, + uint32_t Level) { + const auto &Settings = getDebugSettings(); + if (!Settings.Enabled) + return false; + + if (Settings.Filters.empty()) + return Level <= Settings.DefaultLevel; + + for (const auto &DT : Settings.Filters) { + if (DT.Level < Level) + continue; + if (DT.Type.equals_insensitive(Type)) + return true; + if (DT.Type.equals_insensitive(Component)) + return true; + } + + return false; +} + +#define OFFLOAD_DEBUG_BASE(Component, Type, Level, ...) \ + do { \ + if (llvm::offload::debug::isDebugEnabled() && \ + llvm::offload::debug::shouldPrintDebug(Component, Type, Level)) \ + __VA_ARGS__; \ + } while (0) + +#define OFFLOAD_DEBUG_RAW(Type, Level, X) \ + OFFLOAD_DEBUG_BASE(GETNAME(TARGET_NAME), Type, Level, X) + +#define OFFLOAD_DEBUG_1(X) \ + OFFLOAD_DEBUG_BASE(GETNAME(TARGET_NAME), "default", 1, \ + llvm::offload::debug::dbgs() \ + << DEBUG_PREFIX << " --> " << X) + +#define OFFLOAD_DEBUG_2(Type, X) \ + OFFLOAD_DEBUG_BASE(GETNAME(TARGET_NAME), Type, 1, \ + llvm::offload::debug::dbgs() \ + << DEBUG_PREFIX << " --> " << X) + +#define OFFLOAD_DEBUG_3(Type, Level, X) \ + OFFLOAD_DEBUG_BASE(GETNAME(TARGET_NAME), Type, Level, \ + llvm::offload::debug::dbgs() \ + << DEBUG_PREFIX << " --> " << X) + +#define OFFLOAD_SELECT(Type, Level, X, NArgs, ...) OFFLOAD_DEBUG_##NArgs + +// To be used in liboffload and plugins +#define OFFLOAD_DEBUG(...) OFFLOAD_SELECT(__VA_ARGS__, 3, 2, 1)(__VA_ARGS__) + +// To be used in libomptarget only +#define OPENMP_DEBUG(...) OFFLOAD_DEBUG(__VA_ARGS__) + +#else + +// Don't print anything if debugging is disabled +#define OFFLOAD_DEBUG_BASE(Component, Type, Level, ...) +#define OFFLOAD_DEBUG_RAW(Type, Level, X) +#define OFFLOAD_DEBUG(...) +#define OPENMP_DEBUG(...) + +#endif + +} // namespace llvm::offload::debug + #endif // OMPTARGET_SHARED_DEBUG_H diff --git a/offload/libomptarget/OffloadRTL.cpp b/offload/libomptarget/OffloadRTL.cpp index 04bd21ec91a49..9f9ec5c8bfabe 100644 --- a/offload/libomptarget/OffloadRTL.cpp +++ b/offload/libomptarget/OffloadRTL.cpp @@ -35,7 +35,7 @@ void initRuntime() { RefCount++; if (RefCount == 1) { - DP("Init offload library!\n"); + OPENMP_DEBUG("Init offload library!\n"); #ifdef OMPT_SUPPORT // Initialize OMPT first llvm::omp::target::ompt::connectLibrary(); diff --git a/offload/libomptarget/PluginManager.cpp b/offload/libomptarget/PluginManager.cpp index c8d6b42114d0f..915fd8df80eb7 100644 --- a/offload/libomptarget/PluginManager.cpp +++ b/offload/libomptarget/PluginManager.cpp @@ -36,7 +36,7 @@ void PluginManager::init() { return; } - DP("Loading RTLs...\n"); + OPENMP_DEBUG("Init", "Loading RTLs\n"); // Attempt to create an instance of each supported plugin. #define PLUGIN_TARGET(Name) \ diff --git a/offload/plugins-nextgen/host/src/rtl.cpp b/offload/plugins-nextgen/host/src/rtl.cpp index eb4ecac9907a1..3baadfebc541c 100644 --- a/offload/plugins-nextgen/host/src/rtl.cpp +++ b/offload/plugins-nextgen/host/src/rtl.cpp @@ -451,6 +451,8 @@ struct GenELF64PluginTy final : public GenericPluginTy { if (auto Err = Plugin::check(ffi_init(), "failed to initialize libffi")) return std::move(Err); #endif + OFFLOAD_DEBUG("Init", 2, + "GenELF64 plugin detected" << NUM_DEVICES << " devices\n"); return NUM_DEVICES; }

saiislam

Seems like a good idea, but have some questions. Please see inline.

offload/include/Shared/Debug.h

github-actions · 2025-10-29T15:22:03Z

✅ With the latest revision this PR passed the C/C++ code formatter.

offload/include/Shared/Debug.h

offload/plugins-nextgen/host/src/rtl.cpp

Meinersbur · 2025-11-05T15:26:16Z

Have you considered to model this after https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/Support/DebugLog.h? That is,

LDBG() << "message";

instead of

LLVM_DEBUG(llvm::dbgs() << "message");

Besides being shorter, advantages include is that it works better with code-completion than in a macro argument, and you do not get "unused variable" warnings in non-assert builds. LDBG() expands to if (0) llvm::nulls()¹ in non-assert builds, so it will be optimized away.

Actually for (bool _c = false; _c; _c = false) ::llvm::nulls() to avoid compiler warnings. ↩

jhuber6 · 2025-11-05T15:27:54Z

Have you considered model this after https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/Support/DebugLog.h? That is,
LDBG() << "message";
instead of
LLVM_DEBUG(llvm::dbgs << "message"); 
Besides being shorter, advantages include is that it works better with code-completion than in a macro argument, and you do not get "unused variable" warnings in non-assert builds. LDBG() expands to if (0) llvm::nulls()1 in non-assert builds, so it will be optimized away.

Footnotes
1. Actually `for (bool _c = false; _c; _c = false) ::llvm::nulls()` to avoid compiler warnings. [↩](#user-content-fnref-1-0c3704e3ba10abed1be797a3542229c7) 

That'd probably work well, we could make it like LDBG(Tags, ...).

adurang · 2025-11-05T15:44:43Z

Have you considered to model this after https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/Support/DebugLog.h? That is,
LDBG() << "message";
instead of
LLVM_DEBUG(llvm::dbgs() << "message"); 
Besides being shorter, advantages include is that it works better with code-completion than in a macro argument, and you do not get "unused variable" warnings in non-assert builds. LDBG() expands to if (0) llvm::nulls()1 in non-assert builds, so it will be optimized away.

Footnotes

Actually for (bool _c = false; _c; _c = false) ::llvm::nulls() to avoid compiler warnings. ↩

I don't remember seeing LDBG when I looked at that file. Sounds like a good suggestion, thanks @Meinersbur. I'll give it a try.

adurang · 2025-11-11T18:42:17Z

I've adjusted the implementation to behave like @Meinersbur suggested. Additionally, I have extended it with support to be able to adjust the information from a single debug message depending on the debug level, which is something we discussed would be useful.

adurang · 2025-11-11T18:50:29Z

Some examples of how it would be used:

 ODBG("type1", 2) << "This is a level 2 message of type1"; ODBG("Init") << "This is a default level of the init type"; ODBG() << "This is a level 1 message of the default type"; ODBG("Init", 3) << NumDevices << " were initialized"; ODBG("Kernel") << "Launching " << KernelName << " on device " << DeviceId;

Also, the content can be adjusted midway to the actual level to print more detailed or completely different information.
ODBG_IF_LEVEL can be used to guards parts of a stream to a higher level providing additional detail. Using ODBG_RESET_LEVEL will reset the level back to the original level:

 ODBG("Mapping", 2) << "Function F" << ODBG_IF_LEVEL(3) << " with argument value=" << Arg << ODBG_IF_LEVEL(4) << " and address=" << &Arg << ODBG_RESET_LEVEL() << " called";

ODBG_ONLY_LEVEL can be used to print parts of a stream only at a specific level. This allows to adjust parts of a message to completely depending on the debug level:

 ODBG() << "Starting computation " << ODBG_ONLY_LEVEL(1) << "on a device" << ODBG_ONLY_LEVEL(2) << "and mapping data on device" << DeviceId; << ODBG_ONLY_LEVEL(3) << dumpDetailedMappingInfo(DeviceId);

Message output can be controlled by setting LIBOMPTARGET_DEBUG or LIBOFFLOAD_DEBUG environment variables with the following syntax:

[integer]|all|[:][,[:],...]

0 : Disable all debug messages
all : Enable all level 1 debug messages
integer : Set the default level for all messages
: Enable only messages of the specified type and leve (more than one can be specified). Components are also supported as types.
: Set the verbosity level for the specified type (default is 1)

Some examples usage examples:

LIBOFFLOAD_DEBUG=1 (Print all messages of level 1 or lower)
LIBOFFLOAD_DEBUG=5 (Print all messages of level 5 or lower)
LIBOFFLOAD_DEBUG=init (Print messages of type "init" of level 1 or lower)
LIBOFFLOAD_DEBUG=init:3,mapping:2 (Print messages of type "init" of level 3 or lower and messages of type "mapping" of level 2 or lower)
LIBOFFLOAD_DEBUG=omptarget:4, init (Print messages from component "omptarget" of level 4 or lower and messages of type "init" of level 1 or lower)

adurang · 2025-11-17T16:24:57Z

@jhuber6 @mjklemm ping

offload/libomptarget/PluginManager.cpp

offload/plugins-nextgen/host/src/rtl.cpp

offload/include/Shared/Debug.h

jhuber6 · 2025-11-19T23:18:54Z

offload/include/Shared/Debug.h

+ // and add a liboffload independent one.
+ char *Env = getenv("LIBOMPTARGET_DEBUG");
+ if (!Env) {
+ Env = getenv("LIBOFFLOAD_DEBUG");


So, LIBOMPTARGET overrides the value of LIBOFFLOAD? Do we want that?

Eventually we want to remove both from here, but for the time being my thought was to give preference to what is already in existence. But I don't have a strong opinion on this, do you prefer to change the order?

Guess it's not a big deal, but could definitely be simplified

offload/include/Shared/Debug.h

mjklemm

LGTM

jhuber6

Sure, let's iterate on this. What we need next is a list of acceptable kinds

…lvm#165416) This PR introduces new debug macros that allow a more fined control of which debug message to output and introduce C++ stream style for debug messages. Changing existing messages (except a few that I changed for testing) will come in subsequent PRs. I also think that we should make debug enabling OpenMP agnostic but, for now, I prioritized maintaing the current libomptarget behavior for now, and we might need more changes further down the line as we we decouple libomptarget.

adurang added 2 commits October 28, 2025 15:33

[OFFLOAD] Add support for more fine grained debug messages control

2965164

formatting

7bc7ce8

adurang requested review from hansangbae and jhuber6 October 28, 2025 15:14

llvmbot added the offload label Oct 28, 2025

saiislam reviewed Oct 29, 2025

View reviewed changes

offload/include/Shared/Debug.h Outdated Show resolved Hide resolved

offload/include/Shared/Debug.h Outdated Show resolved Hide resolved

offload/include/Shared/Debug.h Outdated Show resolved Hide resolved

add some env var examples

c9fc54a

jhuber6 reviewed Nov 4, 2025

View reviewed changes

offload/include/Shared/Debug.h Outdated Show resolved Hide resolved

offload/include/Shared/Debug.h Show resolved Hide resolved

offload/plugins-nextgen/host/src/rtl.cpp Outdated Show resolved Hide resolved

adurang added 3 commits November 4, 2025 21:31

remove forgotten printf

da323af

minor comment format fix

562c93e

nit

b13c116

New implementation based on LDBG from Support/DebugLog

948354b

jhuber6 reviewed Nov 19, 2025

View reviewed changes

offload/libomptarget/PluginManager.cpp Show resolved Hide resolved

offload/plugins-nextgen/host/src/rtl.cpp Outdated Show resolved Hide resolved

jhuber6 reviewed Nov 19, 2025

View reviewed changes

adurang added 2 commits November 20, 2025 06:16

minor refactoring, format

04e44a7

simplify description

a045ce8

mjklemm approved these changes Nov 20, 2025

View reviewed changes

jhuber6 approved these changes Nov 20, 2025

View reviewed changes

adurang merged commit 66ddc9b into llvm:main Nov 20, 2025
10 checks passed

adurang deleted the new_offload_debug branch November 20, 2025 17:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[OFFLOAD] Add support for more fine grained debug messages control #165416

[OFFLOAD] Add support for more fine grained debug messages control #165416

Uh oh!

adurang commented Oct 28, 2025

adurang commented Oct 28, 2025

llvmbot commented Oct 28, 2025

saiislam left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Meinersbur commented Nov 5, 2025 •

edited

Loading

jhuber6 commented Nov 5, 2025

Footnotes

adurang commented Nov 5, 2025

Footnotes

adurang commented Nov 11, 2025

adurang commented Nov 11, 2025 •

edited

Loading

adurang commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

jhuber6 Nov 19, 2025

adurang Nov 20, 2025 •

edited

Loading

jhuber6 Nov 20, 2025

Uh oh!

mjklemm left a comment

jhuber6 left a comment

Uh oh!

Labels

6 participants

[OFFLOAD] Add support for more fine grained debug messages control #165416

[OFFLOAD] Add support for more fine grained debug messages control #165416

Uh oh!

Conversation

adurang commented Oct 28, 2025

adurang commented Oct 28, 2025

llvmbot commented Oct 28, 2025

saiislam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Meinersbur commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

jhuber6 commented Nov 5, 2025

Footnotes

adurang commented Nov 5, 2025

Footnotes

adurang commented Nov 11, 2025

adurang commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

adurang commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

jhuber6 Nov 19, 2025

Choose a reason for hiding this comment

adurang Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

jhuber6 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

mjklemm left a comment

Choose a reason for hiding this comment

jhuber6 left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

6 participants

github-actions bot commented Oct 29, 2025 •

edited

Loading

Meinersbur commented Nov 5, 2025 •

edited

Loading

adurang commented Nov 11, 2025 •

edited

Loading

adurang Nov 20, 2025 •

edited

Loading