CANN/driver IBV扩展库开发指南

最新推荐文章于 2026-06-05 07:40:58 发布
原创 最新推荐文章于 2026-06-05 07:40:58 发布 · 310 阅读 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

Table of Contents

【免费下载链接】driver 本项目是CANN提供的驱动模块，实现基础驱动和资源管理及调度等功能，使能昇腾芯片。 【免费下载链接】driver 项目地址: https://gitcode.com/cann/driver

Introduction

Component Introduction

ibverbs_extend (hereinafter referred to as ibv_extend) component is an extension of standard RDMA verbs interfaces, specifically designed to support direct-drive RDMA communication in NDA (NPUDirect Async) scenarios.

The ibv_extend component achieves north-south decoupling through unified interfaces, as shown in the following figure:

  • North-bound shields underlying network card hardware differences, providing transparent access for HCOMM and other communication transport layers.
  • South-bound interfaces with 1825, Yunmai, and other network card extension drivers, enabling them to work without sensing upper-layer computing and memory resource differences.

ibv_extend Logic Diagram

The ibv_extend component provides the following core features:

  • Dynamic loading and management of extension drivers
  • Creation and destruction of extended QP/CQ/SRQ
  • Support for north-south custom hardware operations through callback function mechanism

Application Scenarios

This component mainly applies to the following scenarios:

  • Direct data transfer between NPU and RDMA network card.
  • Bypassing Host CPU, direct-drive control plane between NPU and RDMA network card.

Application Scenario Diagram

Installation

Method 1: Install through Source Compilation

  1. Obtain source code and enter the component source directory.

    Method 1: Use git clone to get source code.

    git clone https://link.gitcode.com/i/1c86bd106dbfbff5a0d23147c09f9fa2.git
cd driver/src/custom/nda/ibv_extend/

    Method 2: Web page download source code.

    Browse to driver repository, click "Download zip" button to get the latest source code. After uploading to the server, execute the following command.

    unzip driver-master.zip
cd driver-master/src/custom/nda/ibv_extend/

  2. Compilation dependencies.

    For openEuler/CentOS/RHEL:

    yum install gcc gcc-c++ make cmake libnl3-devel libudev-devel pkgconfig python3-devel

    For Ubuntu/Debian:

    apt install gcc g++ make cmake libnl-3-dev libnl-route-3-dev libudev-dev pkg-config python3-dev

  3. Compile and install.

    bash ./build.sh

    Table 1 Compilation Parameter Description

    Parameter

    Long Parameter

    Description

    Usage Example

    -t=TYPE

    --type=TYPE

    Compile type:

    • release: Default value, automatically downloads rdma-core.
    • debug: Debug type.

    Compile in debug mode.

    bash ./build.sh -t=debug

    -b=PATH

    --libibverbs-build-dir=PATH

    Specify pre-compiled rdma-core build directory. Mutually exclusive with -s parameter, cannot be used together.

    Use pre-compiled rdma-core build library.

    bash ./build.sh -b=/path/to/rdma-core/build

    -s=PATH

    --libibverbs-source-dir=PATH

    Specify rdma-core source directory. Mutually exclusive with -b parameter, cannot be used together.

    Use custom rdma-core source code.

    bash ./build.sh -s=/path/to/rdma-core

    -x=PATH

    --libboundscheck-build-dir=PATH

    Specify pre-compiled libboundscheck build directory. Mutually exclusive with -e parameter, cannot be used together.

    Use pre-compiled libboundscheck build library.

    bash ./build.sh -x=/path/to/libboundscheck

    -e=PATH

    --libboundscheck-source-dir=PATH

    Specify libboundscheck source directory. Mutually exclusive with -x parameter, cannot be used together.

    Use custom libboundscheck source code.

    bash ./build.sh -e=/path/to/libboundscheck

    -h

    --help

    Show help information.

    View help information.

    bash ./build.sh -h

  4. After compilation, check output files.

    output/
├── lib/
│    └── libibv_extend.so    # Shared library
└── include/
│    └── ibv_extend.h        # Header file

  5. Dynamically specify environment variables at runtime.

    export LD_LIBRARY_PATH=/path/to/ibv_extend/output/lib:$LD_LIBRARY_PATH

    Note:

    • /path/to/ is a variable, replace with actual path.
    • Users can also copy dynamic libraries under output/lib or header files under output/include to other custom paths or system paths for subsequent use.

  6. Compile security function library.

    Go to libboundscheck release page to get the latest source code package, for example download libboundscheck-v1.1.16.tar.gz.

    Compile libc_sec.so dynamic library.

    # Extract to current directory
tar zxvf libboundscheck-v1.1.16.tar.gz
# Enter current directory
cd libboundscheck-v1.1.16
# Compile
make -j
# After successful compilation, dynamic link library will be generated in lib directory, need to rename to libc_sec.so.
cp lib/libboundscheck.so lib/libc_sec.so

    Need to depend on libc_sec.so dynamic library during use.

    export LD_LIBRARY_PATH=/path/to/libboundscheck-v1.1.16/lib:$LD_LIBRARY_PATH

Method 2: Install through HDK Package

  1. Install HDK software package supporting NDA feature.

    1. Obtain HDK software package supporting NDA feature.
    2. Refer to the corresponding hardware product NPU Driver and Firmware Installation Guide to complete HDK driver package installation.

  2. After installation, check output files.

    1. libibv_extend.so is in /usr/local/Ascend/driver/lib64/driver/ directory.
    2. ibv_extend.h is in /usr/local/Ascend/driver/include/ directory.

  3. Need to load environment variables at runtime.

    • If cann-toolkit and other development tool packages are already installed, execute the following command to load environment variables (the following command uses root user installation as example, please use actual CANN installation path).

      source /usr/local/Ascend/cann/set_env.sh

    • If cann-toolkit and other development tool packages are not installed, execute the following command to load environment variables.

      export LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64/common:$LD_LIBRARY_PATH

Usage

The ibv_extend component allows upper-layer communication libraries to dynamically load extension drivers through configuring environment variables or configuration files.

Before running, you can configure environment variable export IBV_EXTEND_SHOW_WARNINGS=1 to let ibv_extend component print Warning level logs to console. Warning level logs are not printed by default.

Before running, ensure that libibverbs.so and libibv_extend.so are correctly installed. For specific operations, refer to the table below.

Table 1 Pre-run Check

System TypeCheck/Install libibverbs.soCheck/Install libibv_extend.so
openEuler/CentOS/RHEL
  1. Check if libibverbs.so is installed.
    ldconfig -p | grep libibverbs
    If there is output, libibverbs.so is installed, done.
    If no output, continue.
  2. Install libibverbs.
    sudo yum install rdma-core
  1. Check if libibv_extend.so is installed.
    ll /path/to/ibv_extend/output/lib
    ll /usr/local/Ascend/driver/lib64/driver/ | grep libibv_extend.so
    If either has output, libibv_extend is installed, done.
    If no output, continue.
  2. Install libibv_extend, refer to Installation.
Ubuntu/Debian
  1. Check if libibverbs.so is installed.
    ldconfig -p | grep libibverbs
    If there is output, libibverbs.so is installed, done.
    If no output, continue.
  2. Install libibverbs.
    sudo apt update
    sudo apt install libibverbs1 ibverbs-utils rdma-core

Configure Environment Variables

Upper-layer applications can configure IBV_EXTEND_DRIVERS environment variable before running to let ibv_extend component find and load specified driver extension libraries (such as 1825, Yunmai, and other network card extension drivers). Multiple drivers are separated by : or ;.

Example: export IBV_EXTEND_DRIVERS="/path1/my_ext_driver1.so:/path2/my_ext_driver2.so"

Configuration File

  1. Create extension driver configuration file directory.

    mkdir -p /etc/libibverbs_extend.d/

  2. Enter extension driver configuration file directory.

    cd /etc/libibverbs_extend.d/

  3. Create extension driver configuration file. Configuration file content format: driver <driver_name>.

    Assume driver extension library is libmy_ext_driver.so, use the following command to generate configuration file.

    echo "driver my_ext_driver" > my_ext_driver.conf

API Reference

Common Interfaces

ibv_extend_get_version

Function Prototype

const char *ibv_extend_get_version(uint32_t *major, uint32_t *minor, uint32_t *patch)

Function

Get ibv_extend dynamic library version number.

Parameters

  • major: Major version number. Output parameter. Can be null, null means not getting, interface does not allocate additional memory.
  • minor: Minor version number. Output parameter. Can be null, null means not getting, interface does not allocate additional memory.
  • patch: Patch version number. Output parameter. Can be null, null means not getting, interface does not allocate additional memory.

Return Value

Version string pointer (static string, no need to release).

Usage Example

// Call function to get version information
version_str = ibv_extend_get_version(&major, &minor, &patch);
// Print version information
printf("Version string: %s\n", version_str);
printf("Version number: %u.%u.%u\n", major, minor, patch);

ibv_extend_check_version

Function Prototype

int ibv_extend_check_version(uint32_t driver_major, uint32_t driver_minor, uint32_t driver_patch)

Function

Check version compatibility. This function checks at runtime whether the header file version used during driver and upper-layer application compilation is compatible with current library version. We recommend calling this function for version check during driver and upper-layer application initialization.

Compatibility rules:

  1. Major version number completely matches, then compatible.
  2. Software minor version number <= library minor version number, then compatible.
  3. If minor version numbers are the same, and software patch version number <= library patch version number, then compatible.

Parameters

  • driver_major: Major version number, recommend passing IBV_EXTEND_VERSION_MAJOR.
  • driver_minor: Minor version number, recommend passing IBV_EXTEND_VERSION_MINOR.
  • driver_patch: Patch version number, recommend passing IBV_EXTEND_VERSION_PATCH.

Return Value

  • 0: Compatible.
  • -1: Incompatible.

Usage Example

// Call function to check version compatibility
ret = ibv_extend_check_version(IBV_EXTEND_VERSION_MAJOR, IBV_EXTEND_VERSION_MINOR, IBV_EXTEND_VERSION_PATCH);
// Check return value
if (ret == 0) {
    printf("Version compatible, can use normally\n");
    // Continue subsequent initialization operations
} else {
    printf("Version incompatible, driver compilation version does not match current library version\n");
    // Refuse to run
    return -1;
}

North-bound Interfaces (Application Layer Call)

ibv_open_extend

Function Prototype

struct ibv_context_extend *ibv_open_extend(struct ibv_context *context)

Function

Call driver registered interface to create extension context. Must call ibv_open_extend to create extension context before calling other interfaces.

Parameters

context: Standard ibv context, created using ibv_open_device, non-null.

Return Value

  • Extension context pointer: Extension context creation successful.
  • NULL: Extension context creation failed.

Usage Example

// Get device list
int num_devices = 0;
struct ibv_device **dev_list = ibv_get_device_list(&num_devices);
if (!dev_list) {
    fprintf(stderr, "Failed to get device list\n");
    return -1;
}

// Check if there are available devices
if (num_devices <= 0) {
    fprintf(stderr, "No device found\n");
    ibv_free_device_list(dev_list);
    return -1;
}

// Select first device
struct ibv_device *device = dev_list[0];

// Open device
struct ibv_context *ctx = ibv_open_device(device);
if (!ctx) {
    fprintf(stderr, "Failed to open device\n");
    ibv_free_device_list(dev_list);
    return -1;
}

// Open extension context
struct ibv_context_extend *ext_ctx = ibv_open_extend(ctx);
if (!ext_ctx) {
    fprintf(stderr, "Failed to open extend context\n");
    ibv_close_device(ctx);
    ibv_free_device_list(dev_list);
    return -1;
}

ibv_close_extend

Function Prototype

int ibv_close_extend(struct ibv_context_extend *context)

Function

Call driver registered interface to release extension context resources. Cannot use this extension context resource after ibv_close_extend call.

Parameters

context: Extension context, must be created using ibv_open_extend, non-null.

Return Value

  • 0: Success.
  • EINVAL: Invalid parameter.
  • ENOENT: Matching driver not found.
  • Driver abnormal return value.

Usage Example

int ret = ibv_close_extend(ext_ctx);
if (ret != 0) {
    fprintf(stderr, "Failed to close extend context: %d\n", ret);
}

South-bound Interfaces (Driver Call)

verbs_register_driver_extend

Function Prototype

void verbsregister_driver_extend(const struct verbs_device_extend_ops *ops) or PROVIDER_EXTEND_DRIVER(drv)

Function

Register driver to extend_driver_list linked list. Can use encapsulated PROVIDER_EXTEND_DRIVER macro.

Parameters

  • ops: Device extension operation structure pointer, non-null, internally does not allocate additional address, must be global static variable.
  • drv: Must be struct verbs_device_extend_ops type.

Return Value

None.

Usage Example

static const struct verbs_device_extend_ops my_driver_ops = {
    .name = "my_driver",    // Must be consistent with standard driver
    .alloc_context = my_alloc_context,
    .free_context = my_free_context,
};
PROVIDER_EXTEND_DRIVER(my_driver_ops);  // Use macro for automatic registration

Key Structure Description

Enum Type Definition

queue_buf_dma_mode

During extended QP/CQ/SRQ resource creation, for these resources, driver needs to establish host address and device address mapping so that device can directly drive network card resources. Currently supports two mapping modes.

VariableValueDescription
QU_BUF_DMA_MODE_DEFAULT0PCIe mapping mode, this mode supports general hardware form, network card and NPU are both PCIe devices, usually use peerMem mode mapping.
QU_BUF_DMA_MODE_INDEP_UB1UB mapping mode, network card and NPU are PCIe devices, but data path uses independent UB bus addressing, usually use UbShmem mode mapping.

doorbell_map_mode

During extended QP/CQ/SRQ resource creation, driver will allocate doorbell address, and needs to map host-side allocated doorbell address to NPU card side. Mapping logic (db_mmap) is implemented by upper layer. Upper layer needs to perform mapping based on mapping type provided by driver.

VariableValueDescription
DB_MAP_MODE_HOST_VA0Based on Host virtual address mapping, usually third-party network card resource mapping method.
DB_MAP_MODE_UB_RES1Use UB device resource descriptor for mapping, 1825 network card used mapping method.

memcpy_direction

When driver uses upper-layer registered memcpy_s to copy memory, need to specify copy direction to allow driver read/write access to device memory.

VariableValueDescription
MEMCPY_DIR_HOST_TO_HOST0memcpy_s parameter, copy within Host.
MEMCPY_DIR_HOST_TO_DEVICE1memcpy_s parameter, memory copy from Host to Device (NPU).
MEMCPY_DIR_DEVICE_TO_HOST2memcpy_s parameter, memory copy from Device (NPU) to Host.
MEMCPY_DIR_DEVICE_TO_DEVICE3memcpy_s parameter, memory copy within Device (NPU) or between Device (NPU).

ibv_qp_init_cap

When creating extended QP, upper-layer communication library can specify additional extension configuration, optional.

VariableValueDescription
QP_ENABLE_DIRECT_WQE1If network card supports direct_wqe capability, by configuring this field, can enable network card direct_wqe capability when calling ibv_create_qp_extend interface to create sq_info.

ibv_extend_device_cap

Device supported extension capabilities, returned by extension query interface. Driver queries whether current device supports NDR/NDA.

VariableValueDescription
IBV_EXTEND_NDR1Device supports NDR (NPUDirect Rdma) capability.
IBV_EXTEND_NDA2Device supports NDA (NPUDirect Async) capability.

Core Data Structures

doorbell_map_desc

doorbell_map descriptor is used for driver-side mapping of hardware doorbell address. After driver allocates doorbell, maps it to NPU through db_map callback passed by upper layer for NPU direct-drive.

VariableTypeDescription
hvauint64_tdoorbell virtual address on host side, PCIe network card uses, address must be valid.
ub_resstructUB mapping method uses, uses UB resource descriptor, address must be valid. See structure description below.
sizeuint64_tdoorbell mapping length.
typeuint32_tdoorbell mapping type (doorbell_map_mode type).
resvuint32_tReserved.
struct {
    uint64_t guid_l;   /* GUID low 64 bits */
    uint64_t guid_h;   /* GUID high 64 bits */
    struct {
        uint64_t resource_id : 4; /* Corresponds to res_id in GUID */
        uint64_t offset : 32;  /* Offset within a page id of a certain GUID */
        uint64_t resv : 28;   /* Reserved */
    } bits;
} ub_res;

ibv_extend_ops

This series of functions is used for memory allocation and mapping during QP/CQ/SRQ and other resource creation. Upper-layer application implements, for underlying driver callback use.

Function PrototypeFunction DescriptionInput Parameter DescriptionReturn Value Description
void *(*alloc)(size_t size)Used by underlying driver to request memory resources, can be NPU memory resources or host memory resources.size: Allocation length, unit bytes, greater than 0.Allocated memory pointer, returns null on allocation failure.
void (*free)(void *ptr)Used by underlying driver for memory release. Memory must be address allocated by alloc callback function.ptr: Memory pointer allocated using alloc interface.No return value.
void (*memset_s)(void *dst, int value, size_t count)Used by underlying driver for memory initialization operation. Memory must be address allocated by alloc callback function.
  • dst: Memory pointer allocated using alloc interface, memset_s does not perform boundary check on memory area pointed by pointer ptr, so need to ensure memory area pointed by ptr is large enough when using, avoid out-of-bounds access.
  • value: Usually an int type value, but actually only uses low 8 bits of this value. This means values outside range 0 to 255 may produce undefined behavior.
  • count: Number of bytes to set, usually calculated through sizeof() or other means. count cannot be greater than size requested during alloc.
No return value.
int (*memcpy_s)(void *dst, size_t dst_max_size, void *src, size_t size, uint32_t direct)Used by underlying driver for memory copy operation. Memory must be address allocated by alloc callback function.
  • dst: Copy destination address, can be host-side allocated address or alloc allocated address.
  • dst_max_size: For copy safety, need to pass maximum length of dst destination address.
  • src: Copy source address, can be host-side allocated address or alloc allocated address.
  • size: Actual copy length in src, unit bytes, cannot exceed src allocated length.
  • direct: Copy direction, type is enum memcpy_direction.
Returns 0 on success, returns negative error code on failure.
void *(*db_mmap)(struct doorbell_map_desc *desc)Used for doorbell mapping logic, usually implements logic of mapping host doorbell address to NPU memory.
Implementation constraint: Must support repeated mapping of same desc, that is, multiple mappings of same desc address must return same mapping address.		desc: doorbell resource mapping descriptor.Mapped address pointer, returns null on failure.
int (*db_unmap)(void *ptr, struct doorbell_map_desc *desc)Used for unmapping logic.
  • ptr: Address mapped by calling db_mmap interface.
  • desc: Input desc during db_mmap.
Returns 0 on success, returns negative error code on failure.

FAQ

Error when calling ibv_open_extend interface to initialize extension library

Problem Description

Upper-layer communication component calls ibv_open_extend interface returns null pointer. After configuring IBV_EXTEND_SHOW_WARNINGS environment variable, error as follows:

ibv_extend: Warning: no available ops for open extend context

Possible Cause

ibv_extend extension library cannot find corresponding network card extension driver or dlopen corresponding network card extension driver reports error.

Solution

  1. Refer to Usage, let ibv_extend find corresponding network card extension driver location.

  2. If using dlopen method to load libibv_extend.so, need to export symbols to global symbol table through RTLD_GLOBAL. Usage example as follows.

    void *handle = dlopen("libibv_extend.so", RTLD_LAZY | RTLD_GLOBAL);
if (!handle) {
    fprintf(stderr, "dlopen libibv_extend.so error: %s\n", dlerror());
    return;
}

  3. If still reports error, contact corresponding network card extension driver vendor to confirm whether extension driver package is correct.

【免费下载链接】driver 本项目是CANN提供的驱动模块，实现基础驱动和资源管理及调度等功能，使能昇腾芯片。 【免费下载链接】driver 项目地址: https://gitcode.com/cann/driver

创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值