Video Pipeline

1. Overview

This section mainly describes the usage, API introduction, and related data structures of the Video Pipeline module. The Video Pipeline module provides a unified management interface for functions such as video encoding/decoding and image rotation, simplifying the complexity of multi-module collaborative work.

For Video Pipeline example projects, please refer to:
- Video Pipeline Example Project
For API reference about Video Pipeline, please refer to:
- Video Pipeline API
For H264 encoding image effect modification, please refer to:
- H264 Hardware Encoding Development Guide

2. Video Pipeline Basic Information

2.1 Supported Features

Supports H.264 video encoding function;

Supports image rotation function (hardware rotation and software rotation);

Supports multiple rotation angles (90 degrees, 180 degrees, 270 degrees, etc.);

Supports module status query;

Supports callback mechanism to handle encoding/decoding completion events;

Supports JPEG decoding integration;

Supports extended control interface.

Warning

This module is not called when using DVP camera, and is called by default when using UVC camera. It can also be used separately for decoding JPEG images to control output images in callbacks;
When hard decoding, the data given to the H264 module is the data before rotation. When soft decoding, the data given to the H264 module is the data after rotation;
In hard decoding, 90 degrees and 270 degrees support hardware rotation. If 180 degrees rotation is needed, software rotation must be used, which is slower than hardware rotation;

4) Rotation parameters need to be consistent; The rotation parameter of the rotation module is video_pipeline_config.decode_cfg.rotate_angle, The rotation parameter of the H264 module is video_pipeline_config.h264e_cfg.sw_rotate_angle, Otherwise, the screen display of hard decoding and soft decoding cameras will be inconsistent;

5) The data format after hardware rotation is RGB565. If other processing or display is needed, it needs to be configured as RGB565 format; The data format after software rotation is YUYV format; if other processing or display is needed, it needs to be configured as YUYV format;

2.3 Reference Project

The complete path of this example: ./projects/video_pipeline_example

This example mainly shows how to use the Video Pipeline module, including the use of H.264 encoding and image rotation functions and their combined use.

The main process of this example is as follows:

Create a Video Pipeline instance.

Configure and open the corresponding functional modules (H.264 encoder or rotation module, see parameter configuration in the usage examples below).

Perform corresponding operations (encoding or rotation).

Query module status (optional).

Close the corresponding functional modules.

Delete the Video Pipeline instance.

2.4 Memory Usage

The video pipeline module requires at most (DECODE_MAX_PIPELINE_LINE_SIZE + ROTATE_MAX_PIPELINE_LINE_SIZE) memory, used to store decoded images and rotated images. The specific size can be found in the macro configuration in mux_pipeline.h. By default, decoding an 864*480 image requires 0X1B000 memory;

Through the macro CONFIG_BT_REUSE_MEDIA_MEMORY, it can be configured whether Bluetooth memory can be used for processing when multimedia is running. If this macro is configured as Y, then Bluetooth needs to be turned off before calling bk_video_pipeline_open_h264e and bk_video_pipeline_open_rotate in the video pipeline module, otherwise an abnormal dump will occur;

If CONFIG_BT_REUSE_MEDIA_MEMORY is configured as N, the video pipeline module will automatically apply for memory when opening the corresponding module for processing, and will not release it afterward; This avoids memory fragmentation caused by repeated application and release, which may result in insufficient memory;

If the decoded image width exceeds 864, the IMAGE_MAX_WIDTH macro configuration needs to be changed;

3. Common Application Scenarios

Common scenarios are mainly UVC camera hardware decoding, UVC camera software decoding, and DVP camera decoding;

It is not recommended to use this module in DVP camera scenarios;

UVC camera hardware decoding and UVC camera software decoding internally select whether to use hardware decoding or software decoding based on image information after obtaining the first frame of image; The output formats of hardware decoding and software decoding are inconsistent, and attention should be paid when using them.

3.1 UVC Camera Hardware Decoding

3.1.1 Image Input

For UVC camera hardware decoding, the image input needs to be in YUV422 format, the image width needs to be an integer multiple of 32, and the height needs to be an integer multiple of 16;

3.1.2 Image Output

In UVC hardware decoding scenarios, it is recommended to configure hardware rotation. The output image format is RGB565. Hardware rotation does not support 180-degree rotation. If 180-degree rotation is needed, please configure software rotation with YUYV output format;

Warning

If YUYV format output is needed, please configure software rotation, which supports 0, 90, 180, and 270-degree rotation;
If RGB888 format output is needed, please disable the macro CONFIG_SW_ROTATE_TO_YUYV_AND_DMA2D_TO_YUYV_NOT_RGB888 (RGB888 output does not support 0-degree rotation)

3.1.3 H264 Encoding Image Output

The data after H264 encoding is output through the complete function in h264e_cbs callback, which contains the encoding result and a pointer of type frame_buffer_t. After successful encoding, the data can be processed further, such as storing to a file or sending over the network. When encoding fails, the frame_buffer_t type pointer needs to be released;

In the frame_buffer_t structure, the current image type can be obtained through frame_buffer_t.h264_type;

If frame_buffer_t.h264_type is H264_TYPE_I, it is an I-frame;
If frame_buffer_t.h264_type is H264_TYPE_P, it is a P-frame;
The I-frame output interval is configured through the macro CONFIG_H264_P_FRAME_CNT, with a default value of 29, meaning one I-frame is output every 30 frames;

For H264 encoding effect modification methods, see the document: H264 Hardware Encoding Development Guide

3.2 UVC Camera Software Decoding

3.2.1 Image Input

For UVC camera hardware decoding, the image input needs to be in YUV420 format, the image width needs to be an integer multiple of 2, and there is no limit on height;

The output format is YUYV. In YUYV format, two Y components share a set of UV, so the number of Y needs to be a multiple of 2;

3.2.2 Image Output

The UVC camera decoded data is in YUYV format, and the decoded data has already been rotated, so it does not need to go through the rotation module again;

3.2.3 H264 Encoding Image Output

The data after H264 encoding is output through the complete function in h264e_cbs callback, which contains the encoding result and a pointer of type frame_buffer_t. After successful encoding, the data can be processed further, such as storing to a file or sending over the network. When encoding fails, the frame_buffer_t type pointer needs to be released;

In the frame_buffer_t structure, the current image type can be obtained through frame_buffer_t.h264_type;

If frame_buffer_t.h264_type is H264_TYPE_I, it is an I-frame;
If frame_buffer_t.h264_type is H264_TYPE_P, it is a P-frame;
The I-frame output interval is configured through the macro CONFIG_H264_P_FRAME_CNT, with a default value of 29, meaning one I-frame is output every 30 frames;

For H264 encoding effect modification methods, see the document: H264 Hardware Encoding Development Guide

Warning

During soft decoding, the data given to the H264 module has been rotated, so the image rotation angle is inconsistent with hard decoding. Adjustments need to be made according to actual conditions;

3.3 DVP Camera

It is not recommended to use this module in DVP scenarios because the video pipeline module requires a complete JPEG image, while DVP can directly output YUV format data, which can be directly given to the H264 module and display module;

4. Module Parameter Configuration and Function Calls

The following describes the module parameter configuration and function calls in detail;

4.1 Module Initialization

4.1.1 Initialization Parameter Configuration

During initialization, jpeg_cbs and decode_cbs need to be passed in, and neither structure can be empty;

jpeg_cbs is used during decoding, and both hardware decoding and software decoding will use the malloc function in this structure to obtain a frame of image for decoding, and call the complete function in this structure to release the image after decoding;

decode_cbs is used during rotation and software decoding,

Before rotation, the malloc in the structure is used to apply for a frame of frame to store the decoded data. After rotation is completed, the complete function is used to release it. If the release fails, free is used directly to release it;
During software decoding, the malloc in the structure is first used to apply for a frame of frame for decoding, and after decoding is completed, the complete function is used to release it;

The complete function in decode_cbs returns when decoding is finished. If the decoding is successful, the frame is available for use (e.g., for display). However, if decoding fails, the frame should be freed directly.

// Global video pipeline handle
static bk_video_pipeline_handle_t g_video_pipeline_handle = NULL;

// JPEG callback function, called when decoding is completed, used to release the pre-decoded image frame. Custom processing logic can be added here.
static bk_err_t jpeg_complete(bk_err_t result, frame_buffer_t *out_frame)
{
    frame_buffer_encode_free(out_frame);
    return BK_OK;
}

// JPEG read function, internally obtains a frame of image for decoding before decoding. Needs to be modified according to actual conditions.
static frame_buffer_t *jpeg_read(uint32_t timeout_ms)
{
    // TODO: Get a frame of image data from JPEG cache. Needs to be modified according to actual conditions.
    frame_buffer_t *frame = frame_buffer_encode_malloc(jpeg_data_length);
    if (frame == NULL) {
        LOGE("jpeg_read malloc failed\r\n");
        return NULL;
    }
    // Copy JPEG data to frame_buffer_t structure
    os_memcpy(frame->frame, jpeg_data, jpeg_data_length);
    frame->width = 864;
    frame->height = 480;
    frame->fmt = PIXEL_FMT_JPEG;
    frame->length = jpeg_data_length;
    return frame;
}
// JPEG decoding configuration. Needs to be modified according to actual conditions.
static const jpeg_callback_t jpeg_cbs = {
    .read = jpeg_read,
    .complete = jpeg_complete,
};

// Display frame release function. Needs to be modified according to actual conditions.
static bk_err_t display_frame_free_cb(void *frame)
{
    frame_buffer_display_free((frame_buffer_t *)frame);
    return BK_OK;
}
// Decoding completion callback function. Called when decoding is completed. format_type is the format type of the completed decoding, result is the decoding result, out_frame is the decoded image frame.
// Custom processing logic can be added here, such as displaying images, saving images, etc.
static bk_err_t decode_complete(uint32_t format_type, uint32_t result, frame_buffer_t *out_frame)
{
    bk_err_t ret = BK_FAIL;
    if (result != BK_OK)
    {
        // Release the corresponding buffer directly when decoding fails
        display_frame_free_cb(out_frame);
        return BK_OK;
    }
    if (format_type == HW_DEC_END)
    {
        LOGI("Decode complete with result: %d\n", result);
        // TODO: Give to display module for display
        display_frame_free_cb(out_frame);
        return BK_OK;
    }
    else if (format_type == SW_DEC_END)
    {
        // TODO: Give to display module for display
        display_frame_free_cb(out_frame);
    }
    return ret;
}

// Frame application function after decoding. Needs to be modified according to actual conditions.
static frame_buffer_t *decode_malloc(uint32_t size)
{
    return frame_buffer_encode_malloc(size);
}
// Frame release function after decoding. Needs to be modified according to actual conditions.
static bk_err_t decode_free(frame_buffer_t *frame)
{
    frame_buffer_encode_free(frame);
    return BK_OK;
}
// Decoding callback structure. Needs to be modified according to actual conditions.
static const decode_callback_t decode_cbs = {
    .malloc = decode_malloc,
    .free = decode_free,
    .complete = decode_complete,
};

4.1.2 Initialization Function Configuration

bk_err_t ret = BK_FAIL;
bk_video_pipeline_config_t video_pipeline_config = {0};

if (g_video_pipeline_handle == NULL) {
    video_pipeline_config.jpeg_cbs = &jpeg_cbs; // Configure JPEG callback function, used to obtain a frame of JPEG image and release the image
    video_pipeline_config.decode_cbs = &decode_cbs; // Configure decoding callback function, used to apply for storing decoded images, release images, and notify upper layer of decoded image status
    // Create a new video pipeline instance
    ret = bk_video_pipeline_new(&g_video_pipeline_handle, &video_pipeline_config);
    if (ret != BK_OK) {
        LOGE("bk_video_pipeline_new failed, ret: %d\n", ret);
        return ret;
    }
}

4.2 Opening Rotation Function

4.2.1 Rotation Function Parameter Configuration

The main parameters of the rotation function are rotation mode and rotation angle,

Rotation modes include hardware rotation and software rotation;
Rotation angle uses the media_rotate_t enumeration type, configurable as ROTATE_NONE, ROTATE_90, ROTATE_180, ROTATE_270.

This parameter is not effective during soft decoding. During soft decoding, the rotation angle is configured in bk_video_pipeline_h264e_config_t.

Note

The rotation angle parameter must use the media_rotate_t enumeration type, not numeric values (such as 0, 90, 180, 270).

Unless there are special requirements, it is recommended to use hardware rotation because hardware rotation is fast and does not affect decoding performance.

Hardware rotation does not support 180-degree rotation. If 180-degree rotation is needed, please configure software rotation with YUYV output format;

Warning

If YUYV format output is needed, please configure software rotation, which supports 0, 90, 180, and 270-degree rotation;
If RGB888 format output is needed, please disable the macro CONFIG_SW_ROTATE_TO_YUYV_AND_DMA2D_TO_YUYV_NOT_RGB888 (RGB888 output does not support 0-degree rotation)

bk_video_pipeline_decode_config_t video_pipeline_decode_config = {0};
// Configure hardware rotation
video_pipeline_decode_config.rotate_mode = HW_ROTATE; // Hardware rotation mode
video_pipeline_decode_config.rotate_angle = ROTATE_90; // 90-degree rotation (using media_rotate_t enumeration type)

4.2.2 Rotation Function Configuration

// Open rotation function
ret = bk_video_pipeline_open_rotate(g_video_pipeline_handle, &video_pipeline_decode_config);
if (ret != BK_OK) {
    LOGE("bk_video_pipeline_open_rotate failed, ret: %d\n", ret);
    return ret;
}

4.3 Opening H.264 Encoding Function

4.3.1 H264 Encoding Function Parameter Configuration

Callback Functions

// H.264 encoding callback function
static frame_buffer_t *h264e_frame_malloc(uint32_t size)
{
    return frame_buffer_encode_malloc(size);
}

static void h264e_frame_complete(frame_buffer_t *frame, int result)
{
    if (result != AVDK_ERR_OK) {
        LOGI("H264 encode failed with result: %d\n", result);
        frame_buffer_encode_free(frame);
    } else {
        LOGI("H264 encode success, frame size: %d\n", frame->length);
        // TODO: Process encoded frame
        frame_buffer_encode_free(frame);
    }
}

static const bk_h264e_callback_t h264e_cbs = {
    .malloc = h264e_frame_malloc,
    .complete = h264e_frame_complete,
};

Parameter Configuration

bk_video_pipeline_h264e_config_t video_pipeline_h264e_config = {0};

// Configure video pipeline H.264 encoder
video_pipeline_h264e_config.width = 864; // Encoding resolution width, unit px
video_pipeline_h264e_config.height = 480; // Encoding resolution, unit px
video_pipeline_h264e_config.fps = FPS30; // Encoding frame rate, unit fps
video_pipeline_h264e_config.sw_rotate_angle = ROTATE_NONE; // Software rotation angle (using media_rotate_t enumeration type), ROTATE_NONE means no rotation, supports ROTATE_NONE, ROTATE_90, ROTATE_180, ROTATE_270
// Configure video pipeline H.264 encoder callback function
video_pipeline_h264e_config.h264e_cb = &h264e_cbs;

4.3.2 H264 Encoding Function Configuration

// Open H.264 encoder
ret = bk_video_pipeline_open_h264e(g_video_pipeline_handle, &video_pipeline_h264e_config);
if (ret != BK_OK) {
    LOGE("bk_video_pipeline_open_h264e failed, ret: %d\n", ret);
    return ret;
}

4.4 Closing Rotation Function

// Close rotation function
ret = bk_video_pipeline_close_rotate(g_video_pipeline_handle);
if (ret != BK_OK) {
    LOGE("bk_video_pipeline_close_rotate failed, ret: %d\n", ret);
    return ret;
}

4.5 Closing H.264 Encoding Function

// Close H.264 encoder
ret = bk_video_pipeline_close_h264e(g_video_pipeline_handle);
if (ret != BK_OK) {
    LOGE("bk_video_pipeline_close_h264e failed, ret: %d\n", ret);
    return ret;
}

4.6 Getting Module Status

// Get H.264 encoder status
ret = bk_video_pipeline_get_module_status(g_video_pipeline_handle, VIDEO_PIPELINE_MODULE_H264E, &h264e_state);
// Get rotation module status
ret = bk_video_pipeline_get_module_status(g_video_pipeline_handle, VIDEO_PIPELINE_MODULE_ROTATE, &rotate_state);

4.7 Resetting Decoding Status

This function is only called when switching cameras, so that internal modules can determine whether to use the hardware decoding process or software decoding process when obtaining JPEG images again;

// Reset decoding status
ret = bk_video_pipeline_reset_decode(g_video_pipeline_handle);
if (ret != BK_OK) {
    LOGE("bk_video_pipeline_reset_decode failed, ret: %d\n", ret);
    return ret;
}

4.8 Resource Cleanup

if (g_video_pipeline_handle != NULL) {
    // Delete internal resources
    bk_video_pipeline_delete(g_video_pipeline_handle);
    // Set the external global pointer to NULL
    g_video_pipeline_handle = NULL;
}

Video Pipeline

1. Overview

2. Video Pipeline Basic Information

2.1 Supported Features

2.2 Related Header Files

2.3 Reference Project

2.4 Memory Usage

3. Common Application Scenarios

3.1 UVC Camera Hardware Decoding

3.1.1 Image Input

3.1.2 Image Output

3.1.3 H264 Encoding Image Output

3.2 UVC Camera Software Decoding

3.2.1 Image Input

3.2.2 Image Output

3.2.3 H264 Encoding Image Output

3.3 DVP Camera

4. Module Parameter Configuration and Function Calls

4.1 Module Initialization

4.1.1 Initialization Parameter Configuration

4.1.2 Initialization Function Configuration

4.2 Opening Rotation Function

4.2.1 Rotation Function Parameter Configuration

4.2.2 Rotation Function Configuration

4.3 Opening H.264 Encoding Function

4.3.1 H264 Encoding Function Parameter Configuration

4.3.2 H264 Encoding Function Configuration

4.4 Closing Rotation Function

4.5 Closing H.264 Encoding Function

4.6 Getting Module Status

4.7 Resetting Decoding Status

4.8 Resource Cleanup