VolcEngine RTC

[中文]

1. VolcEngine RTC Introduction

VolcEngine Real-Time Communication (veRTC) provides globally reliable, high-concurrency, low-latency real-time audio/video communication capabilities, enabling various types of real-time communication and interaction. For detailed information, please refer to: https://www.volcengine.com/product/veRTC

2. VolcEngine RTC Code Distribution

  • VolcEngine RTC HAL layer and lib code directory: Located in Armino SMP SDK, path is components/bk_thirdparty/VolcEngineRTCLite

  • BK and VolcEngine RTC transmission integration: components/network_transfer/volc_rtc

  • VolcEngine RTC demo project directory: projects/volc_rtc

3. Prerequisites for Using VolcEngine RTC

Using VolcEngine RTC with large models for AI conversation requires enabling related services and configuring permissions. For details, please refer to VolcEngine online documentation: https://www.volcengine.com/docs/6348/1315561

4. Agent (Intelligent Agent) Startup Methods

AIDK includes two methods for starting Agent: starting Agent through BK server and starting Agent through developer-deployed server. AIDK defaults to starting Agent through BK server for demonstrating VolcEngine RTC effects.

Attention

Starting Agent through BK server for AI conversation is a demo, with each conversation limited to 5 minutes. If you need longer conversation duration, please contact the development board Taobao customer service to obtain a license file. For details, please refer to Chapter 5 license function description. We strongly recommend developers to set up their own servers for convenient duration control, device management, and differentiated deployment

4.1 Starting Agent Through BK Server

After network provisioning is completed, the program will call bk_sconf_start_network_transfer(char *device_id) to start network transfer. This function will call bk_byte_start(device_id) based on configuration, which in turn calls volc_agent_start() to request BK server to start Agent and start device-side RTC, then requires the device to join the room where Agent is located for AI conversation.

Code paths: - bk_sconf_start_network_transfer: components/bk_smart_config/src/core/bk_smart_config_core.c - bk_byte_start: components/network_transfer/volc_rtc/bk_volc_api.c - volc_agent_start: components/network_transfer/volc_rtc/volc_agent_engine.c

4.2 Starting Agent Through Developer-Deployed Server

To facilitate device management for developers, AIDK also supports developers to start Agent through personally deployed servers. For server deployment, please refer to VolcEngine open source code and guide: https://github.com/volcengine/rtc-aigc-embedded-demo To use this method to start Agent, the following configuration is required:

4.2.1 Modify VolcEngine RTC Project Configuration File

Modify the projects/volc_rtc/ap/config/bk7258_ap/config file by setting CONFIG_STARTUP_AGENT_FROM_BK_SERVER=n to disable starting Agent from BK server and change to starting Agent from custom server.

# Disable starting Agent from BK server (enabled by default)
# CONFIG_STARTUP_AGENT_FROM_BK_SERVER is not set

4.2.2 Modify VolcEngine Configuration File

Modify components/network_transfer/volc_rtc/volc_config.h to configure parameters required for requesting custom server.

#if !CONFIG_STARTUP_AGENT_FROM_BK_SERVER
// RTC APP ID
#define CONFIG_RTC_APP_ID    "67********************d1"
// Server address
#define CONFIG_AGENT_SERVER_HOST   "1**.1**.**.**:8080"
#endif

5. License Function Description

VolcEngine RTC supports license billing. License billing is a special billing rule. Please communicate and confirm with VolcEngine before use. AIDK supports License mode starting from v2.0.1.9. To use License billing, ensure that the CONFIG_VOLC_RTC_ENABLE_LICENSE macro in projects/volc_rtc/ap/config/bk7258_ap/config file is configured as y.

CONFIG_VOLC_RTC_ENABLE_LICENSE=y

License is bound to device UID by default. Device UID can be obtained through bk_byte_rtc_print_finger().

After obtaining the License file, please name the License file as VolcEngineRTCLite.lic and copy the License file to the root directory of the file system.

Note

BK7258 AIDK file system uses SD-NAND by default. License file needs to be copied to SD-NAND root directory. For specific usage of SD NAND, please refer to Nand Disk Usage Notes. Developers can contact staff who provide AIDK development boards to obtain license files for experience.

6. Customization Configuration

6.1 Audio Codec Configuration

VolcEngine supports common audio codec formats. AIDK VolcEngine RTC project uses opus by default. If you want to change to G722 codec, you can modify the audio encoder/decoder configuration in projects/volc_rtc/ap/config/bk7258_ap/config file.

# Audio encoder type
CONFIG_AE_AUDIO_ENCODER_TYPE="G722"
# Audio decoder type
CONFIG_AE_AUDIO_DECODER_TYPE="G722"

VolcEngine RTC audio codec related operations include the following:

Code path: components/network_transfer/volc_rtc/

  • Call byte_rtc_set_audio_codec interface to set audio encoding type before byte_rtc_join_room

    Code location: __byte_rtc_start() function in volc_rtc_engine.c

  • Send audio data through byte_rtc_send_audio_data. For details, please refer to bk_byte_rtc_audio_data_send() function implementation

    Code location: bk_volc_api.c

  • Configure corresponding audio encoding type through audio_codec parameter when starting Agent. For details, please refer to volc_start_agent_from_bk_server() or volc_start_agent_from_customer_server() function implementation

    Code location: volc_agent_engine.c

6.2 Vision Recognition Function

VolcEngine project supports vision recognition function by default. AIDK VolcEngine project does not enable vision recognition function by default after network provisioning. You can switch between vision recognition and normal AI conversation functions through S2 button. If you want to enable vision recognition function by default after network provisioning, you can configure CONFIG_VOLC_ENABLE_VISION_BY_DEFAULT=y in projects/volc_rtc/ap/config/bk7258_ap/config file.

CONFIG_VOLC_ENABLE_VISION_BY_DEFAULT=y

When enabling vision recognition, note the following:

Code path: components/network_transfer/volc_rtc/

  • When byte_rtc_join_room, the options parameter needs to set auto_publish_video to true

    Code location: __byte_rtc_start() function in volc_rtc_engine.c

  • Send video data through byte_rtc_send_video_data. For details, please refer to bk_byte_rtc_video_data_send() function implementation

    Code location: bk_volc_api.c

  • When starting Agent, set mode parameter to "vision" to enable vision understanding and fill in corresponding image configuration. For details, please refer to volc_start_agent_from_bk_server() or volc_start_agent_from_customer_server() function implementation

    Code location: volc_agent_engine.c

  • Backend large model needs to support vision recognition function

6.3 Subtitle Function

VolcEngine project disables subtitle function by default. If you want to enable subtitle function, you can configure CONFIG_VOLC_ENABLE_SUBTITLE_BY_DEFAULT=y in projects/volc_rtc/ap/config/bk7258_ap/config file.

CONFIG_VOLC_ENABLE_SUBTITLE_BY_DEFAULT=y

Subtitle receiving function is __on_message_received() function in components/network_transfer/volc_rtc/volc_rtc_engine.c file.

Code path: components/network_transfer/volc_rtc/volc_rtc_engine.c

For VolcEngine subtitle format, please refer to VolcEngine documentation: https://www.volcengine.com/docs/6348/1337284