This project is based on an end-to-cloud and cloud-to-large-model design solution.
It supports dual-screen display, providing a visual and voice companionship experience along with emotional value.
The solution enables seamless edge-to-cloud integration, supporting various general-purpose large model designs that can directly connect with platforms like OpenAI, DouBao, and DeepSeek.
It effectively leverages cloud-based distributed deployment to reduce network latency and enhance interaction experience.
The solution supports edge-side AEC (Acoustic Echo Cancellation) and NS (Noise Suppression) audio processing algorithms, as well as G711/G722 codec formats. It also supports KWS (Keyword Spotting) wake-up functions and prompt tone playback functions.
The design includes reference solutions and demos for common peripherals, such as gyroscopes, NFC, buttons, vibration motors, Nand Flash, LED light effects, power management, DVP cameras, and dual QPSI screens.
There are three button on the lower right side of the board, corresponding to the silk screen markings S1, S2, and S3; and there is one button K1 on the right side.
power on/off
1.power on: Long press(>= 3 seconds) the button S2 to power on.
2.power off: When the system is in the powered - on state, long press(>= 3 seconds) the button S2 to power off.
LLMs Switch
1.Fisrt time boot and wakeup, default use Large Language Model
2.click the button S2 to switch to Image Recognition Large Model
3.click the button S2 again to switch to Large Language Model
4.Loop operation between step 2 and step 3
Network Provisioning
1.Network Provisioning: When the system is in the powered - on state, long press(>= 3 seconds) the button S1 to enter the state of waiting for network configuration.
Speaker volume control
1.Increase the volume: Single - click the S1 button to turn up the volume.
2.Decrease the volume: Single - click the S3 button to turn down the volume.
restore to factory settings
1.restore to factory settings: Long press the S3 button to restore the device to its factory settings.
reset button K1
1.Reset in shutdown state: Single - click the K1 button to power on the system from the shutdown state.
2.Reset in the powered-on state: Single - click the K1 button, and the system will perform a hard restart while it is powered-on.
Button Function Configuration, Refer to projects/common_components/bk_key_app/key_app_config.h and key_app_service.c, the developer can fill in the corresponding IO pins and the callback function events for the buttons in the table.
The configuration for the long button press duration should refer to the LONG_TICKS macro defined in the multi_button.h header file.
All button events are moved to be executed in a task. If the program executing the button event is blocked or takes too long, it will affect the button response speed.
2.Precautions for GPIO buttons
Ensure that GPIO pins are exclusively used for button functions; otherwise, conflicting functions on the same GPIO pin may result in ineffective button operation.
If the developer’s board is different from the bekan_genie development board, please reconfigure the GPIOs according to the hardware design of your development board.
For details on GPIO usage, refer to the documents in the bk_avdk/bk_idk/docs/bk7258/zh_CN/api-reference/peripheral/bk_gpio.rst.
The development board features red and green status indicator lights. Important information is indicated by red light blinking, general notifications by green light blinking,
and special reminders are signaled by alternating red and green light blinking. For reference code for LED effects development, see led_blink.c.
Green light remains on or continuously off as an indicator
1.When power is turned on, the green light remains on until the user performs an operation or the next event begins.
2.When a conversation starts, the green light turns off.
Blinking of red and green lights indicates specific information
1.User network configuration: User is configuring the network.
Green light blinking status information
1.During power-on networking: Green light blink quickly.
2.Large model server connection successful: Green light blink slowly.
1.The SD-NAND stores local resource files, such as image resource files on the display screen.
2.The SD-NAND storage device defaults to using the FAT32 file system, allowing applications to indirectly invoke the open-source FATFS program interface through the VFS interface for file access.
3.On the PC side, files on the SD-NAND can be accessed for reading and writing via the USB interface.(The USB port located on the left side of the development board.)
4.Please note that files deleted on the PC side may still be in use by the local application, which can lead to system anomalies. It is essential to ensure that deleted files are no longer being accessed.
1.The charging management chip model used in the current development board is ETA3422.
2.When the battery is fully charged, the red light near the charging port will turn off, and the green light will turn on. The red light being on indicates that charging is in progress.
3.Note: During the charging process or when an external power source is connected, the system switches to the external input voltage source for voltage detection instead of using the battery voltage. At this time, the voltage obtained through commands will be the external input voltage.
4.Charging status monitoring depends on GPIO51 and GPIO26.
GPIO51 is responsible for detecting the charging status. When GPIO51 is high, there is an external power supply input; otherwise, there is none.
GPIO26 indicates whether the battery is charging. When GPIO26 is high, the battery is charging; when it is low, the battery is fully charged.
Note: This function requires confirming whether the R14 resistor is soldered on the hardware. If not, additional soldering is needed.
The specific hardware information is subject to the schematic diagram of the project.
5.To enable the charging management function, configure CONFIG_BAT_MONITOR=y. To enable the test cases for charging management, configure CONFIG_BATTERY_TEST=y.
6.After enabling the battery test commands, battery information can be obtained using the battery command:
“battery init” initializes the battery monitoring task.
“battery get_voltage” checks the current battery voltage.
“battery get_level” checks the current battery level.
Note: When using the above commands, ensure that the system is not powered by an external power source; otherwise, the detected voltage will be the external power voltage.
For other specific commands, simply enter “battery” to print the supported commands. For further details, refer to the definitions in cli_battery.c.
7.When the battery level is equal to or below 20%, a low battery warning event will be triggered:
The warning is triggered only when the device is not plugged in; it will not be triggered when connected to external power or charging.
The warning is sent only once per low-battery occurrence.
At this time, the red LED on the other side will blink slowly for 30 seconds.
8.When charging, the charging management task will print “Device is charging…”.
9.When fully charged, it will print “Battery is full.”.
10.Although the battery has low-voltage protection, users are advised to charge the battery promptly when the battery is low to extend its lifespan.
11.If users use batteries from other manufacturers, they need to modify the charge interpolation table (s_chargeLUT) and the battery information in iot_battery_open accordingly.
12.In our SDK, we provide API functions for current, voltage, and battery level. Currently, the battery only supports voltage and battery level detection.
Note: Although the API interface for current detection is retained, it has not been implemented yet. If the user’s device supports current detection, they need to implement the current detection API themselves.
13.Since the hardware currently only supports battery level detection in a non-charging state, hardware modifications are needed to detect voltage during charging:
Removing the D6 diode and R21 resistor will enable voltage detection during charging.
14.The USB port next to the button serves as both a charging port and a serial port for interaction.
15.The ADC interface for power sampling is the ADC0 inside the chip, and there is no need to connect the resistor voltage divider circuit and add ADC channel acquisition externally. ADC0 is directly connected to the VBAT monitoring channel, which is a dedicated interface within the chip, and there is no external connection.
16.Note: The maximum detection voltage of the battery is 4.35V. Above this voltage there is a risk of burning out the system.
1.The LDO is connected to the positive terminal of the motor, while the PWM is connected to the negative terminal. The motor’s vibration strength can be controlled by adjusting the duty cycle of the PWM signal.
2.When the power is turned on by long-pressing the button, the motor will vibrate.
3.Detailed usage examples for PWM can be found in cli_pwm.c.
The development board plays corresponding prompt tones during operation based on different events. Below are the prompt tones associated with each event:
Provision network Over Bluetooth LE
1.Provision Network Over Bluetooth LE: PleaseuseBluetoothLEfornetworkprovision
1.The network configuration countdown is 5 minutes. If the network is not configured within 5 minutes, the chip will enter deep sleep mode(Shutdown).
2.The network error countdown is 5 minutes. If a network error occurs and the network is not restored within 5 minutes, the chip will enter deep sleep mode.
3.The standby state countdown is 3 minutes. After the system powers on, it defaults to standby mode. When you say byebyearmino, the system will also enter standby mode. If no other events occur, the chip will enter deep sleep mode after 3 minutes.
4.You can modify the countdown time in the s_ticket_durations[COUNTDOWN_TICKET_MAX] array in countdown_app.c.
*Thedevicecollectsvoicethroughthemicrophone.*ThevoicedataistransmittedtoAgora's servers using the Agora SDK.*Agora's servers handle communication with the AI Agent large model.*TheserversendsthevoicetotheAIAgent,receivesaresponse,andforwardsthevoiceresponsetothedevice's speaker for playback.Forimageprocessing::*Thedevicecapturesimagedata.*EachframeoftheimageissenttoAgora's servers via the Agora SDK.*TheserverthentransmitstheimagetotheAIAgentlargemodelforrecognition.
To enable the Agora function library, the following configurations need to be enabled on cpu0:
Kconfig
CPU
Format
Value
CONFIG_AGORA_IOT_SDK
CPU0
bool
y
To enable Network Provisioning and start agent, the following configurations need to be enabled on cpu0:
Kconfig
CPU
Format
Value
CONFIG_BK_SMART_CONFIG
CPU0
bool
y
To enable dual screen display and avi play function, the following configurations need to be enabled:
Kconfig
CPU
Format
Value
CONFIG_LCD_SPI_GC9D01
CPU1
bool
y
CONFIG_LCD_SPI_DEVICE_NUM
CPU1
int
2
CONFIG_AVI_PLAY
CPU1
bool
y
CONFIG_DUAL_SCREEN_AVI_PLAY
CPU0 & CPU1
bool
y
CONFIG_LVGL
CPU1
bool
y
CONFIG_LV_IMG_UTILITY_CUSTOMIZE
CPU1
bool
y
CONFIG_LV_COLOR_DEPTH
CPU1
int
16
CONFIG_LV_COLOR_16_SWAP
CPU1
bool
y
2.5 BLE Network Provisioning and Agent Policy Customization Guide
The BLE Network Provisioning and Agent-Startup code are mainly distributed in the directory:”projects/common_components/bk_boarding_service” and “projects/common_components/bk_smart_config”. Customers can refer to the following instructions to customize their own solutions.
1.bk_sconf_prepare_for_smart_config: Entering BLE Network Provisioning mode
void bk_sconf_prepare_for_smart_config(void)
{
smart_config_running = true;
first_time_for_network_reconnect = true;
#if CONFIG_STA_AUTO_RECONNECT
first_time_for_network_provisioning = true;
#endif
app_event_send_msg(APP_EVT_NETWORK_PROVISIONING, 0);//Enter Network Provisioning – indicated by alternating red and green lights.
network_reconnect_stop_timeout_check(); //Disable reconnect timeout check
bk_sconf_trans_stop(); //Close Rtc on device and Multimedia Services
bk_wifi_sta_stop(); //Stop wifi
#if CONFIG_BK_MODEM
extern bk_err_t bk_modem_deinit(void);
bk_modem_deinit(); //If support 4G, disable 4G
#endif
#if !CONFIG_STA_AUTO_RECONNECT //CONFIG_STA_AUTO_RECONNECT default n, use beken reconnect policy
demo_erase_network_auto_reconnect_info();
bk_sconf_erase_agent_info();
#endif
#if CONFIG_NET_PAN && !CONFIG_A2DP_SINK_DEMO && !CONFIG_HFP_HF_DEMO
bk_bt_enter_pairing_mode(0); //Reset BT to initial state
#else
BK_LOGW(TAG, "%s pan disable !!!\n", __func__);
#endif
extern bool ate_is_enabled(void);
if (!ate_is_enabled())
{
bk_genie_boarding_init(); //BLE Network Provisioning init
wifi_boarding_adv_start(); //BLE broadcasting enabled
}
......
}
2.bk_genie_message_handle: Responsible for BLE interaction with mobile app during network provisioning. Customer can disable below code to implement their own solutions.
static void bk_genie_message_handle(void)
{
……
case DBEVT_START_AGORA_AGENT_START:
{
LOGI("DBEVT_START_AGORA_AGENT_START\n");
char payload[256] = {0};
__maybe_unused uint16_t len = 0;
//Uploads module UID and related information to Beken's server. Customers who have deployed their own servers may:
//1.Remove this code section entirely
//2.Modify the implementation by referring to Section 3's bk_sconf_send_agent_info as a reference.
len = bk_sconf_send_agent_info(payload, 256);
bk_genie_boarding_event_notify_with_data(BOARDING_OP_SET_AGENT_INFO, 0, payload, len);
}
break;
case DBEVT_START_AGORA_AGENT_RSP:
{
LOGI("DBEVT_START_AGORA_AGENT_RSP\n");
//Receives channel name and other configuration data from Beken's server. For customers operating their own servers, they may:
//1.Disable this code section entirely
//2.Modify the implementation by referring to Section 4's bk_sconf_prase_agent_info function
bk_sconf_prase_agent_info((char *)msg.param, 1);
}
break;
……
}
3.bk_sconf_send_agent_info: Responsible for transmitting agent configuration parameters to the mobile app (APK) during network provisioning.
4.bk_sconf_prase_agent_info: Responsible for parsing server response parameters (e.g., app_id, channel_name) after agent initialization during provisioning, and activating device-side RTC accordingly.
6.bk_sconf_netif_event_cb: Manages post-WiFi connection processes including:Agent initialization, WiFi/agent information storage, Post-provisioning agent wakeup,(Customizable - customers may replace with their own implementation)
7.bk_sconf_erase_agent_info,bk_sconf_save_agent_info,bk_sconf_get_agent_info: These are all Beken agent background maintenance solutions. Customers may replace them with their own implementations.
8.ir_mode_switch_main: Handles multimodal switching. For custom implementations, customers must either: Implement their own bk_sconf_update_agent_info function, or Adapt Beken’s solution by replacing bk_get_bk_server_url() with their private server endpoint
void ir_mode_switch_main(void)
{
if (!agora_runing) {
BK_LOGW(TAG, "Please Run AgoraRTC First!");
goto exit;
}
ir_mode_switching = 1;
if (!video_started) {
//Switch to image recognition large model
bk_sconf_upate_agent_info("text_and_image");
while (g_agent_offline)
{
if (!agora_runing)
{
goto exit;
}
rtos_delay_milliseconds(100);
}
//open camera
video_turn_on();
#if (CONFIG_DUAL_SCREEN_AVI_PLAY)
if (lvgl_app_init_flag == 1) {
media_app_lvgl_switch_ui(LVGL_UI_DISP_IN_TEXT_AND_IMAGE);
}
#endif
} else {
//close camera
video_turn_off();
//Switch to large language model
bk_sconf_upate_agent_info("text");
#if (CONFIG_DUAL_SCREEN_AVI_PLAY)
if (lvgl_app_init_flag == 1) {
media_app_lvgl_switch_ui(LVGL_UI_DISP_IN_TEXT);
}
#endif
}
exit:
config_ir_mode_switch_thread_handle = NULL;
ir_mode_switching = 0;
rtos_delete_thread(NULL);
}
9.bk_sconf_start_agora_rtc: Manages initialization of both the Agora agent and device-side RTC. The ‘reset’ parameter determines whether to force revert to initial agent configuration on Beken’s server.
Convert the avi video files to be used using the format conversion tool located at <bk_aidksourcecodepath>/bk_avdk/components/multimedia/tools/aviconvert/bk_avi.7z in the SDK. For detailed usage instructions, please refer to the readme.txt file included with the tool.
Place the converted files back into the SD NAND and rename them to contain only English letters or numbers.
Modify the file name passed to the function bk_avi_play_open() in the file <bk_aidksourcecodepath>/project/common_components/dual_screen_avi_play/lvgl_ui.c.
Store the avi video file to be played in the SD NAND. For specific usage of SD NAND, refer to Nand Disk Usage Notes <../../api-reference/nand_disk_note.html>_.
Store the file <bk_aidk source code path>/project/beken_genie/main/resource/genie_eye.avi from the SDK into the SD NAND.
Burn the compiled all-app.bin file and power on to execute.
b)long press Key 2 for 3s to enter network provisioning mode:
c)click the device scan by smart phone
d)Say the wake-up word hiarmino to the onboard mic, the device will play the prompt tone aha after waking up,
and then you can have an AI conversation
Say the key word byebyearmino to the onboard mic, the device will play the prompt tone byebye after detecting it,
then go to sleep and stop talking to the AI
Before reconfiguring the network, you need to remove the device from the original network configuration on your phone, and then repeat the steps in the previous chapter.
To remove the device, follow these steps:
Long press the indicated area, and a prompt box will pop up.
Click OK to complete the operation.
Note
For more APP operations, please refer to the APP documentation:
Debugging commands are intended solely for developers who have a good understanding of the code.
If you are not familiar with the code and process, please first go through the demonstration steps below to familiarize yourself with and understand the code before deciding whether to
According to the operation manual provided by Agora AI Agent, start the AI Agent on the server side.
Currently, the AI Agent on the PC end is started, and the POST command reference please refer to the user manual provided by Agora <bk_aidksourcecodepath>/docs/thirdparty/agora_ai_agent
Q: Why doesn’t the application layer report mic data?
A: Currently, beken_genie defaults to supporting command word-based voice wake-up functionality. Only after wake-up will it report the mic-collected data to the application layer, which then sends the data to AI for conversation. If the customer does not need voice wake-up functionality, they can disable this feature by defining the macro “CONFIG_AUD_INTF_SUPPORT_AI_DIALOG_FREE”.
Q: Why UI resource display abnormally?
A: In this project, the UI resource used must be in AVI format, with a resolution of 320x160, and must be converted using an AVI conversion tool before they can be used. Please check if the UI resource meet the above requirements first.
Q: Which 4G Cat1 modules are already adapted for USB interface?
A: Fibocom LE270/370, luat Air780E, Quectel EC-800M, MobileTek I511
Note: When using the Quectel EC-800M, some additional modifications are required for the adaptation code. The specific patch can be obtained from this link: https://armino.bekencorp.com/article/34.html
Q: How does beken genie connect an external 4G module through USB interface?
