Compare commits

..
Author SHA1 Message Date
hrz aa2f44dac6 唤醒词文档里 tokens.txt 的行数改成 227
实际包里的 tokens.txt 是 227 行,之前写的 228 行不对,按实测值改过来。
2026-10-27 10:23:37 +08:00
hrz c2c050dbd0 唤醒词文档改准:int8 的定位和 decoder 的版本
int8 只决定选方案一还是方案二,不是 ModelScope 下载的前提,运行时只按四个固定文件名找模型。官方包里 decoder 也有 int8 版,不是只有 ModelScope 那份才有,方案二里 decoder 仍用 fp32 是因为量化对 decoder 收益不大。
2026-10-27 09:13:42 +08:00
hrz 04f8b17d97 文档写清 ModelScope 换模型要连 avg 一起换
方法2 下载的仓库里只有 epoch-12 的模型,文件名里是 avg-2 不是 avg-1。原来只写把 epoch-99 换成 epoch-12,照着换会得到不存在的文件名,cp 直接报错。现在写明要换成 epoch-12-avg-2,并说明仓库里缺 int8 那两份时方法2 不能用,只能改用方法1。
2026-10-26 22:12:21 +08:00
hrz c822648882 唤醒词文档补上拷贝改名这步并改对精度说法
ModelScope 下的文件留在自己的目录里,运行时只认 models/ 下的四个固定名字,不拷进去改名就会报找不到模型。官方包里 decoder 只有 fp32 一版,速度优先那套写成了 int8 三件套,跟同文件的注意事项打架,一并改对。
2026-10-26 21:38:47 +08:00
hrz 4ec37c6e33 文档改准:缺模型时是唤醒词没起来,不是麦克风坏了
评审指出原来那两句说反了。音频插件先把麦克风打开,唤醒词插件才去找模型文件,缺文件的是后者,所以是检测没起来、数据没人消费,麦克风一直在采。keywords.txt 也是启动时自己生成,不用提前准备。
2026-10-26 21:32:37 +08:00
hrz 7af5630b2a 在数字人文档里写清唤醒词模型的前置条件
模型文件不在仓库里,默认配置下缺模型只会打印一句唤醒词服务启动失败,
页面能开但麦克风采集起不来。digital-human 的 README、唤醒词专题文档
和 paddlespeech 部署文档三处都补上这个前置条件,并指向模型下载那一节。
2026-10-26 21:16:47 +08:00
hrz 43f30c06c1 给唤醒词运行时补上 numpy 依赖
core/detector.py 和 core/microphone.py 在模块层直接 import numpy,
requirements.txt 里没有这一条。干净环境里照文档装完依赖就启动,
会报 ModuleNotFoundError,页面都开不起来。版本跟 xiaozhi-server 那份一样,
两个模块共用一个环境时不会冲突。
2026-10-26 21:16:47 +08:00
13 changed files with 160 additions and 142 deletions

No files matched your search

+3 -3
View File
@@ -212,7 +212,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
| 工具名称 | 位置 | 使用方法 | 功能说明 |
|:---:|:---|:---:|:---:|
| 音频交互测试工具 | main》digital-human》index.html | 在 `main/digital-human` 按其 README 装好依赖后执行 `python start.py`,然后访问 `http://127.0.0.1:8006/index.html` | 测试音频播放和接收功能,验证Python端音频处理是否正常 |
| 音频交互测试工具 | main》digital-human》index.html | 在 `main/digital-human` 执行 `python start.py` 后访问 `http://127.0.0.1:8006/index.html` | 测试音频播放和接收功能,验证Python端音频处理是否正常 |
| 模型响应测试工具 | main》xiaozhi-server》performance_tester.py | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、VLLM(视觉模型)、TTS(语音合成)等核心模块的响应速度 |
> 💡 提示:测试模型速度时,只会测试配置了密钥的模型。
@@ -256,7 +256,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
| 使用方式 | 支持平台 | 免费平台 |
|:---:|:---:|:---:|
| openai 接口调用 | 阿里百炼、火山引擎、DeepSeek、智谱、Gemini、科大讯飞、LM Studio | 智谱、Gemini |
| openai 接口调用 | 阿里百炼、火山引擎、DeepSeek、智谱、Gemini、科大讯飞 | 智谱、Gemini |
| ollama 接口调用 | Ollama | - |
| dify 接口调用 | Dify | - |
| fastgpt 接口调用 | Fastgpt | - |
@@ -300,7 +300,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
| 使用方式 | 支持平台 | 免费平台 |
|:---:|:---:|:---:|
| 本地使用 | FunASR、SherpaASR、SherpaParaformerASR、VoskASR | FunASR、SherpaASR、SherpaParaformerASR、VoskASR |
| 接口调用 | FunASRServer、火山引擎、科大讯飞、腾讯云、阿里云及百炼、百度云、OpenAI ASR、GroqASR、Qwen3ASRFlash | FunASRServer |
| 接口调用 | FunASRServer、火山引擎、科大讯飞、腾讯云、阿里云、百度云、OpenAI ASR、GroqASR、Qwen3ASRFlash | FunASRServer |
---
+20 -8
View File
@@ -28,16 +28,22 @@ wget https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/sherpa-o
tar xvf sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01.tar.bz2
# 方法2:使用ModelScope
# 这份仓库里只有 epoch-12 的模型,文件名里的 avg 是 avg-2,不是 avg-1
# 照抄下面「配置模型文件」的命令时,把 epoch-99-avg-1 换成 epoch-12-avg-2(avg 一起换,只换 epoch 数字会找不到文件)
# int8 只决定你选方案一还是方案二,不是方法2 的前提:这份仓库里 epoch-12 的 fp32 三件套齐全,
# 用方法2 + 方案一(精度优先)照样能配出四个必需文件
pip install modelscope
python -c "
from modelscope import snapshot_download
snapshot_download('pkufool/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01', cache_dir='./models')
snapshot_download('pkufool/sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01', local_dir='./sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01')
"
```
运行时只认 `models/` 目录下的四个固定文件名(`encoder.onnx`、`decoder.onnx`、`joiner.onnx`、`tokens.txt`),也不搜子目录(判定见 `wakeword_runtime/core/detector_assets.py`)。文件留在下载目录里不做拷贝改名,启动会报 `No valid model directory found`。两种方法下载完都要接着做下面「配置模型文件」那节的拷贝改名。
#### 2. 配置模型文件
模型包下载后包含以下文件:
模型包下载后包含以下文件(epoch-99 那几行只有官方包有,ModelScope 那份只有 epoch-12):
```
sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/
@@ -45,7 +51,9 @@ sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01/
├── encoder-epoch-12-avg-2-chunk-16-left-64.onnx
├── encoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx # 速度优先
├── encoder-epoch-99-avg-1-chunk-16-left-64.onnx # 精度优先
├── decoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx # 速度优先
├── decoder-epoch-12-avg-2-chunk-16-left-64.onnx
├── decoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx # 速度优先
├── decoder-epoch-99-avg-1-chunk-16-left-64.onnx # 精度优先
├── joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx # 速度优先
├── joiner-epoch-12-avg-2-chunk-16-left-64.onnx
@@ -87,7 +95,8 @@ cd sherpa-onnx-kws-zipformer-wenetspeech-3.3M-2024-01-01
# 创建模型目录
mkdir -p ../models
# 复制速度优先的epoch-99 int8三件套
# 复制速度优先的epoch-99 三件套:encoder 和 joiner 用 int8,decoder 用 fp32
# 官方包里 decoder 也有 int8 版,但量化对 decoder 收益不大,一般仍用 fp32
cp encoder-epoch-99-avg-1-chunk-16-left-64.int8.onnx ../models/encoder.onnx
cp decoder-epoch-99-avg-1-chunk-16-left-64.onnx ../models/decoder.onnx
cp joiner-epoch-99-avg-1-chunk-16-left-64.int8.onnx ../models/joiner.onnx
@@ -98,9 +107,10 @@ cp tokens.txt ../models/tokens.txt
**注意事项**:
- **不要混用 fp32 与 int8**:三个模型文件必须保持一致的精度
- **精度怎么选**:encoder、joiner、decoder 三件套都有 int8 和 fp32 两版,官方包和 ModelScope 那份都是这样;官方文档的说法是量化对 decoder 收益不大,所以方案二里 encoder 和 joiner 用 int8、decoder 仍用 fp32
- **int8 只决定选方案一还是方案二**:运行时只按上面的文件名找模型,不看你拷的是 int8 还是 fp32,所以方法2(ModelScope)配方案一(精度优先)照样能跑
- **优先选择 epoch-99**:比 epoch-12 训练更充分,精度更高
- **必需文件**:`encoder.onnx` + `decoder.onnx` + `joiner.onnx` + `tokens.txt` + `keywords.txt`
- **必需文件**:`encoder.onnx` + `decoder.onnx` + `joiner.onnx` + `tokens.txt` 四个,`keywords.txt` 不在其中,它不用自己准备,首次启动会用配置里的唤醒词自动生成
### 最终模型文件结构
@@ -111,7 +121,7 @@ wakeword_runtime/models/
├── encoder.onnx # 编码器模型(重命名后)
├── decoder.onnx # 解码器模型(重命名后)
├── joiner.onnx # 连接器模型(重命名后)
├── tokens.txt # 拼音 Token 映射表(228行版本)
├── tokens.txt # 拼音 Token 映射表(227行版本)
├── keywords.txt # 关键词配置文件(首次启动自动生成)
└── keywords_raw.txt # 可选,runtime 不依赖
```
@@ -194,12 +204,14 @@ python start.py
### 首次使用
1. 准备 `models/` 目录下的模型文件和 `tokens.txt`
2. 确认 `models/keywords.txt` 存在
1. 先按上面「模型下载(必需)」那节把模型包下载解压,再按「配置模型文件」那节把模型放到 `models/` 目录,该目录下要有 `encoder.onnx`、`decoder.onnx`、`joiner.onnx` 和 `tokens.txt` 四个文件
2. `models/keywords.txt` 不用自己准备,首次启动会用配置里的唤醒词自动生成
3. 在 `digital-human` 目录运行 `python start.py`
4. 浏览器打开 `http://127.0.0.1:8006/index.html`
5. 进入设置页检查“唤醒词”配置
模型没放到位就启动,只会打印「警告: 唤醒词服务启动失败」:页面能打开,麦克风的输入流也已经打开,但没有唤醒词检测在消费这些数据,语音唤不醒。
### 修改唤醒词
1. 打开数字人页面设置
+2
View File
@@ -91,6 +91,8 @@ TTS:
```py
python app.py
```
启动`main/digital-human`下的`python start.py`前,先按[唤醒词模型说明](./digital-human-wakeword.md)把模型下载并配置到`main/digital-human/wakeword_runtime/models/`,默认配置下缺模型只会打印「警告: 唤醒词服务启动失败」。页面能打开,麦克风的输入流也已经打开,但没有唤醒词检测在消费这些数据,语音唤不醒。
启动`main/digital-human`下的`python start.py`后,打开`http://127.0.0.1:8006/index.html`,测试连接和发送消息时paddlespeech端是否有输出日志
输出日志参考:
+27 -26
View File
@@ -6,7 +6,7 @@
Dieses Projekt basiert auf der Theorie und Technologie der Mensch-Maschine-symbiotischen Intelligenz zur Entwicklung intelligenter Terminal-Hardware- und Software-Systeme<br/>und bietet Backend-Dienste für das Open-Source-Hardware-Projekt
<a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
Implementiert mit Python, Java und Vue gemäß dem <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi-Kommunikationsprotokoll</a><br/>
Unterstützt MQTT+UDP-Protokoll, Websocket-Protokoll, MCP-Endpunkte, Stimmabdruckerkennung und Wissensdatenbank
Unterstützt MQTT+UDP-Protokoll, Websocket-Protokoll, MCP-Endpunkte und Stimmabdruckerkennung
</p>
<p align="center">
@@ -55,31 +55,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
<td>
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
<picture>
<img alt="Gefühl der Antwortgeschwindigkeit" src="../images/demo9.png" /></picture>
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
<picture>
<img alt="Das Geheimnis der Geschwindigkeitsoptimierung" src="../images/demo6.png" /></picture>
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
<picture>
<img alt="Xiaozhi-Digitalmensch mit Sprachaktivierung" src="../images/demo8.png" /></picture>
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
<picture>
<img alt="Anruf von Gerät zu Gerät" src="../images/demo0.png" /></picture>
<img alt="设备呼叫设备,打电话" src="../images/demo0.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
<picture>
<img alt="Komplexes medizinisches Szenario" src="../images/demo1.png" /></picture>
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
</a>
</td>
</tr>
@@ -87,31 +87,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="Musik abspielen, Wetter abfragen und Nachrichten vorlesen" src="../images/demo7.png" /></picture>
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
<picture>
<img alt="Unterbrechung in Echtzeit" src="../images/demo10.png" /></picture>
<img alt="实时打断" src="../images/demo10.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
<picture>
<img alt="Objekte per Foto erkennen" src="../images/demo12.png" /></picture>
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
<picture>
<img alt="Haushaltsgeräte ein- und ausschalten" src="../images/demo5.png" /></picture>
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
<picture>
<img alt="Aufgaben mit mehreren Befehlen" src="../images/demo11.png" /></picture>
<img alt="多指令任务" src="../images/demo11.png" /></picture>
</a>
</td>
</tr>
@@ -119,31 +119,31 @@ Möchten Sie die Nutzungseffekte sehen? Klicken Sie auf die Videos unten 🎥
<td>
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
<picture>
<img alt="MCP-Endpunkt" src="../images/demo13.png" /></picture>
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
<picture>
<img alt="MQTT-Befehlsübermittlung" src="../images/demo4.png" /></picture>
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
<picture>
<img alt="Stimmabdruckerkennung" src="../images/demo14.png" /></picture>
<img alt="声纹识别" src="../images/demo14.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
<picture>
<img alt="Eigene Stimmfarbe" src="../images/demo2.png" /></picture>
<img alt="自定义音色" src="../images/demo2.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
<picture>
<img alt="Gespräch auf Kantonesisch" src="../images/demo3.png" /></picture>
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
</a>
</td>
</tr>
@@ -210,7 +210,7 @@ Dieses Projekt bietet die folgenden Testwerkzeuge, um Ihnen bei der Überprüfun
| Werkzeugname | Standort | Verwendungsmethode | Funktionsbeschreibung |
|:---:|:---|:---:|:---:|
| Audio-Interaktionstesttool | main》digital-human》index.html | In `main/digital-human` zuerst die Abhängigkeiten wie in der README installieren, dann `python start.py` ausführen und `http://127.0.0.1:8006/index.html` öffnen | Testet Audio-Wiedergabe- und Empfangsfunktionen, überprüft, ob die Python-seitige Audioverarbeitung normal ist |
| Audio-Interaktionstesttool | main》digital-human》index.html | In `main/digital-human` `python start.py` ausführen und dann `http://127.0.0.1:8006/index.html` öffnen | Testet Audio-Wiedergabe- und Empfangsfunktionen, überprüft, ob die Python-seitige Audioverarbeitung normal ist |
| Modell-Reaktionstesttool | main》xiaozhi-server》performance_tester.py | Ausführen `python performance_tester.py` | Testet die Reaktionsgeschwindigkeit von Kernmodulen wie ASR (Spracherkennung), LLM (großes Modell), VLLM (Vision-Modell), TTS (Sprachsynthese) |
> 💡 Hinweis: Beim Testen der Modellgeschwindigkeit werden nur Modelle mit konfigurierten Schlüsseln getestet.
@@ -218,7 +218,7 @@ Dieses Projekt bietet die folgenden Testwerkzeuge, um Ihnen bei der Überprüfun
---
## Funktionsliste ✨
### Implementiert ✅
![Architekturdiagramm der Installation aller Module](../images/deploy2.png)
![请参考-全模块安装架构图](../images/deploy2.png)
| Funktionsmodul | Beschreibung |
|:---:|:---|
| Kernarchitektur | Basierend auf [MQTT+UDP-Gateway](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mqtt-gateway-integration.md), WebSocket und HTTP-Servern, bietet vollständiges Konsolenverwaltungs- und Authentifizierungssystem |
@@ -226,7 +226,7 @@ Dieses Projekt bietet die folgenden Testwerkzeuge, um Ihnen bei der Überprüfun
| Stimmabdruckerkennung | Unterstützt Mehrbenutzer-Stimmabdruckregistrierung, -verwaltung und -erkennung, verarbeitet parallel mit ASR, Echtzeit-Sprecheridentitätserkennung und Weitergabe an LLM für personalisierte Antworten |
| Intelligenter Dialog | Unterstützt mehrere LLM (große Sprachmodelle), implementiert intelligenten Dialog |
| Visuelle Wahrnehmung | Unterstützt mehrere VLLM (Vision Large Models), implementiert multimodale Interaktion |
| Absichtserkennung | Unterstützt externe LLM-Absichtserkennung, autonome Funktionsaufrufe durch große Modelle, bietet plugin-basierten Absichtsverarbeitungsmechanismus |
| Absichtserkennung | Unterstützt LLM-Absichtserkennung, Function Call-Funktionsaufruf, bietet plugin-basierten Absichtsverarbeitungsmechanismus |
| Gedächtnissystem | Unterstützt lokales Kurzzeitgedächtnis, mem0ai-Schnittstellengedächtnis, PowerMem intelligentes Gedächtnis, mit Gedächtniszusammenfassungsfunktion |
| Wissensdatenbank | Unterstützt RAGFlow-Wissensdatenbank, ermöglicht großem Modell die Bewertung, ob Wissensdatenbank benötigt wird, bevor geantwortet wird |
| Werkzeugaufruf | Unterstützt Client-IOT-Protokoll, Client-MCP-Protokoll, Server-MCP-Protokoll, MCP-Endpunktprotokoll, benutzerdefinierte Werkzeugfunktionen |
@@ -254,7 +254,7 @@ Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|:---:|:---:|:---:|
| OpenAI-Schnittstellenaufrufe | Alibaba Bailian, Volcano Engine Doubao, Volcano Engine AI Gateway, DeepSeek, Zhipu ChatGLM, Gemini, iFLYTEK, LM Studio | Zhipu ChatGLM, Gemini |
| OpenAI-Schnittstellenaufrufe | Alibaba Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini, iFLYTEK | Zhipu ChatGLM, Gemini |
| Ollama-Schnittstellenaufrufe | Ollama | - |
| Dify-Schnittstellenaufrufe | Dify | - |
| FastGPT-Schnittstellenaufrufe | FastGPT | - |
@@ -280,7 +280,7 @@ Tatsächlich kann jedes VLLM, das OpenAI-Schnittstellenaufrufe unterstützt, int
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|:---:|:---:|:---:|
| Schnittstellenaufrufe | EdgeTTS, iFLYTEK, Volcano Engine Doubao TTS, Volcano Engine AI Gateway TTS, Tencent Cloud, Alibaba Cloud TTS, AliYun Stream TTS, Alibaba Bailian CosyVoice TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS, Volcano Dual-Stream TTS | EdgeTTS, CosyVoiceSiliconflow (teilweise) |
| Schnittstellenaufrufe | EdgeTTS, iFLYTEK, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, AliYun Stream TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS, Volcano Dual-Stream TTS | EdgeTTS, CosyVoiceSiliconflow (teilweise) |
| Lokale Dienste | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
---
@@ -298,7 +298,7 @@ Tatsächlich kann jedes VLLM, das OpenAI-Schnittstellenaufrufe unterstützt, int
| Verwendungsmethode | Unterstützte Plattformen | Kostenlose Plattformen |
|:---:|:---:|:---:|
| Lokale Verwendung | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
| Schnittstellenaufrufe | DoubaoASR, Doubao Streaming ASR, Doubao Streaming ASR V2, FunASRServer, iFLYTEK, TencentASR, AliyunASR, Aliyun Streaming ASR, Alibaba Bailian Streaming ASR, Baidu ASR, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
| Schnittstellenaufrufe | DoubaoASR, Doubao Streaming ASR, FunASRServer, iFLYTEK, TencentASR, AliyunASR, Aliyun Streaming ASR, Baidu ASR, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
---
@@ -354,10 +354,11 @@ Die Stimmabdruckerkennung erfordert einen separat bereitgestellten [voiceprint-a
| <img src="../images/logo_contributors.png" width="160"> | [Code-Mitwirkende](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors) | Dank an [alle Code-Mitwirkenden](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors), Ihre Bemühungen haben das Projekt robuster und leistungsfähiger gemacht. |
<a href="https://www.star-history.com/?repos=xinnan-tech%2Fxiaozhi-esp32-server&type=date&legend=top-left">
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&theme=dark&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
</picture>
</a>
+25 -24
View File
@@ -55,31 +55,31 @@ Want to see the usage effects? Click the videos below 🎥
<td>
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
<picture>
<img alt="The feel of the response speed" src="../images/demo9.png" /></picture>
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
<picture>
<img alt="The secret to speeding things up" src="../images/demo6.png" /></picture>
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
<picture>
<img alt="Xiaozhi digital human with voice wake-up" src="../images/demo8.png" /></picture>
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
<picture>
<img alt="Device-to-device calling" src="../images/demo0.png" /></picture>
<img alt="设备呼叫设备,打电话" src="../images/demo0.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
<picture>
<img alt="A complex medical scenario" src="../images/demo1.png" /></picture>
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
</a>
</td>
</tr>
@@ -87,31 +87,31 @@ Want to see the usage effects? Click the videos below 🎥
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="Play music, check the weather, and read the news" src="../images/demo7.png" /></picture>
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
<picture>
<img alt="Real-time interruption" src="../images/demo10.png" /></picture>
<img alt="实时打断" src="../images/demo10.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
<picture>
<img alt="Take a photo to identify objects" src="../images/demo12.png" /></picture>
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
<picture>
<img alt="Turn home appliances on and off" src="../images/demo5.png" /></picture>
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
<picture>
<img alt="Multi-command tasks" src="../images/demo11.png" /></picture>
<img alt="多指令任务" src="../images/demo11.png" /></picture>
</a>
</td>
</tr>
@@ -119,31 +119,31 @@ Want to see the usage effects? Click the videos below 🎥
<td>
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
<picture>
<img alt="MCP access point" src="../images/demo13.png" /></picture>
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
<picture>
<img alt="MQTT command delivery" src="../images/demo4.png" /></picture>
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
<picture>
<img alt="Voiceprint recognition" src="../images/demo14.png" /></picture>
<img alt="声纹识别" src="../images/demo14.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
<picture>
<img alt="Custom voice timbre" src="../images/demo2.png" /></picture>
<img alt="自定义音色" src="../images/demo2.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
<picture>
<img alt="Talking in Cantonese" src="../images/demo3.png" /></picture>
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
</a>
</td>
</tr>
@@ -210,7 +210,7 @@ This project provides the following testing tools to help you verify the system
| Tool Name | Location | Usage Method | Function Description |
|:---:|:---|:---:|:---:|
| Audio Interaction Test Tool | main》digital-human》index.html | In `main/digital-human`, install dependencies as its README says, then run `python start.py` and open `http://127.0.0.1:8006/index.html` | Tests audio playback and reception functions, verifies if Python-side audio processing is normal |
| Audio Interaction Test Tool | main》digital-human》index.html | Run `python start.py` in `main/digital-human`, then open `http://127.0.0.1:8006/index.html` | Tests audio playback and reception functions, verifies if Python-side audio processing is normal |
| Model Response Test Tool | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Tests response speed of core modules such as ASR(speech recognition), LLM(large model), VLLM(vision model), TTS(speech synthesis) |
> 💡 Note: When testing model speed, only models with configured keys will be tested.
@@ -218,7 +218,7 @@ This project provides the following testing tools to help you verify the system
---
## Feature List ✨
### Implemented ✅
![Full-module installation architecture diagram](../images/deploy2.png)
![请参考-全模块安装架构图](../images/deploy2.png)
| Feature Module | Description |
|:---:|:---|
| Core Architecture | Based on [MQTT+UDP gateway](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mqtt-gateway-integration.md), WebSocket and HTTP servers, provides complete console management and authentication system |
@@ -226,7 +226,7 @@ This project provides the following testing tools to help you verify the system
| Voiceprint Recognition | Supports multi-user voiceprint registration, management, and recognition, processes in parallel with ASR, real-time speaker identity recognition and passes to LLM for personalized responses |
| Intelligent Dialogue | Supports multiple LLM(large language models), implements intelligent dialogue |
| Visual Perception | Supports multiple VLLM(vision large models), implements multimodal interaction |
| Intent Recognition | Supports pluggable external LLM intent recognition and autonomous LLM function calling, provides plugin-based intent processing mechanism |
| Intent Recognition | Supports LLM intent recognition, Function Call function calling, provides plugin-based intent processing mechanism |
| Memory System | Supports local short-term memory, mem0ai interface memory, PowerMem intelligent memory, with memory summarization functionality |
| Knowledge Base | Supports RAGFlow knowledge base, enabling LLM to judge whether to schedule the knowledge base after receiving the user's question, and then answer the question |
| Tool Calling | Supports client IOT protocol, client MCP protocol, server MCP protocol, MCP endpoint protocol, custom tool functions |
@@ -254,7 +254,7 @@ Xiaozhi is an ecosystem. When using this product, you can also check out other [
| Usage Method | Supported Platforms | Free Platforms |
|:---:|:---:|:---:|
| OpenAI interface calls | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK, LM Studio | Zhipu, Gemini |
| OpenAI interface calls | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK | Zhipu, Gemini |
| Ollama interface calls | Ollama | - |
| Dify interface calls | Dify | - |
| FastGPT interface calls | FastGPT | - |
@@ -298,7 +298,7 @@ In fact, any VLLM that supports OpenAI interface calls can be integrated and use
| Usage Method | Supported Platforms | Free Platforms |
|:---:|:---:|:---:|
| Local use | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
| Interface calls | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud and Bailian, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
| Interface calls | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
---
@@ -354,10 +354,11 @@ Voiceprint recognition requires a separately deployed [voiceprint-api](https://g
| <img src="../images/logo_contributors.png" width="160"> | [Code Contributors](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors) | Thanks to [all code contributors](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors), your efforts have made the project more robust and powerful. |
<a href="https://www.star-history.com/?repos=xinnan-tech%2Fxiaozhi-esp32-server&type=date&legend=top-left">
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&theme=dark&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
</picture>
</a>
+14 -16
View File
@@ -168,8 +168,8 @@ Este proyecto ofrece dos formas de despliegue. Elija según lo que necesite:
#### 🚀 Elección del método de despliegue
| Método de despliegue | Características | Escenarios | Documentación de despliegue | Requisitos | Tutoriales en vídeo |
|---------|------|---------|---------|---------|---------|
| **Instalación simplificada** | Diálogo inteligente, gestión de un solo agente | Entornos con pocos recursos; los datos se guardan en archivos de configuración y no hace falta base de datos | [①Versión Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Despliegue desde el código fuente](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| Ejecutar `FunASR` en local consume bastante memoria; si todo va por API, el consumo de recursos es bajo | - |
| **Instalación de todos los módulos** | Diálogo inteligente, gestión multiusuario, gestión de varios agentes, uso de la interfaz de la consola inteligente | Experiencia con todas las funciones; los datos se guardan en una base de datos |[①Versión Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Despliegue desde el código fuente](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Tutorial de actualización automática desde el código fuente](../dev-ops-integration.md) | Ejecutar `FunASR` en local consume bastante memoria, y además requiere recursos de base de datos; si todo va por API, el consumo de recursos es bajo | [Tutorial en vídeo para arrancar el código fuente en local](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
| **Instalación simplificada** | Diálogo inteligente, gestión de un solo agente | Entornos con pocos recursos; los datos se guardan en archivos de configuración y no hace falta base de datos | [①Versión Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Despliegue desde el código fuente](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 núcleos y 4 GB si usa `FunASR`; 2 núcleos y 2 GB si todo va por API | - |
| **Instalación de todos los módulos** | Diálogo inteligente, gestión multiusuario, gestión de varios agentes, uso de la interfaz de la consola inteligente | Experiencia con todas las funciones; los datos se guardan en una base de datos |[①Versión Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Despliegue desde el código fuente](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Tutorial de actualización automática desde el código fuente](../dev-ops-integration.md) | 4 núcleos y 8 GB si usa `FunASR`; 2 núcleos y 4 GB si todo va por API | [Tutorial en vídeo para arrancar el código fuente en local](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
Las preguntas frecuentes y los tutoriales relacionados están en [este enlace](../FAQ.md)
@@ -210,8 +210,8 @@ Este proyecto incluye las siguientes herramientas para comprobar el sistema y el
| Herramienta | Ubicación | Uso | Qué hace |
|:---:|:---|:---:|:---:|
| Prueba de interacción de audio | main》digital-human》index.html | En `main/digital-human`, instale las dependencias como indica su README, después ejecute `python start.py` y abra `http://127.0.0.1:8006/index.html` | Prueba la reproducción y la recepción de audio, y comprueba que el procesamiento de audio en Python funciona |
| Prueba de respuesta de los modelos | main》xiaozhi-server》performance_tester.py | Ejecute `python performance_tester.py` | Mide la velocidad de respuesta de los principales módulos: ASR (reconocimiento de voz), LLM (modelo de lenguaje), VLLM (modelo de visión) y TTS (síntesis de voz) |
| Prueba de interacción de audio | main》digital-human》index.html | En `main/digital-human`, ejecute `python start.py` y abra `http://127.0.0.1:8006/index.html` | Prueba la reproducción y la recepción de audio, y comprueba que el procesamiento de audio en Python funciona |
| Prueba de respuesta de los modelos | main》xiaozhi-server》performance_tester.py | Ejecute `python performance_tester.py` | Mide la velocidad de respuesta de los cuatro módulos principales: ASR (reconocimiento de voz), LLM (modelo de lenguaje), VLLM (modelo de visión) y TTS (síntesis de voz) |
> 💡 Nota: al medir la velocidad de los modelos, solo se prueban los modelos que tienen una clave configurada.
@@ -226,15 +226,15 @@ Este proyecto incluye las siguientes herramientas para comprobar el sistema y el
| Huella de voz | Admite el registro, la gestión y el reconocimiento de la huella de voz de varios usuarios; se procesa en paralelo con el ASR, identifica en tiempo real quién habla y se lo pasa al LLM para una respuesta personalizada |
| Diálogo inteligente | Admite varios LLM (modelos de lenguaje) para mantener un diálogo inteligente |
| Percepción visual | Admite varios VLLM (modelos de visión) para una interacción multimodal |
| Reconocimiento de intención | Admite el reconocimiento de intención con un modelo de lenguaje (LLM) externo y conectable, y la llamada autónoma a funciones del modelo, y ofrece un mecanismo de procesamiento de intenciones mediante plugins |
| Reconocimiento de intención | Admite el reconocimiento de intención con el modelo de lenguaje (LLM) y la llamada autónoma a funciones del modelo, y ofrece un mecanismo de procesamiento de intenciones mediante plugins |
| Sistema de memoria | Admite memoria local a corto plazo, memoria por la interfaz mem0ai y memoria inteligente PowerMem, con resumen de la memoria |
| Base de conocimiento | Admite la base de conocimiento RAGFlow, para que el modelo decida si debe consultarla antes de responder |
| Llamada a herramientas | Admite el protocolo IOT del cliente, el protocolo MCP del cliente, el protocolo MCP del servidor, el protocolo de punto de acceso MCP y funciones de herramientas personalizadas |
| Envío de instrucciones | Apoyado en el protocolo MQTT, permite enviar instrucciones MCP desde la consola inteligente a los dispositivos ESP32 |
| Backend de administración | Ofrece una interfaz web de administración, con gestión de usuarios, configuración del sistema y gestión de dispositivos. La interfaz se muestra en chino simplificado, chino tradicional, inglés, alemán, vietnamita y portugués de Brasil |
| Herramientas de prueba | Incluye herramientas de prueba de rendimiento y de interacción de audio |
| Backend de administración | Ofrece una interfaz web de administración, con gestión de usuarios, configuración del sistema y gestión de dispositivos. La interfaz se muestra en chino simplificado, chino tradicional e inglés |
| Herramientas de prueba | Incluye herramientas de prueba de rendimiento, de modelos de visión y de interacción de audio |
| Soporte de despliegue | Admite el despliegue con Docker y el despliegue local, y ofrece una gestión completa de los archivos de configuración |
| Sistema de plugins | Admite extensiones de plugins funcionales y el desarrollo de plugins propios; los nuevos archivos de plugin se cargan automáticamente en la siguiente conexión del dispositivo, y los cambios en el código de un plugin ya cargado requieren reiniciar el servicio |
| Sistema de plugins | Admite extensiones de plugins funcionales, el desarrollo de plugins propios y la carga en caliente de plugins |
### En desarrollo 🚧
@@ -254,7 +254,7 @@ Xiaozhi es un ecosistema. Al usar este producto, también puede ver otros [proye
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|:---:|:---:|:---:|
| Llamadas por la interfaz OpenAI | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK, LM Studio | Zhipu, Gemini |
| Llamadas por la interfaz OpenAI | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK | Zhipu, Gemini |
| Llamadas por la interfaz Ollama | Ollama | - |
| Llamadas por la interfaz Dify | Dify | - |
| Llamadas por la interfaz FastGPT | FastGPT | - |
@@ -280,8 +280,8 @@ En la práctica, se puede integrar cualquier VLLM que admita llamadas por la int
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|:---:|:---:|:---:|
| Llamadas por interfaz | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud y Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow (parcial) |
| Servicio local | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
| Llamadas por interfaz | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud y Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS, MinimaxTTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow (parcial) |
| Servicio local | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, Index-TTS, PaddleSpeech | Index-TTS, PaddleSpeech, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
---
@@ -297,8 +297,8 @@ En la práctica, se puede integrar cualquier VLLM que admita llamadas por la int
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|:---:|:---:|:---:|
| Uso local | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
| Llamadas por interfaz | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud y Bailian, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
| Uso local | FunASR, SherpaASR | FunASR, SherpaASR |
| Llamadas por interfaz | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR | FunASRServer |
---
@@ -306,9 +306,7 @@ En la práctica, se puede integrar cualquier VLLM que admita llamadas por la int
| Forma de uso | Plataformas compatibles | Plataformas gratuitas |
|:---:|:---:|:---:|
| Llamadas por interfaz | Servicio de reconocimiento de huella de voz (voiceprint-api) | - |
El reconocimiento de huella de voz requiere desplegar por separado el servicio [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) (basado en 3D-Speaker), que depende de MySQL; consulte la [guía de integración de la huella de voz](../voiceprint-integration.md)
| Uso local | 3D-Speaker | 3D-Speaker |
---
+26 -25
View File
@@ -55,31 +55,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
<td>
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
<picture>
<img alt="A sensação da velocidade de resposta" src="../images/demo9.png" /></picture>
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
<picture>
<img alt="O segredo para otimizar a velocidade" src="../images/demo6.png" /></picture>
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
<picture>
<img alt="Humano digital Xiaozhi com ativação por voz" src="../images/demo8.png" /></picture>
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
<picture>
<img alt="Chamadas de dispositivo para dispositivo" src="../images/demo0.png" /></picture>
<img alt="设备呼叫设备,打电话" src="../images/demo0.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
<picture>
<img alt="Cenário médico complexo" src="../images/demo1.png" /></picture>
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
</a>
</td>
</tr>
@@ -87,31 +87,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="Reproduzir música, consultar o tempo e ler as notícias" src="../images/demo7.png" /></picture>
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
<picture>
<img alt="Interrupção em tempo real" src="../images/demo10.png" /></picture>
<img alt="实时打断" src="../images/demo10.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
<picture>
<img alt="Reconhecer objetos com uma foto" src="../images/demo12.png" /></picture>
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
<picture>
<img alt="Controlar o acionamento de eletrodomésticos" src="../images/demo5.png" /></picture>
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
<picture>
<img alt="Tarefas com vários comandos" src="../images/demo11.png" /></picture>
<img alt="多指令任务" src="../images/demo11.png" /></picture>
</a>
</td>
</tr>
@@ -119,31 +119,31 @@ Quer ver os efeitos de uso? Clique nos vídeos abaixo 🎥
<td>
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
<picture>
<img alt="Ponto de acesso MCP" src="../images/demo13.png" /></picture>
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
<picture>
<img alt="Envio de comandos MQTT" src="../images/demo4.png" /></picture>
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
<picture>
<img alt="Reconhecimento de impressão vocal" src="../images/demo14.png" /></picture>
<img alt="声纹识别" src="../images/demo14.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
<picture>
<img alt="Tom de voz personalizado" src="../images/demo2.png" /></picture>
<img alt="自定义音色" src="../images/demo2.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
<picture>
<img alt="Conversa em cantonês" src="../images/demo3.png" /></picture>
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
</a>
</td>
</tr>
@@ -210,7 +210,7 @@ Este projeto fornece as seguintes ferramentas de teste para ajudá-lo a verifica
| Nome da Ferramenta | Localização | Método de Uso | Descrição da Função |
|:---:|:---|:---:|:---:|
| Ferramenta de Teste de Interação por Áudio | main》digital-human》index.html | Em `main/digital-human`, instale as dependências como indica o README, depois execute `python start.py` e abra `http://127.0.0.1:8006/index.html` | Testa as funções de reprodução e recepção de áudio, verifica se o processamento de áudio no lado Python está normal |
| Ferramenta de Teste de Interação por Áudio | main》digital-human》index.html | Execute `python start.py` em `main/digital-human` e depois abra `http://127.0.0.1:8006/index.html` | Testa as funções de reprodução e recepção de áudio, verifica se o processamento de áudio no lado Python está normal |
| Ferramenta de Teste de Resposta de Modelo | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Testa a velocidade de resposta de módulos principais como ASR(reconhecimento de fala), LLM(modelo de linguagem), VLLM(modelo de visão), TTS(síntese de voz) |
> 💡 Nota: Ao testar a velocidade dos modelos, apenas os modelos com chaves configuradas serão testados.
@@ -218,7 +218,7 @@ Este projeto fornece as seguintes ferramentas de teste para ajudá-lo a verifica
---
## Lista de Funcionalidades ✨
### Implementado ✅
![Diagrama de arquitetura da instalação de todos os módulos](../images/deploy2.png)
![请参考-全模块安装架构图](../images/deploy2.png)
| Módulo de Funcionalidade | Descrição |
|:---:|:---|
| Arquitetura Principal | Baseado em [gateway MQTT+UDP](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mqtt-gateway-integration.md), servidores WebSocket e HTTP, fornece sistema completo de gerenciamento de console e autenticação |
@@ -226,7 +226,7 @@ Este projeto fornece as seguintes ferramentas de teste para ajudá-lo a verifica
| Reconhecimento de Impressão Vocal | Suporta registro, gerenciamento e reconhecimento de impressão vocal de múltiplos usuários, processa em paralelo com o ASR, reconhecimento de identidade do falante em tempo real e repassa ao LLM para respostas personalizadas |
| Diálogo Inteligente | Suporta múltiplos LLM (modelos de linguagem de grande porte), implementa diálogo inteligente |
| Percepção Visual | Suporta múltiplos VLLM (modelos de visão de grande porte), implementa interação multimodal |
| Reconhecimento de Intenção | Suporta reconhecimento de intenção por LLM externo e conectável, chamada de função autônoma do modelo de linguagem, fornece mecanismo de processamento de intenção baseado em plugins |
| Reconhecimento de Intenção | Suporta reconhecimento de intenção por LLM, Function Call (chamada de função), fornece mecanismo de processamento de intenção baseado em plugins |
| Sistema de Memória | Suporta memória local de curto prazo, memória via interface mem0ai, memória inteligente PowerMem, com funcionalidade de resumo de memória |
| Base de Conhecimento | Suporta base de conhecimento RAGFlow, permitindo que o LLM julgue se deve acionar a base de conhecimento após receber a pergunta do usuário, e então responda à pergunta |
| Chamada de Ferramentas | Suporta protocolo IOT do cliente, protocolo MCP do cliente, protocolo MCP do servidor, protocolo de endpoint MCP, funções de ferramentas personalizadas |
@@ -254,7 +254,7 @@ Xiaozhi é um ecossistema. Ao utilizar este produto, você também pode conferir
| Método de Uso | Plataformas Suportadas | Plataformas Gratuitas |
|:---:|:---:|:---:|
| Chamadas via interface OpenAI | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK, LM Studio | Zhipu, Gemini |
| Chamadas via interface OpenAI | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK | Zhipu, Gemini |
| Chamadas via interface Ollama | Ollama | - |
| Chamadas via interface Dify | Dify | - |
| Chamadas via interface FastGPT | FastGPT | - |
@@ -298,7 +298,7 @@ Na verdade, qualquer VLLM que suporte chamadas via interface OpenAI pode ser int
| Método de Uso | Plataformas Suportadas | Plataformas Gratuitas |
|:---:|:---:|:---:|
| Uso local | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
| Chamadas via interface | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud e Bailian, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
| Chamadas via interface | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
---
@@ -308,7 +308,7 @@ Na verdade, qualquer VLLM que suporte chamadas via interface OpenAI pode ser int
|:---:|:---:|:---:|
| Chamadas via interface | Serviço de voiceprint (voiceprint-api) | - |
O reconhecimento de impressão vocal exige um serviço [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) implantado separadamente (baseado em 3D-Speaker) e MySQL. Veja o [guia de integração de voiceprint](../voiceprint-integration.md).
O reconhecimento de voz por impressão digital exige um serviço [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) implantado separadamente (baseado em 3D-Speaker) e MySQL. Veja o [guia de integração de voiceprint](../voiceprint-integration.md).
---
@@ -354,10 +354,11 @@ O reconhecimento de impressão vocal exige um serviço [voiceprint-api](https://
| <img src="../images/logo_contributors.png" width="160"> | [Contribuidores de Código](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors) | Agradecimentos a [todos os contribuidores de código](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors), seus esforços tornaram o projeto mais robusto e poderoso. |
<a href="https://www.star-history.com/?repos=xinnan-tech%2Fxiaozhi-esp32-server&type=date&legend=top-left">
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&theme=dark&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
</picture>
</a>
+31 -30
View File
@@ -6,7 +6,7 @@
Dự án này dựa trên lý thuyết và công nghệ trí tuệ cộng sinh người-máy để phát triển hệ thống phần mềm và phần cứng thiết bị đầu cuối thông minh<br/>Cung cấp dịch vụ backend cho dự án phần cứng thông minh mã nguồn mở
<a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
Được triển khai bằng Python, Java, Vue theo <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">giao thức truyền thông Xiaozhi</a><br/>
Hỗ trợ giao thức MQTT+UDP, giao thức Websocket, điểm truy cập MCP, nhận dạng vân giọng và kho tri thức
Hỗ trợ giao thức MQTT+UDP, giao thức Websocket, điểm truy cập MCP, nhận dạng giọng nói và kho tri thức
</p>
<p align="center">
@@ -39,7 +39,7 @@ Spearheaded by Professor Siyuan Liu's Team (South China University of Technology
</br>
Được dẫn dắt bởi nhóm Giáo sư Lưu Tư Nguyên (Đại học Bách khoa Nam Trung Quốc)
</br>
<img src="../images/hnlg.jpg" alt="Đại học Bách khoa Nam Trung Quốc" width="50%">
<img src="../images/hnlg.jpg" alt="华南理工大学" width="50%">
</p>
---
@@ -55,31 +55,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
<td>
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
<picture>
<img alt="Cảm giác tốc độ phản hồi" src="../images/demo9.png" /></picture>
<img alt="响应速度感受" src="../images/demo9.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1vchQzaEse" target="_blank">
<picture>
<img alt="Bí quyết tối ưu tốc độ" src="../images/demo6.png" /></picture>
<img alt="速度优化秘诀" src="../images/demo6.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1WEcxzFEAT" target="_blank">
<picture>
<img alt="Người ảo số Xiaozhi có đánh thức bằng giọng nói" src="../images/demo8.png" /></picture>
<img alt="小智数字人 支持语音唤醒" src="../images/demo8.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CKVz6UEuB" target="_blank">
<picture>
<img alt="Gọi giữa các thiết bị" src="../images/demo0.png" /></picture>
<img alt="设备呼叫设备,打电话" src="../images/demo0.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1C1tCzUEZh" target="_blank">
<picture>
<img alt="Tình huống y tế phức tạp" src="../images/demo1.png" /></picture>
<img alt="复杂医疗场景" src="../images/demo1.png" /></picture>
</a>
</td>
</tr>
@@ -87,31 +87,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="Phát nhạc, tra cứu thời tiết và đọc tin tức" src="../images/demo7.png" /></picture>
<img alt="播放音乐查询天气播报新闻" src="../images/demo7.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
<picture>
<img alt="Ngắt giữa chừng theo thời gian thực" src="../images/demo10.png" /></picture>
<img alt="实时打断" src="../images/demo10.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
<picture>
<img alt="Nhận dạng vật thể qua ảnh chụp" src="../images/demo12.png" /></picture>
<img alt="拍照识物品" src="../images/demo12.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
<picture>
<img alt="Bật tắt thiết bị gia dụng" src="../images/demo5.png" /></picture>
<img alt="控制家电开关" src="../images/demo5.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
<picture>
<img alt="Nhiệm vụ nhiều lệnh" src="../images/demo11.png" /></picture>
<img alt="多指令任务" src="../images/demo11.png" /></picture>
</a>
</td>
</tr>
@@ -119,31 +119,31 @@ Muốn xem hiệu quả sử dụng? Hãy xem video 🎥
<td>
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
<picture>
<img alt="Điểm truy cập MCP" src="../images/demo13.png" /></picture>
<img alt="MCP接入点" src="../images/demo13.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1zUW5zJEkq" target="_blank">
<picture>
<img alt="Gửi lệnh MQTT" src="../images/demo4.png" /></picture>
<img alt="MQTT指令下发" src="../images/demo4.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
<picture>
<img alt="Nhận dạng vân giọng" src="../images/demo14.png" /></picture>
<img alt="声纹识别" src="../images/demo14.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
<picture>
<img alt="Giọng nói tùy chỉnh" src="../images/demo2.png" /></picture>
<img alt="自定义音色" src="../images/demo2.png" /></picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
<picture>
<img alt="Trò chuyện bằng tiếng Quảng Đông" src="../images/demo3.png" /></picture>
<img alt="使用粤语交流" src="../images/demo3.png" /></picture>
</a>
</td>
</tr>
@@ -170,7 +170,7 @@ Dự án này cung cấp hai phương pháp triển khai, vui lòng chọn theo
| Phương pháp triển khai | Đặc điểm | Tình huống áp dụng | Tài liệu triển khai | Yêu cầu cấu hình | Video hướng dẫn |
|---------|------|---------|---------|---------|---------|
| **Cài đặt tối giản** | Đối thoại thông minh, quản lý đơn tác nhân | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](../Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| `FunASR` cục bộ cần nhiều bộ nhớ; dùng toàn API thì mức dùng tài nguyên thấp | - |
| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, thao tác qua bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](../dev-ops-integration.md) | `FunASR` cục bộ cần nhiều bộ nhớ, thêm tài nguyên cơ sở dữ liệu; dùng toàn API thì mức dùng tài nguyên thấp| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](../Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](../dev-ops-integration.md) | `FunASR` cục bộ cần nhiều bộ nhớ, thêm tài nguyên cơ sở dữ liệu; dùng toàn API thì mức dùng tài nguyên thấp| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
Câu hỏi thường gặp và hướng dẫn liên quan, vui lòng tham khảo [liên kết này](../FAQ.md)
@@ -193,7 +193,7 @@ Công cụ kiểm tra dịch vụ: https://2662r3426b.vicp.fun/test/
>
> 2. `Cấu hình streaming`: Phù hợp với demo, đào tạo, hơn 2 đồng thời, v.v., sử dụng công nghệ xử lý streaming, tốc độ phản hồi nhanh hơn, trải nghiệm tốt hơn.
>
> Từ phiên bản `0.5.2`, dự án hỗ trợ cấu hình streaming, so với các phiên bản trước đó, tốc độ phản hồi cải thiện khoảng `2.5 giây`, cải thiện đáng kể trải nghiệm người dùng.
> Từ phiên bản `0.5.2`, dự án hỗ trợ cấu hình streaming, so với phiên bản đầu, tốc độ phản hồi cải thiện khoảng `2.5 giây`, cải thiện đáng kể trải nghiệm người dùng.
| Tên module | Cài đặt miễn phí cho người mới | Cấu hình streaming |
|:---:|:---:|:---:|
@@ -211,7 +211,7 @@ Dự án này cung cấp các công cụ kiểm tra sau để giúp bạn xác m
| Tên công cụ | Vị trí | Phương pháp sử dụng | Mô tả chức năng |
|:---:|:---|:---:|:---:|
| Công cụ kiểm tra tương tác âm thanh | main》digital-human》index.html | Trong `main/digital-human`, cài đặt các phụ thuộc theo README, rồi chạy `python start.py` và mở `http://127.0.0.1:8006/index.html` | Kiểm tra chức năng phát và nhận âm thanh, xác minh xử lý âm thanh phía Python có bình thường không |
| Công cụ kiểm tra tương tác âm thanh | main》digital-human》index.html | Chạy `python start.py` trong `main/digital-human`, sau đó mở `http://127.0.0.1:8006/index.html` | Kiểm tra chức năng phát và nhận âm thanh, xác minh xử lý âm thanh phía Python có bình thường không |
| Công cụ kiểm tra phản hồi mô hình | main》xiaozhi-server》performance_tester.py | Thực hiện `python performance_tester.py` | Kiểm tra tốc độ phản hồi của các module cốt lõi như ASR(Nhận dạng giọng nói), LLM(Mô hình lớn), VLLM(Mô hình thị giác), TTS(Tổng hợp giọng nói) |
> 💡 Gợi ý: Khi kiểm tra tốc độ mô hình, chỉ kiểm tra các mô hình đã cấu hình khóa.
@@ -219,7 +219,7 @@ Dự án này cung cấp các công cụ kiểm tra sau để giúp bạn xác m
---
## Danh sách tính năng ✨
### Đã thực hiện ✅
![Sơ đồ kiến trúc cài đặt đầy đủ các module](../images/deploy2.png)
![请参考-全模块安装架构图](../images/deploy2.png)
| Module chức năng | Mô tả |
|:---:|:---|
| Kiến trúc cốt lõi | Dựa trên [cổng MQTT+UDP](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mqtt-gateway-integration.md), WebSocket, máy chủ HTTP, cung cấp hệ thống quản lý bảng điều khiển và xác thực hoàn chỉnh |
@@ -255,7 +255,7 @@ Xiaozhi là một hệ sinh thái, khi bạn sử dụng sản phẩm này, bạ
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|:---:|:---:|:---:|
| Gọi giao diện openai | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFLYTEK, LM Studio | Zhipu, Gemini |
| Gọi giao diện openai | Alibaba Bailian, Volcano Engine, DeepSeek, Zhipu, Gemini, iFlytek | Zhipu, Gemini |
| Gọi giao diện ollama | Ollama | - |
| Gọi giao diện dify | Dify | - |
| Gọi giao diện fastgpt | Fastgpt | - |
@@ -281,7 +281,7 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|:---:|:---:|:---:|
| Gọi giao diện | EdgeTTS, iFLYTEK, Volcano Engine, Tencent Cloud, Alibaba Cloud và Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow(một phần) |
| Gọi giao diện | EdgeTTS, iFlytek, Volcano Engine, Tencent Cloud, Alibaba Cloud và Bailian, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, CustomTTS, OpenAITTS, MinimaxTTS | EdgeTTS, CosyVoiceSiliconflow(một phần) |
| Dịch vụ cục bộ | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, IndexStreamTTS, PaddleSpeechTTS | IndexStreamTTS, PaddleSpeechTTS, FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3 |
---
@@ -299,7 +299,7 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|:---:|:---:|:---:|
| Sử dụng cục bộ | FunASR, SherpaASR, SherpaParaformerASR, VoskASR | FunASR, SherpaASR, SherpaParaformerASR, VoskASR |
| Gọi giao diện | FunASRServer, Volcano Engine, iFLYTEK, Tencent Cloud, Alibaba Cloud và Bailian, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
| Gọi giao diện | FunASRServer, Volcano Engine, iFlytek, Tencent Cloud, Alibaba Cloud, Baidu Cloud, OpenAI ASR, GroqASR, Qwen3ASRFlash | FunASRServer |
---
@@ -307,9 +307,9 @@ Trên thực tế, bất kỳ VLLM nào hỗ trợ gọi giao diện openai đ
| Phương pháp sử dụng | Nền tảng hỗ trợ | Nền tảng miễn phí |
|:---:|:---:|:---:|
| Gọi giao diện | Dịch vụ nhận dạng vân giọng (voiceprint-api) | - |
| Gọi giao diện | Dịch vụ nhận dạng giọng nói (voiceprint-api) | - |
Nhận dạng vân giọng cần dịch vụ [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) được triển khai riêng (dựa trên 3D-Speaker) và MySQL. Xem [hướng dẫn tích hợp voiceprint](../voiceprint-integration.md).
Nhận dạng giọng nói theo dấu vân tay cần dịch vụ [voiceprint-api](https://github.com/xinnan-tech/voiceprint-api) được triển khai riêng (dựa trên 3D-Speaker) và MySQL. Xem [hướng dẫn tích hợp voiceprint](../voiceprint-integration.md).
---
@@ -355,10 +355,11 @@ Nhận dạng vân giọng cần dịch vụ [voiceprint-api](https://github.com
| <img src="../images/logo_contributors.png" width="160"> | [Người đóng góp mã](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors) | Cảm ơn [tất cả người đóng góp mã](https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors), sự cống hiến của bạn khiến dự án mạnh mẽ và vững chắc hơn. |
<a href="https://www.star-history.com/?repos=xinnan-tech%2Fxiaozhi-esp32-server&type=date&legend=top-left">
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&theme=dark&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<img alt="Star History Chart" src="https://api.star-history.com/chart?repos=xinnan-tech/xiaozhi-esp32-server&type=date&legend=top-left&sealed_token=cQovHKgZjqEnQ-svfYfN392irNvGuq-6pyv4cA8nd2jQEhQLz1ETV4YHTVk2UZyLMFbCQuZA7jduRh3YbeK5WPYaRLrfmIimGQa3lram652jJL9oQk-UuSZA5H6L4dPIhZc8KCc-Ur_UAUNbly7TePpnTR2otGknBLCOjOliD4fk1st6z7tPEDVjSRx5" />
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
</picture>
</a>
+2
View File
@@ -16,6 +16,8 @@ digital-human 是独立的数字人测试模块,负责提供本地测试页面
pip install -r wakeword_runtime/requirements.txt
```
唤醒词模型要另外下载:仓库里不带模型文件,默认配置(`wakeword_runtime/config.json` 里 `wakeword.enabled` 为 true)下,`wakeword_runtime/models/` 里没有模型文件时启动会打印「警告: 唤醒词服务启动失败」。页面能打开,麦克风的输入流也已经打开,但没有唤醒词检测在消费这些数据,语音唤不醒,页面上的其它功能不受影响。模型下载地址和摆放位置见[唤醒词专题文档](../../docs/digital-human-wakeword.md)。
启动模块:
```bash
@@ -31,8 +31,6 @@ class AudioPlugin(Plugin):
self.source.stop()
def shutdown(self) -> None:
if self.source is not None:
self.source.stop()
if self.app is not None:
self.app.audio_source = None
self.source = None
@@ -1,3 +1,6 @@
sherpa-onnx==1.12.29
sounddevice>=0.4.4
pypinyin==0.55.0
# 唤醒词运行时在 core/detector.py、core/microphone.py 里直接用 numpy,
# 版本跟 main/xiaozhi-server/requirements.txt 一致,两个模块共用一个环境时不会冲突
numpy==1.26.4
@@ -51,14 +51,13 @@ class TestRuntimeApplication:
logger.info("test runtime application started")
def stop(self) -> None:
was_running = self._is_running
self._is_running = False
if was_running:
self.event_bridge.publish_service_stopping()
logger.info("test runtime application stopped")
# 启动中途失败时 _is_running 还是 False,但插件可能已经开过采集流,
# 这里不提前返回,保证采集流一定被关掉
if not self._is_running:
return
self.event_bridge.publish_service_stopping()
self.plugins.stop_all()
self._is_running = False
logger.info("test runtime application stopped")
def shutdown(self) -> None:
self.stop()
+1 -1
View File
@@ -2,7 +2,7 @@ import os
import importlib.util
import asyncio
print("使用前请根据仓库根目录下 docs/performance_tester.md 的说明准备配置。")
print("使用前请根据docs/performance_tester.md的说明准备配置。")
def list_performance_tester_modules():