infer-rkllm-openaigitee开源地址https://gitee.com/Vanishi/infer-rkllm-openaigithub开源地址https://github.com/beixiaocai/infer-rkllm-openai基于 Rockchip RKLLM 的视觉语言模型推理服务提供完全兼容 OpenAI API 格式的 HTTP 接口。提示: Intel CPU/GPU 用户请查看 https://gitee.com/Vanishi/infer-openvino-openai作者已准备的模型下载地址https://pan.quark.cn/s/d2b152fbea26硬件要求开发板: Rockchip RK3576 或 RK3588运行时库:/usr/local/lib/librkllmrt.so和/usr/local/lib/librknnrt.so内存: 建议 4GB快速开始1. 安装依赖pip install -r requirements.txt2. 启动服务python infer-rkllm-openai.py \ --model-path /path/to/model.rkllm \ --vision-model /path/to/vision.rknn \ --platform rk3576 \ --host 0.0.0.0 --port 9696常用参数--model-path: RKLLM 模型路径必填--vision-model: Vision 模型路径可选--platform: 平台类型rk3576或rk3588--host: 服务地址默认0.0.0.0--port: 服务端口默认9696--rknn-cores: NPU 核心数 (1/2/3)默认 23. 访问服务首页: http://localhost:9696/管理后台: http://localhost:9696/dashboardAPI: http://localhost:9696/v1/chat/completionsAPI 使用Python SDKfrom openai import OpenAI client OpenAI( api_keysk-rebucca, base_urlhttp://localhost:9696/v1 ) # 文本对话 response client.chat.completions.create( modelqwen3-vl-4b, messages[{role: user, content: 你好}] ) print(response.choices[0].message.content) # 图片理解 import base64 with open(demo.jpeg, rb) as f: image_b64 base64.b64encode(f.read()).decode() response client.chat.completions.create( modelqwen3-vl-4b, messages[{ role: user, content: [ {type: text, text: 描述这张图片}, {type: image_url, image_url: {url: fdata:image/jpeg;base64,{image_b64}}} ] }] ) # 流式输出 stream client.chat.completions.create( modelqwen3-vl-4b, messages[{role: user, content: 你好}], streamTrue ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end, flushTrue)curlcurl http://localhost:9696/v1/chat/completions \ -H Content-Type: application/json \ -H Authorization: Bearer sk-rebucca \ -d {model:qwen3-vl-4b,messages:[{role:user,content:你好}]}运行测试python tests.py # 或指定地址 python tests.py --base-url http://192.168.1.15:9696/v1环境变量export RKLLM_LIB/opt/rkllm/librkllmrt.so export RKNN_LIB/opt/rknn/librknnrt.so注意事项必须在 Rockchip RK3576/RK3588 设备上运行模型必须为.rkllm和.rknn格式W4A16 量化模型内存占用约 2-3GB单线程处理请求会排队版本Python: 3.8Flask: 3.0.0支持平台: RK3576, RK3588
