This guide reorganizes a FreeDiDi tutorial published on July 4, 2026. The commands and performance observations come from the source and were not independently reproduced by LingjingHub. Check current project documentation before installation because Python, CUDA, PyTorch, and model versions can change.
Architecture
The proposed local pipeline has four parts: Whisper converts speech to text; a local model served by llama.cpp or Ollama generates the response; Qwen3-TTS converts the response back to speech; and a local browser interface provides microphone control, live transcription, and status feedback. Once models and dependencies are downloaded, the main conversation path can run locally.
What you need
Use a current Windows 10 or Windows 11 system with Python 3.11, Git, FFmpeg, enough disk space for the selected models, and a supported GPU if you want low latency. GPU memory requirements depend on model size, quantization, context length, and whether speech recognition and synthesis share the same device.
1. Prepare an isolated Python environment
Install Python, Git, and FFmpeg from their official sources. Create a dedicated virtual environment instead of installing packages globally. If PowerShell blocks activation, consult Microsoft's execution-policy documentation and use the narrowest scope appropriate for your machine; do not disable security controls globally.
2. Install the speech pipeline
Activate the virtual environment and install the speech-to-speech package documented by the project. The source tutorial uses Whisper for Chinese recognition and Qwen3-TTS for Chinese speech synthesis. Confirm the package's supported Python, PyTorch, CUDA, and model versions before proceeding.
3. Start a local language-model endpoint
The source uses llama.cpp with a quantized Qwen model and an OpenAI-compatible endpoint on 127.0.0.1. Ollama can be used instead if it already serves the selected model. Keep the endpoint bound to localhost unless you have deliberately configured authentication, firewall rules, and network access controls.
4. Connect speech recognition, the LLM, and speech synthesis
Configure the speech service to use Whisper, the local OpenAI-compatible endpoint, and Qwen3-TTS. A successful start should expose the local real-time WebSocket service and show that the selected models have loaded. The first launch can take longer because model files may need to be downloaded.
5. Open the local browser interface
Run the interface from the Hugging Face voice demo project, open its localhost address in Chrome or Edge, set the local WebSocket endpoint, and grant microphone access only to the local page. The interface should indicate listening, processing, speaking, and error states without embedding a third-party video player.
Troubleshooting
If the model server starts but the voice interface cannot connect, verify the port, protocol, and service order. If CUDA acceleration is unavailable, confirm the installed driver, PyTorch build, and llama.cpp package match the hardware. Reduce model size, context length, or speech-model size when memory is insufficient.
Security boundary
Download executables, models, and scripts from official project pages whenever possible. Do not bypass Windows SmartScreen for unsigned batch files without inspecting them. Do not expose the local model or WebSocket ports to the public internet, and do not assume that local processing automatically makes every downloaded model or dependency trustworthy.
Official references
Source article
Break the goal into input, action, and output before adding review checkpoints.
Keep examples, failure conditions, and review criteria so the tutorial stays reusable.