Happy, the AI hardware robot – speech to text

These are the steps to set up the speech to text converter process for the Building Happy, the AI hardware robot project

Happy’s speech-to-text works in two parts. The ESP32 records your voice and sends it over Wi-Fi to your PC, and Whisper turns it into text on the PC. Whisper is too big to run on the ESP32 itself.

Setup on the PC

1. Install Python 3.12:

winget install Python.Python.3.12

2. Open a new terminal in the brain folder. Install PyTorch with GPU support first; this command is for an Nvidia card:

pip install torch --index-url https://download.pytorch.org/whl/cu126

3. Install the rest:

pip install -r requirements.txt

4. Start the server. The first start downloads the Whisper model.

python stt_server.py --model small --language en

5. Let the ESP32 reach the server through Windows Firewall. Run this in an admin PowerShell:

New-NetFirewallRule -DisplayName "Scrappy STT" -Direction Inbound -Protocol TCP -LocalPort 8000 -Action Allow

6. Optionally, test with any 16-bit WAV file before the robot is involved:

curl -X POST -H "Content-Type: audio/wav" --data-binary "@test.wav" http://localhost:8000/stt

Setup on the ESP32

1. At the top of the sketch, fill in WIFI_SSID, WIFI_PASS and STT_URL. For the URL, use your PC’s IPv4 address from ipconfig.
2. Upload it the same way you did mic_test.
3. Open the Serial Monitor, hold BOOT, and say something.