2026-11-01 — views
NVIDIA Engineers Build Grumpy Pirate House Using Nemotron Voice Agent
Read this because Demonstrates how open small language models and real-time voice pipelines can create complex, low-latency interactive characters on local hardware.
An NVIDIA engineer created an interactive haunted house using Nemotron Voice Agent, Bolt E4B, and Audio2Face-3D on a single workstation.
The MonsterHouse Concept
Rick Grandy, an NVIDIA engineer, developed MonsterHouse, an interactive projection for Halloween that functions as a grumpy old pirate ship reincarnated as a garage door. The character listens to trick-or-treaters, answers with nautical complaints, and softens when visitors use polite phrases like “please” and “thank you.” The system is designed to be self-contained, running on a single workstation equipped with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an NVIDIA RTX PRO 4000 Blackwell.
Technical Architecture
The pipeline relies on several NVIDIA Nemotron components. NVIDIA Nemotron 3 ASR handles speech-to-text conversion, while NVIDIA Nemotron 3.5 Bolt E4B 4GB serves as the conversational brain. This open small language model was selected for its low compute footprint and suitability for local agents. A LoRA adapter, trained on over 5,000 multi-turn conversations, shapes the model’s personality. For speech synthesis, NVIDIA Magpie TTS Zeroshot generates audio from short reference samples, and NVIDIA Audio2Face-3D animates the character’s mouth in Unreal Engine 5. NVIDIA Nemotron Voice Agent coordinates the entire workflow.
Performance and Latency
To maintain immersion, the system prioritizes low latency. The pipeline releases speakable text before an entire answer is generated, allowing the house to begin speaking while the rest of the sentence is still being processed. Median end-of-speech-to-audio latency with Bolt Q4 and TTS is 1.70 seconds, dropping to 0.72 seconds for cached responses. The workload is split across two GPUs: the language and voice model pipeline runs on the RTX PRO 6000, while Unreal Engine graphics and Audio2Face-3D run on the RTX PRO 4000. This separation ensures consistent frame rates for animation regardless of processing load.
Character Design and Animation
The visual design features a wooden garage door with jagged lips and virtual fire effects, created in Autodesk Maya and exported to Unreal Engine. The character lacks eyes or eyebrows, relying on Audio2Face-3D for expression. The voice uses zero-shot conditioning with a reference sample to achieve a deep, demonic tone. Seven distinct moods, from grumpy to delighted, are controlled by emotion tags provided by the LLM. A “Sample your Voice” interface allows the creator to record specific emotional deliveries for different scenarios.
Practitioner note
Developers building real-time interactive agents should consider using small language models like Bolt E4B with LoRA adapters to balance personality and compute constraints. Splitting heavy graphics workloads and language processing across separate GPUs is critical for maintaining consistent animation frame rates in real-time applications.