A real-time assistant inside the Zoom SDK that uses speech-to-text and vision models to read audio and video, interpret intent, and surface contextual prompts and answers — for meetings, demos and classrooms.
The most useful moment in a call is often missed: the question nobody answered, the slide that needed a source, the action item that vanished. We wanted an assistant that lives inside the call and catches it.
The challenge
Doing this live means fusing audio and video, tracking who means what, and responding fast enough to be useful — all inside Zoom's real-time constraints without derailing the conversation.
Our approach
Running inside the Zoom SDK, the agent streams speech-to-text and vision together, interprets speaker intent from both channels, and surfaces the right prompt, answer or source at the moment it's relevant — quietly, alongside the call rather than on top of it.
Under the hood
- Audio — OpenAI speech-to-text, live
- Vision — Vision models on the video stream
- Platform — Zoom SDK integration
- Reasoning — Multimodal intent detection in real time
What we built
- A live multimodal pipeline over audio + video
- Speaker-intent detection fused across channels
- Contextual prompts and answers surfaced in-call
- A Zoom-native experience for meetings, demos and classes
Results
- Questions get answered before the call moves on
- Meetings leave with sources and actions captured
A Lightning Leap case study — one of the AI systems we've designed, built, and shipped.