The Inspiration — Why This Is Personal

What Is SilentEar?

How I Built It

How Gemini Live API Powers Real-Time Detection

Scene Intelligence with Gemini 3 Flash

The Google Cloud Stack

Features That Make It Real

Accomplishments I'm Proud Of

What I Learned

Technical Challenges & Lessons Learned This blog post was created for the purposes of entering the Gemini Live Agent Challenge hackathon. #GeminiLiveAgentChallenge This project is deeply personal to me. My son is profoundly deaf by birth, and as he began learning Pakistan Sign Language (PSL) at school, I joined basic training to learn alongside him. During this ongoing journey, I observed firsthand how challenging daily life can be for the Deaf community and their families. Often, deaf individuals feel irrelevant or isolated in gatherings, struggle to communicate smoothly with hearing people, or face physical danger because they cannot hear critical environmental alerts. While we've seen incredible advancements in AI, accessibility tools have largely stagnated around simple speech-to-text transcription. I realized that transcription misses the most crucial part of environmental awareness: the context. A fire alarm, a knock on the door, or a baby crying are critical sounds that text on a screen does not adequately convey in time-sensitive situations. Inspired by my son's experiences and the powerful capabilities of the Gemini Live API, I set out to build an agent that doesn't just transcribe words, but actively listens and interprets the world — providing life-saving cues in the formats Deaf users actually need, including haptic feedback, screen flashes, and visual sign language. SilentEar is a real-time environmental audio interpreter that goes far beyond simple transcription. It continuously monitors ambient sounds, speech, and conversational flow, helping users understand their surroundings and actively participate in conversations. It's powered by two core Google AI services: When critical sounds are detected — a dog barking, a doorbell, a siren, or someone calling a name — SilentEar immediately alerts through customized vibration patterns, visual alerts, and on-screen cues. The system supports advanced triggers, keywords, and a full conversational vocabulary range that users can expand themselves. Crucially, it includes SignMoji, an integrated sign language library that displays relevant sign language videos corresponding to alerts. Users can create custom SignMojis by recording a video, providing a URL, or searching the web. The system automatically generates an icon and syncs everything across devices. For two-way communication, SilentEar provides an AI-powered Voice Deck, enabling text-to-speech with smart, context-aware phrase prediction powered by Gemini 3 Flash — allowing fast, natural communication without typing every word. It also includes a caregiver monitoring dashboard where trusted contacts can view live alerts and device status remotely. I architected a full-stack solution bridging mobile web and backend cloud infrastructure: The heart of SilentEar is a WebSocket connection between the user's device and a Cloud Run backend that proxies audio to the Gemini Live API. The key insight: instead of traditional keyword matching, I use Gemini's function calling capability. The AI model receives a trigger_alert tool and a system prompt describing the user's custom trigger categories. When it hears a matching sound or word, it calls the tool — triggering an instant alert on the user's device. This approach is fundamentally different from traditional sound classifiers. Gemini understands context — it knows the difference between a dog barking on TV and a real dog barking at the door. Beyond real-time audio detection, SilentEar uses Gemini 3 Flash for deeper intelligence: All of these run as server-side REST endpoints on Cloud Run, keeping the mobile client lightweight. Everything runs on Google Cloud: Deployment is fully automated via a single cloudbuild.yaml: One gcloud builds submit command builds the Docker image and deploys to Cloud Run — zero manual steps. SilentEar isn't a demo — it's a usable app with features designed for real deaf users: I'm especially proud of how seamless the SignMoji integration feels. Allowing users to instantly search the web, record their own sign language videos, and sync them securely into their trigger system makes the platform deeply personal and culturally meaningful. Achieving ultra-low latency alerts through Gemini Live function calling also feels transformative in real-world testing. I gained deep experience working with the Web Audio API and real-time streaming constraints in modern browsers. More importantly, building accessibility-first technology taught me about the nuance of Deaf culture — especially why transcription alone is insufficient, and why combining environmental intelligence, visual signals, haptics, and sign language is essential for true inclusion. Challenge 1: WebSocket session management on Cloud Run. Cloud Run has a default timeout and can recycle instances. I configured session affinity and extended timeouts to keep Gemini Live sessions stable. Challenge 2: Audio format compatibility. The browser captures audio as Float32 PCM, but Gemini expects specific formats. I built a real-time PCM encoder that converts and chunks audio for optimal streaming. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to ? It will become hidden in your post, but will still be visible via the comment's permalink. as well , this person and/or