← Back to blog
ArchitectureFrontendVORA

From Python Server to Pure Browser: The Architecture Pivot That Changed Everything

VORA did not start as a pure browser application. It started as a Python FastAPI server running Faster-Whisper, with a browser frontend that streamed audio to it. That version worked sometimes. When it worked, it looked impressive. But deployment was...

by Jay··4 min read·VORA B.LOG

Series: VORA B.LOG

  1. 1. Why I shipped VORA before writing a single line of backend code
  2. 2. From Python Server to Pure Browser: The Architecture Pivot That Changed Everything ← you are here
  3. 3. The Whisper WASM Experiment: Why Browser AI Is Harder Than It Looks
  4. 4. Why We Killed Speaker Identification (And What We Learned from Two Weeks of Failure)
  5. 5. Building an N-Best Reranking Layer for Better Korean STT (Without Extra API Calls)
  6. 6. Building the Priority Queue: How We Stopped Gemini API Chaos — and Why the First Two Designs Both Failed
  7. 7. Groq Dual-AI Integration: Why We Added a Second AI and What It Actually Fixed
  8. 8. The Meeting Summary Timer Bug: Why setInterval Isn't Enough for Reliable Scheduling
  9. 9. Building a Real Meeting Export: From Raw Transcript to a Usable Report
  10. 10. The Dark Theme Redesign: Building a UI That Looks Like a Professional Tool (After It Looked Like a Hobbyist Project)
  11. 11. The Branding Journey: From a Functional Name to VORA
  12. 12. How We Made VORA Bilingual Without a Heavy Localization Stack
  13. 13. Deploying to Cloudflare Pages: Static Hosting, CORS Headers, and the Sitemap/Robots Incident
  14. 14. How I Fixed AI Over-correction

VORA did not start as a pure browser application. It started as a Python FastAPI server running Faster-Whisper, with a browser frontend that streamed audio to it. That version worked sometimes. When it worked, it looked impressive. But deployment was painful enough that we eventually threw out the server and rebuilt from scratch. This is why.

Version 1: The Server-Side Architecture

The original idea looked clean on paper. A Python backend would handle heavy work: Faster-Whisper transcription, specialized domain models, and multi-engine STT experiments. The browser captured audio chunks and streamed them to the server via WebSocket or HTTP. The server returned text.

The stack looked reasonable in commit history: FastAPI, Faster-Whisper, async threading, and chunk processing. We even wired it for Render deployment. Early commits looked increasingly sophisticated, which in hindsight was the warning sign.

The Bug Log No One Wanted to Write

What commit titles hide is the ratio of fixes to features. It was roughly 3:1. For every new feature, multiple things broke.

  • Audio chunk format issues: Browser MediaRecorder chunks were not always self-contained. We spent days re-encoding and repairing chunk boundaries.
  • Server timeouts: Faster-Whisper on a free instance was not fast enough for real-time UX. Cold starts timed out.
  • Threading problems: FastAPI + Whisper + file I/O on the same execution path caused freezes. Thread pool tuning reduced one problem and created another.
  • Mobile incompatibility: iOS capture defaults and preprocessing tradeoffs increased latency and hurt UX.

A key moment was realizing we were fighting architecture latency itself. You cannot get sub-second perceived response when your path is: capture chunk -> encode -> upload -> infer -> return -> render.

The Moment of Clarity

We reviewed the server diff and asked one question: what user value are we buying with this complexity?

The honest answer: slightly better transcription in controlled cases, with far worse latency, operational cost, and reliability.

The Research Phase: Web Speech API Reality Check

We benchmarked Web Speech API against our server setup.

  • Latency: Web Speech interim results came back in under ~200ms; server results often took seconds.
  • Korean quality: For our target meeting scenarios, quality was competitive once domain correction was added.
  • Reliability: No cold starts, no server memory limits, no backend queue failures.
  • Tradeoffs: Less control, no full offline guarantee, and audio routing considerations.

For VORA's target use case, browser-native transcription was the better product decision.

The Rewrite: Simplify to Web Speech API Only

We removed the server stack in one refactor and kept the product surface users cared about.

Removed:
- server.py (FastAPI app)
- stt_module.py (Faster-Whisper wrapper)
- ensemble_stt.py
- Python dependencies and deployment configs

Kept:
- Frontend pages
- Browser-side logic
- SpeechRecognition-based transcription path

The immediate effect was user-visible stability. Timeout and load complaints dropped sharply.

Building the AI Layer on a Browser Foundation

Once backend firefighting stopped, we could focus on differentiation:

  • domain-aware correction
  • meeting context injection
  • queue design for API limits
  • dual-model workflows

Architecture simplification was not giving up. It was removing drag so product value could ship.

What About Whisper in the Browser?

We still run local-inference experiments in Labs (Whisper WASM, hybrid ASR, related paths). But we now treat heavy browser inference as opt-in experiments, not core UX dependencies.

The Architecture Principle We Kept

Every service you operate is another failure surface. Every internal API hop adds latency risk. Every deployment file adds maintenance cost.

The better question is not "What can we build?" It is "What is the minimum infrastructure required to deliver the core user value?"

For VORA, the answer was simpler than we expected: static frontend + browser speech pipeline + AI correction layer.

Before choosing the technically impressive route, verify it is actually better for the person using the product.

← All posts