Extracting Signal from Non-Verbal Patients: How the Doolittle Engine Works
A developer walkthrough of AiVet's multimodal pipeline — vision, acoustic, and sensor fusion — that turns observation into structured clinical data for animals that cannot speak.
Human clinical AI has a luxury veterinary AI does not: the patient can describe the symptom. A dog cannot tell you its abdomen hurts. AiVet's core thesis is that the missing channel — subjective self-report — can be partially reconstructed from objective signals a machine can measure: gait asymmetry, facial action units, respiratory acoustics, and remote photoplethysmography.
The pipeline
The Doolittle Engine is a fusion pipeline, not a single model. Each modality produces an independent estimate; a Bayesian fusion layer combines them into a structured observation with calibrated uncertainty.
- Vision — pose landmarks (MediaPipe tasks-vision) drive gait and posture analysis; facial ROIs feed grimace/pain scoring.
- Acoustic — respiratory and vocalisation features are classified for cough character and distress.
- rPPG — subtle colour change in exposed skin/mucosa estimates heart and respiratory rate without contact.
- Fusion — per-modality posteriors combine into one observation vector with confidence, never a hard diagnosis.
Design rule: the engine surfaces observations and possible interpretations for a licensed vet to verify. It does not diagnose. This decision-support framing is a hard boundary across every AiVet surface.
Why the browser
The vision stack runs client-side via WebAssembly (the pose landmarker ships as a .task model loaded in-browser). That keeps raw video off our servers by default — a privacy property that matters when the "patient" is filmed in a client's living room. Server round-trips are reserved for the fusion step and report generation, which run behind an authenticated proxy so no model key ever reaches the browser.
// Vision runs in the browser; the AI key never leaves the server.
// Client posts extracted features (not raw video) to a server proxy:
const res = await fetch('/api/analyze', {
method: 'POST',
body: JSON.stringify({ features, prompt }),
});
// The server holds GEMINI_API_KEY (server-only env). No NEXT_PUBLIC_* AI key.Where it fits in the stack
Signals extracted here flow into VetSorcery's event-sourced visit ledger as objective entries in the SOAP "Objective" section. The vet reviews, edits, and approves — the human stays in the loop at the point of clinical decision. That is the difference between decision support and automation.