LeafSense AI is not a vision API with a prompt in front of it. It is a purpose-trained object detection model, a computer vision preprocessing pipeline, a dual-AI cross-validation architecture, and a semantic fuzzy matching engine — deployed as a single production system.
Each stage is an independently engineered system. Remove any one and the product breaks. That is what architecture means.
Every uploaded image is classified using edge density analysis — a classical computer vision technique using PIL's FIND_EDGES filter. Edge pixels above a threshold of 30 are counted; edge density above 0.15 (normalized over 10,000 pixels) identifies a close-up shot. Wide shots route differently than close-ups. This runs with zero API calls and zero latency.
PIL FIND_EDGESDensity Threshold 0.15LANCZOS Resamplingdef detect_image_type(image_data: bytes) -> str: # Resize to 100×100 for fast analysis, convert to greyscale small_img = img.resize((100, 100)).convert('L') edges = small_img.filter(ImageFilter.FIND_EDGES) # Count pixels with strong edge signal edge_pixels = sum(1 for pixel in edges.getdata() if pixel > 30) edge_density = edge_pixels / 10000 # Normalised 0–1 # Close-ups show fine leaf texture → high edge density return "close_up" if edge_density > 0.15 else "wide_view"
A custom-trained YOLOv8 model (best.pt) runs inference on the best-selected image. Critically, YOLO does not identify the plant species — it acts as a binary guard: is there a plant in this image? If not, the request is killed immediately before any OpenAI API call is made. This protects cost, latency, and result quality simultaneously.
Training a domain-specific YOLO model required annotated image datasets for 10 Indian houseplant species under real household conditions — variable lighting, potted plants, partial occlusion, mixed backgrounds. The resulting learned feature weights are irreproducible without that training pipeline.
yolo_results = model("/tmp/temp_detect.jpg", verbose=False) detections = yolo_results[0].boxes if len(detections) == 0: return {"success": False, "error": "No plant detected in image"} # ↑ Request stops here. No OpenAI call made. # Highest-confidence detection wins confidences = detections.conf.cpu().numpy() best_det_idx = confidences.argmax() yolo_confidence = float(confidences[best_det_idx]) yolo_class_name = model.names[int(detections.cls[best_det_idx].item())]
All uploaded images (not just the best one) are sent to GPT-4o independently — with no knowledge of what YOLO detected. Two AI systems, two independent verdicts. This is cross-validation borrowed from clinical diagnostics. Temperature is set to 0.0 — this is a measurement instrument, not a creative tool. Every run must produce the same output for the same input.
GPT-4o visionTemperature 0.010-species whitelist70% confidence gate"""You are an expert plant pathologist specialising in Indian houseplants.
SUPPORTED PLANTS (identify ONLY these 10):
Rosa · Dypsis lutescens · Hibiscus rosa-sinensis
Codiaeum variegatum · Spathiphyllum · Sansevieria trifasciata
Solanum lycopersicum · Ficus lyrata · Epipremnum aureum
Murraya koenigii
RULES:
- Plant NOT in list OR confidence <70% → return "Unknown"
- BE SPECIFIC: 'Nutrient deficiency' → name the NUTRIENT
(Iron / Nitrogen / Magnesium / Calcium / Phosphorus / Potassium)
- Evidence-based only. Do not hallucinate.
"""After GPT-4o responds, a custom JSON extractor handles three output formats (clean JSON, markdown-fenced JSON, JSON embedded in prose). A second post-processing pass enforces domain rules: if the model returns a generic nutrient deficiency, the specific chemical symbol is resolved automatically. A farmer needs to know whether to buy an iron chelate or a nitrogen fertiliser — that distinction is clinically critical.
Regex JSON extractionBrace-matching parserChemical symbol resolutionif "nitrogen" in disease_lower and "deficiency" in disease_lower: diagnosis["disease_scientific_name"] = "N (Nitrogen)" elif "iron" in disease_lower and "deficiency" in disease_lower: diagnosis["disease_scientific_name"] = "Fe (Iron)" elif "magnesium" in disease_lower and "deficiency" in disease_lower: diagnosis["disease_scientific_name"] = "Mg (Magnesium)" # ... Ca (Calcium), P (Phosphorus), K (Potassium)
A custom two-phase similarity engine matches the AI diagnosis against a proprietary product database. Phase 1 tries scientific name + disease. Phase 2 falls back to common name + disease. Each comparison uses Python's SequenceMatcher (Ratcliff/Obershelp algorithm) with a 70% threshold. A faster second path uses RapidFuzz (C extension) with configurable disease/plant weighting (60%/40%). No external service is involved.
SequenceMatcherRapidFuzz (C extension)2-phase fallbackIn-memory cache# Phase 1: scientific name + disease → product DB plant_sim = calculate_similarity(plant_scientific_name, product.scientific_name) disease_sim = calculate_similarity(disease_name, product.disease) combined = (plant_sim * 0.5) + (disease_sim * 0.5) # Phase 2 (fallback): common name + disease if best_score < threshold and plant_common_name: plant_sim = calculate_similarity(plant_common_name, product.scientific_name) combined = (plant_sim * 0.5) + (disease_sim * 0.5) return match if best_score >= 0.70 else None
The AI is only as good as the system surrounding it. These are the engineering decisions that make LeafSense AI reliable in production.
All uploaded images are stored to AWS S3 simultaneously using asyncio.gather(). Sequential uploads would add N × upload_time to total latency. With gather, the overhead is always the time of a single upload regardless of image count.
Database writes never block the HTTP response. FastAPI's BackgroundTasks schedules the database insert after the response is already sent. Users receive their diagnosis result without waiting for any I/O.
pool_pre_ping=True, keepalives (idle: 30s, interval: 10s, count: 5), and 1-hour connection recycling prevent the stale-connection 500 errors common with serverless PostgreSQL like Neon.
Products are loaded from PostgreSQL into a Python list at startup. Every product search scans in-memory rather than querying the database — sub-millisecond search time for any realistic catalog size, regardless of load.
Mobile-first authentication via E2A SMS gateway for Indian numbers. OTP stored with 5-minute TTL, deleted on successful verification. Persistent JWT issued on verification — appropriate for mobile apps where session re-authentication is friction.
Product database is seeded from Excel at startup. If a row has a common plant name but no scientific name, the system fuzzy-matches it against a Plant Map to resolve it automatically. Low-confidence matches are logged for human review; bad data is skipped, not inserted.
Layer by layer.
| Stage | A Generic Wrapper | LeafSense AI |
|---|---|---|
| Image Input | Accepts any image, forwards as-is | Classifies as close-up vs. wide-view using edge density analysis; applies type-aware LANCZOS optimisation; selects best image from multi-upload |
| Plant Detection | None. Sends all images to vision API regardless | Custom-trained YOLO model confirms plant presence; rejects non-plant images before any LLM call; returns class and confidence from learned weights |
| Species ID | Open-ended vision prompt, any output accepted | Whitelist of 10 species; 70% confidence gate; independent of YOLO result (cross-validation); structured JSON schema enforced |
| Disease Diagnosis | "What's wrong with this plant?" prompt | Structured pathology protocol: abiotic-first assessment, systemic vs. isolated damage, specific nutrient chemical symbols enforced in post-processing |
| AI Response | Free text displayed directly to user | JSON extracted via regex + brace-matching parser handling 3 output formats; schema validated; domain rules applied; temperature 0.0 for determinism |
| Product Match | None | Two-phase weighted fuzzy similarity engine (SequenceMatcher + RapidFuzz); 70% threshold; scientific-name-first with common-name fallback; in-memory product cache |
| Storage | None | Parallel async S3 uploads; PostgreSQL persistence via background task; UUID-tracked detection history per user |
| Auth | None or API key passthrough | SMS OTP via E2A gateway; JWT; dual endpoints (authenticated with YOLO + unauthenticated direct mode) |
Every component chosen for reliability and correctness in the Indian market context.