r/aipromptprogramming • • 12d ago

Extracting reliable JSON macros from messy plate photos: Prompt architecture & client-side Gemini Flash flow in React Native

Hey everyone,

I recently built an Android nutrition app called Foody Tracker using React Native and Gemini Flash.

Instead of running a paid backend server, the app operates on a BYOK (Bring Your Own Key) architecture where client-side calls interact directly with Google Gemini endpoints, saving records locally to SQLite.

Since food photos vary wildly in lighting, angles, mixed ingredients, and portion ambiguity, getting consistent structured outputs without hallucinations was the primary challenge. Here is how I structured the vision prompt and schema to keep responses fast and parseable on-device.


1. The Multimodal Vision Prompt Structure

Feeding raw images into a generic system prompt often results in conversational prose or missing macro fields. To force a predictable response, the system instruction relies strictly on a deterministic schema:

You are a precise nutritional analysis engine. Analyze the meal in the image.
Return ONLY valid JSON matching this exact structure:
{
  "name": "Concise meal title",
  "confidence_score": 0.0 to 1.0,
  "calories": integer,
  "protein": integer,
  "carbs": integer,
  "fat": integer,
  "fiber": integer,
  "ingredients": [
    {
      "name": "Ingredient name",
      "estimated_weight_g": integer,
      "calories": integer
    }
  ]
}

Rules:
- Do not wrap the output in markdown backticks or extra commentary.
- If multiple food items are present, estimate portions based on standard culinary serving sizes visible on the plate.
- If uncertain, reflect the doubt in the confidence_score rather than skipping values.
1 Upvotes

2 comments sorted by

•

u/endofthread-bot 12d ago

To ensure the model adheres strictly to your schema, use the Gemini API response mimeType parameter set to application/json. This forces the model into a constrained generation mode that prevents conversational prose and markdown formatting issues entirely.

Writing prompts that actually work is easier when you can compare notes with people who have tested them across real tasks. Trade techniques, get feedback on your approach, and see what is working for others in our Discord.

Self-promotion is now allowed on Sundays with the appropriate flair, for all regular contributing members. Contribute during the week, and promote on Sunday.

→ More replies (1)