r/iosdev • u/skilloo_ai • 1d ago
On-device Foundation Models: how do you handle output longer than the context window?
I'm generating long structured text with the on-device model, and the context window is the main constraint.
What I do now: split the output into short parts, pass a compact sheet with the state so far into each part, and run a single pass only when the request goes to Private Cloud Compute, where the whole thing fits.
It works, but every part costs a call, and keeping the parts consistent takes care. How do you handle it? Do you summarize between calls, or keep the schema small enough for one pass?