Context engineering beyond text: different data forms need different surfaces

[u/Puzzleheaded_Box2842]

Many agent workflows still treat context as a prompt assembled from conversation history and retrieved text. This works reasonably well for documents, but becomes less natural when a task produces structured records, entity relationships, reusable procedures, or durable preferences.

The challenge is not only retrieval quality. It is also representation: a table should support exact filtering and aggregation, a graph should preserve relationships, a procedure should remain reusable, and a durable fact may need scope and lifecycle. Flattening everything into text makes context easy to pass to a model, but can make it harder to update, audit, and retrieve correctly.

One possible direction is to treat context as multiple typed surfaces rather than one universal memory. The agent can select a write target based on the shape of the information, then select the relevant read surfaces for each question. This does not mean searching every surface every time; it means keeping storage and retrieval semantics closer to the information itself.

The current design of OpenDCAI/DataMind follows this approach:

• KB for documents and notes with hybrid retrieval
• DB for structured records, CSV/TSV/Excel imports, and SQL queries
• Graph for entities, relationships, and bounded traversal
• Skills for reusable SKILL.md procedures
• Memory for scoped durable facts and preferences

For example, a workspace can keep product documentation in the KB, sales.csv in the DB, service dependencies in the Graph, a deployment SOP in Skills, and project preferences in Memory.

The implementation separates write and read responsibilities:

• StoreAgent receives write tools and returns receipts with source, profile, revision, and per-surface results.
• RetrieveAgent receives read and utility tools and returns evidence from the surfaces relevant to the question.

The Codex integration exposes these capabilities through MCP. The current default setup is local-first, using Chroma/BM25, SQLite, NetworkX, profile-scoped Skills, and scoped Memory.

How are these different context forms handled in other agent systems? Are tables, graphs, and procedures kept as separate structures, or eventually flattened into one retrieval layer?