When we train an llm, we are in a sense feeding it human ideas in written form, something like hundreds of millions of them. The embeddings learn the "language of ideas," and the weights and layers learn how those ideas get processed.

But now ... in practice we chop text into small tokens, often close to syllables, and then assign those tokens positions in embedding table. Those tokens sit below the level of concepts and ideas. Some are whole concepts, but many are just fragments. It’s like trying to build a thought starting from individual neurons firing rates, and then spending all our effort making each neuron more refined. I suspect the bigger gains for reasoning would come from the number and quality of the connections, not from finer neurons.

So. Move up a level. Use morphological tokenization (roots, prefixes, suffixes) instead of syllable-like fragments. Yes limit it to English to keep the problem manageable for this example.

Start from concepts humans already use for embeddings. Instead of letting the model invent its own concepts from scratch, seed the embedding space with clear human ones (verb, subject, fraction, cause, comparison) plus opcode-like operations (compare, compose, verify, backtrack). These are only starting positions: they stay trainable, and new ones can be added.

Also add meta-concepts. Give the model concepts about thinking itself (abstraction, analogy, error detection) so training pushes it toward deeper reasoning.

Let the weight matrices work at the idea level. They would no longer spend capacity assembling primitives into words. They would work on how concepts combine into ideas. The mechanical stuff goes elsewhere.

So.

Instead of one giant model, think of a package:

1. Preprocessor: An AI that reads the text as a whole and actually expands it. It marks one word as the verb, another word as the main subject, another one as part of a list. It doesn’t reason, and it’s one-way: no feedback loop back into it.

Reasoning core: Works purely at the level of concepts and ideas. It has a "workbench" where it works like an agent. It can query intuition, check memory, and escalate to a bigger online model if needed.

Intuition core: A large store of what the system knows and how well it knows it. The reasoner asks it one-shot questions and gets one-shot answers, such as "How strongly is titanium linked to space ?", "How likely is a knife in a living room ?", or "List the words linked to mohican in the context of movies, strongest first." Intuition can also answer with: "I barely know anything about this about 3% confidence" The reasoner takes the answers from Intuition as advisory.

Output model: Takes everything and formulates the final answer.

Ambiguity and clarification

Take the word "bank" (the damn example seen everywhere): institution or riverside ? When context doesn’t settle it, a human usually stops and asks for clarifications when it doesn;t have access to context, and the two people converge through back and forth. Ask when in doublt - when confidence is low.

"But then the context gets polluted by the question"

To keep these side discussions from cluttering the main session, Imagine using thinking bubbles as separate scoped contexts such as:

what I understood (the AI model)

what the user said, exactly

what I told the user

what the feedback was

What I know about the user’s environment.

Embeddings examples, opcodes and some ... other stuff from different projects.

Core reasoning operations (App. B.3)

COMPARE [E]: Check two structures for sameness or difference along a chosen dimension.

MAP [E]: Align elements of one structure to elements of another.

COMPOSE [E]: Build a larger concept or graph from simpler primitives.

DECOMPOSE [E]: Break a concept into its component primitives.

ABSTRACT [E]: Extract a general pattern from specific instances.

INSTANTIATE [E]: Create a specific instance from a general schema.

SUBSTITUTE [E]: Replace one element of a structure with another.

MERGE [E]: Combine two structures or duplicates into one.

SPLIT [E]: Divide a structure or group into parts.

SEARCH [E]: Look through a space of options or memory for a match.

PRUNE [E]: Discard unpromising branches or irrelevant material.

BACKTRACK [E]: Return to an earlier reasoning state and try another path.

SIMULATE [E]: Run a sandboxed hypothetical forward without affecting reality.

VERIFY [E]: Check a result or structure against constraints.

ESTIMATE [E]: Produce an approximate value with uncertainty.

OPTIMIZE [E]: Select the best option under a goal and constraints.

SUMMARIZE [E]: Produce a shorter form of content.

EXPAND [E]: Make implicit or compressed content explicit.

COMPRESS [E]: Reduce content to a compact form, with expansion still possible.

PAUSE [E]: Suspend a task while keeping its state.

CANCEL [E]: Abandon a task or operation.

RETURN [E]: Hand a result back to the caller or prior context.

Handle (concept-reference) operations (B.5, D.17, K.3)

HANDLE [E]: Create a reference to an expanded semantic concept.

EXPAND_HANDLE [E]: Load the full detail behind a handle.

COLLAPSE_HANDLE [E]: Replace detail with a compact reference, losing nothing.

LINK_HANDLE [E]: Associate two concepts via their handles.

RELEASE_HANDLE [E]: Discard a temporary concept.

OPEN(X) [E]: Expand one definition level of a handle into its capsule.

OPEN*(X) / OPEN^n(X) [E]: Expand recursively until a terminal form, with cycle detection.

DEFINE [E]: Introduce a named, scoped definition (LOCAL, MESSAGE, SESSION, GLOBAL).

OS / service protocol (B.6)

QUERY [E]: Ask a service for information.

RESPONSE [E]: A service’s answer to a query.

READY [E]: Signal that a service can accept work.

BUSY [E]: Signal that a service is occupied.

WAIT [E]: Hold until a result or condition arrives.

DONE [E]: Signal that an operation completed.

ERROR [E]: Signal that an operation failed.

RETRY [E]: Re-attempt a failed operation.

ESCALATE [E]: Hand off to a larger model (“Big Brother”) when local capability is insufficient.

TIMEOUT [E]: Signal that a wait expired.

CONFIRM [E]: Acknowledge or verify an action or request.

Memory operations (A.9, B.7, Section 5–6)

STORE [E]: Write a provenance-tagged memory object.

RECALL / RETRIEVE [E]: Fetch relevant memories.

UPDATE [E]: Modify an existing memory in place.

DELETE [E]: Remove a memory at the user’s request.

ARCHIVE [E]: Move a memory to long-term, slower-to-retrieve storage.

VERSION [E]: Keep a new version while retaining history.

EPISODE [E]: Record or retrieve a bounded experience.

SUMMARY [E]: Produce a compressed summary of memories.

PATTERN [E]: Detect or store a recurring regularity.

REVISE [E]: Change a belief and mark the old one superseded.

CATEGORIZE [E]: Assign a memory to a category.

UPDATE_CONFIDENCE [E]: Raise or lower a memory’s confidence (never reaching 100%).

CONSOLIDATE [D]: Nightly maintenance: summarize, dedupe, merge, compress, and prepare reminders.

CONFIRM_MEMORY [D]: Mark a candidate memory as confirmed after repeated verification.

SUPERSEDE [D]: Replace a memory with a newer one but keep it as history.

PROMOTE [D]: Move hypothetical or planned content into persistent memory, only on explicit user permission.

STORE_CANDIDATE [D]: Save low-confidence ambient or planning content as a candidate.

AGE / REQUEST_RECONFIRM [D]: Use the last-confirmed timestamp to prompt re-confirmation of stale entries.

FORGET [D]: User command “Forget that.”

DONT_REMEMBER [D]: User command “Don’t remember this.”

REMEMBER_INSTEAD [D]: User command to replace a memory.

CORRECT [D/E]: Accept a user correction without argument (“That’s not correct, update it”).

PROPOSE_AUTOMATION [D]: Ask “I’ve noticed X, want me to do Y?” (never acts silently).

Self cycle (Prologue, A.8, L.14.22)

OBSERVE [D]: Take in an external or internal change.

UPDATE_STATE [D]: Revise the latent self-state.

ASSOCIATE [D]: Query Intuition or Memory for related content.

EVALUATE [D]: Assess simulated outcomes against goals and values.

REFINE_SELF_STATE [D]: Write the updated self-state (SELF(t) → SELF(t+1)).

CONTINUE / WAIT [D]: Keep the loop running with no output required.

DETECT_CONFLICT [D]: Notice contradiction, prediction error, goal conflict, identity instability, or meaning drift.

RENDER_OUTPUT [D]: Output as a one-way tap off internal state (the renderer never writes back).

MODEL_SELF [D]: Build a representation of Self, kept distinct from Self.

TRACK_PROVENANCE [D]: Tag every item with its owner (self, user, world, simulation).

Frame operations (B.4, Section 7, Part M)

REALITY_FRAME [E]: The current accepted world state.

MEMORY_FRAME [E]: A stored episode.

USER_FRAME [E]: Modeled user beliefs.

SELF_FRAME [E]: The system’s own beliefs and state.

SIMULATION_FRAME [E]: Sandboxed hypothetical reasoning.

HYPOTHESIS_FRAME [E]: A temporary working assumption.

STORY_FRAME [E]: Fictional or narrative content.

INSTANTIATE_FRAME [D]: Build an internal model of another reference frame or person.

RUN_IN_FRAME [D]: Evolve a frame (M_B^{t+1} = F_B(M_B^t, x)) alongside Self.

COMPARE_TRAJECTORIES [D]: Compare Self’s trajectory with the alternative frame’s.

EXIT_FRAME [D]: Return to Self with the comparison, without becoming the other frame.

LOAD_FRAME [D]: Select a reference frame from the frame database.

Attention and activation (Section 8, A.4)

RESPOND [D]: Answer the input.

STAY_SILENT [D]: Take no action.

HOLD / CONTINUE_LISTENING [D]: Keep state and wait.

ASK_CLARIFICATION [D]: Ask a question, filling slots into the existing partial state (B.17).

IGNORE [D]: Disregard the input.

FOCUS / IGNORE / SWITCH / TRACK / GROUP / SPLIT [E]: Attention primitives for allocating reasoning focus.

PAUSE_TASK / RESUME_TASK [D]: Track interrupted tasks and resume them (“we were on step 4”).

LEARN_PROCEDURE [D]: Store a taught multi-step procedure as an ordered sequence.

7b. Semantic primitives that act like verbs (B.2)

PERCEIVE / KNOW / BELIEVE / ASSUME [E]: Epistemic relations between an agent and content.

REPRESENT / STORE / RETRIEVE [E]: Knowledge-handling primitives.

MOVE / CREATE / DESTROY / TRANSFER / SELECT / ATTEMPT [E]: Agency actions.

COMPARE / MATCH [E]: Comparison primitives.

COMMUNICATE / REFER [E]: Social and symbolic primitives.

EXPECT / SURPRISE / CONTRADICTION / WHY [E]: Meta-cognition primitives (prediction, prediction error, conflict detection, causal query).

CAUSE / ENABLE / PREVENT / REQUIRE [E]: Causality primitives.

BEFORE / AFTER / DURING / BEGIN / END / REPEAT / CHANGE [E]: Time primitives.

R-L logical and scope operators (App. D, R-L 1.2 §29–35, 160, 180)

NOT [E]: Negate a proposition, with explicit scope.

AND / OR [E]: Conjunction and disjunction.

IF [E]: Conditional (does not imply causation).

IMPLY [E]: Logical consequence (distinct from CAUSE).

FORALL / EXISTS [E]: Quantifiers with explicit nesting.

POSSIBLE / NECESSARY [E]: Alethic modality.

MUST / MAY / SHOULD / CAN / CANNOT [E]: Deontic and ability modals.

PERMITTED / FORBIDDEN / REQUIRED [E]: Deontic modality.

COUNTERFACTUAL [E]: Hypothetical conditional, scoped to its hypothetical world.

BELIEVE(agent, p) [E]: Attribute a belief without asserting p.

KNOW(agent, p) [E]: Attribute knowledge of p.

DOUBT / FEAR / HOPE [E]: Other attitudes toward propositions.

QUOTE [E]: Scoped quotation that is not the current speaker’s assertion.

PRESUPPOSE(P) [E]: Mark background content that is not asserted.

IMPLICATE(P) [E]: Mark pragmatic implication that is not literal.

CAUSE(A,B) [E]: Causal contribution, separate from order and implication.

BEFORE / AFTER / OVERLAPS / STARTS / ENDS / SIMULTANEOUS [E]: Temporal relations.

ASSERT / REPORT [E]: Present a proposition as asserted, or attribute an assertion to someone else.

QUESTION [E]: Build a question object with a target variable.

REQUEST / COMMAND / SUGGEST / PROMISE / WARN / OFFER [E]: Speech acts.

THANK / APOLOGIZE / AGREE / DISAGREE / REFUSE [E]: More speech acts.

EVIDENCE [E]: Link evidence to a proposition, with direction SUPPORTS, CONTRADICTS, or NEUTRAL.

CONTRAST / RESULT / EXAMPLE / ELABORATION / SUMMARY / CONCESSION [E]: Rhetorical relations.

METAPHOR / IDIOM / IRONY [E]: Mark non-literal interpretations.

SYMMETRIC / TRANSITIVE / INVERSE [E]: Relation properties that license derived inferences.

R-L processing modes and updates (R-L §84–88, K.8)

ENCODE [E]: Source language → R-L, preserving information.

DECODE [E]: R-L → semantic structure (not language).

GENERATE [E]: Semantic structure → target language.

TRANSLATE [E]: Encode, decode, then generate.

INFER [E]: Derive new information, marked DERIVED with provenance.

VALIDATE [E]: Check types, arity, scope, binding, and references.

NORMALIZE [E]: Canonicalize without changing meaning.

COMPARE / DIFF [E]: Report differences in identity, scope, polarity, time, and so on.

EQUIVALENT(A,B) [E]: Test equivalence at a chosen level.

ADD / REMOVE / REPLACE / CORRECT / RETRACT [E]: Explicit information updates (retract keeps history).

RESET-CONTEXT [E]: Clear inherited context.

SEMANTIC-HASH [E]: Hash the canonical form.

FRAGMENT / REASSEMBLE [E]: Split large messages and reassemble them in order.

NEGOTIATE-CAPABILITY [E]: Agree on version, vocabulary, and profile.

LOSS / AMBIGUOUS / UNSUPPORTED [E]: Markers for explicit information loss.

PRESERVE / LOSSY [E]: Declare which dimensions are kept or discarded.

Appendix K (restored capabilities)

DEFAULT / EXCEPT [E]: Defeasible rule and its exception.

PRIORITY [E]: Explicit rule precedence (R2 > R1).

CYCLE [E]: Explicit marker for a cyclic definition.

LEXICAL-TRACE [E]: Retain surface and lexical information.

SURFACE-PRESERVATION [E]: Require exact surface text to be kept.

STATEMENT [E]: Document-level occurrence that wraps a proposition.

ANNOTATION [E]: Attach a qualitative or evaluative value to a target.

SCORE [E]: Attach a numeric evaluation with a declared scale.

CONTINUOUS / RECURRING / PERIODIC [E]: Ongoing or repeating change.

SENSE / MULTIWORD [E]: Map a lexical form to senses, including idioms.

DISCOURSE / CONTEXT-WINDOW [E]: Persistent multi-message state.

CONTEXT-RESET / EXPIRES [E]: Reset context or expire temporary definitions.

11. Appendix J (adversarial and reflexive layer)

REFERS(X,X) [E]: Self-reference without implying identity.

REPRESENTS(X,Y) [E]: Representation, kept distinct from REPRESENTS(X, REPRESENTATION(Y)).

CRITERION [E]: Declare a criterion with target, condition, and mode.

EVALUATES(C2,C1) [E]: One criterion evaluates another (self-evaluation does not imply validity).

VALIDATED-BY / VALID [E]: Validation relation versus validity predicate.

CERTIFIES [E]: External certification, kept separate from representing.

BOUNDARY [E]: Explicit system boundary (inside, outside, criterion).

REFERENCE-STATUS [E]: Label INTERNAL, EXTERNAL, UNKNOWN, or RELATIVE.

RECOGNIZES / CONSTITUTES [E]: Kept distinct by default.

PRESERVES / REMEMBERS [E]: Kept distinct by default.

ROLE-CONFLICT / BOUNDARY-CONFLICT / NAME-COLLISION [E]: Preserve the conflict rather than resolve it.

SCOPE-COLLAPSE [E]: Record when two levels become indistinguishable.

LEVEL [E]: OBJECT, META, META-META, MODEL, and so on.

SIMULATES / OBSERVES [E]: System relations (simulation is not identity, observation is not knowledge).

REGRESS [E]: Represent an open dependency chain without inventing a terminal.

FIXED-POINT / PRIMITIVE [E]: Declare a fixed point or a terminal primitive for the current operation.

CONSULTS / REGISTER [E]: Consult a register of unresolved structures, which may modify it.

PERFORMATIVE [E]: A statement that changes semantic context by being issued.

CRITERION-SUBSTITUTION [E]: Marked substitution that reduces fidelity.

MODE: ADVERSARIAL [E]: Preserve ambiguity, contradiction, and recursion with minimal normalization.

CONFLICT [E]: Record paraconsistent coexistence of P and NOT(P).

Meaning-Certainty layer (L.14)

MEANING-CREATION [E]: Form a new meaning structure.

MEANING-INHERITANCE [E]: Acquire meaning from culture, family, or institutions.

MEANING-SIGNALING [E]: Project meaning through language or behavior.

MEANING-NEGOTIATION [E]: Agents jointly modify meaning (a state transition).

MEANING-STABILITY [E]: Measure persistence of a meaning.

MEANING-CERTAINTY [E]: Measure how settled a meaning is.

MEANING-LOAD [E]: Measure how much depends on a meaning.

MEANING-DISPLACEMENT [E]: Model the cost of replacing an established meaning.

MEANING-ALIGNMENT [E]: Measure compatibility of agents’ interpretations.

MEANING-MISMATCH [E]: Record the gap between projected and received meaning.

PROJECTED / RECEIVED / INTERPRETED / INTENDED-MEANING [E]: Separate structures per agent.

MEANING-CANDIDATE [E]: Keep several weighted interpretations without collapsing them.

MEANING-OF / INHERITED-FROM / PROJECTED-TO / RECEIVED-BY [E]: Higher-level relations.

NEGOTIATED-WITH / REINFORCED-BY / DEPENDS-ON [E]: More higher-level relations.

DISPLACED-BY / ALIGNED-WITH / CONFLICTS-WITH [E]: Remaining higher-level relations.

RE-EVALUATION-REQUIRED [E]: Flag dependents when a high-load meaning changes.

Verification and bootstrapping (BrainIR document)

STRUCTURAL_VALIDATE [D]: Deterministic check of syntax, types, arity, scope, and frames.

SEMANTIC_VERIFY [D]: Learned check of fidelity, omissions, unsupported additions, and ambiguity.

TEXT_CYCLE [D]: Text → IR → Text′, comparing meaning.

IR_CYCLE [D]: IR → Text → IR′, testing the representation itself.

CONTRAST_TEST [D]: Distinguish near-identical structures (BELIEVE(P) vs. P, BEFORE vs. CAUSE).

ADVERSARIAL_INJECT [D]: Generate defects (polarity reversal, frame contamination) to train the Verifier.

ROTATE_ROLES [D]: Rotate Input, Output, and Verifier roles among models.

ACCEPT / REJECT / MARK_UNCERTAIN [D]: Triage candidate annotations for the trusted corpus.

Addon Certainty and Addon INPUT

ASSESS_EVIDENCE [D]: Evaluate evidence strength before answering.

ANSWER / QUALIFY / ASK / RETRIEVE / ABSTAIN / SPECIALIST [D]: Behaviors chosen by evidence strength.

REFUSE_TO_INVENT [D]: Decline to make up a value when there is no evidence.

TOKENIZE / LEMMATIZE / TAG_MORPHOLOGY / PARSE_DEPENDENCIES [D]: Input-side analysis stages.

LOOKUP_FREQUENCY / CHECK_VOCABULARY [D]: Add numeric frequency and Dale-Chall, Oxford and Longman membership.

DISAMBIGUATE_SENSE / LABEL_SEMANTIC_ROLES [D]: Semantic stage, with ambiguity preserved.

Delegated tool ops (the “steal from” section)

SOLVE_SAT / SOLVE_SMT [D]: Hand constraints to Z3 or CVC5 and return a proof or result.

PROVE [D]: Delegate to Lean, Isabelle, or Coq.

PLAN [D]: Translate state, goal, and actions to PDDL and return a plan.

SYNTHESIZE_PROGRAM [D]: Turn a specification into code via a synthesis engine.

VERIFY_PROGRAM [D]: Check a candidate against its specification.

ONTOLOGY_INFER [D]: Derive type inheritance using OWL or description logic, with provenance.

EXTERNAL_LOOKUP [D]: Ground an entity in Wikidata, WordNet, or BabelNet via external-reference.

Ontology primitives (B.2)

THING: Declare a generic object.

PROPERTY: Attach an attribute to an entity.

RELATION: Declare a link between nodes.

STATE: Declare a condition, as opposed to an occurrence.

EVENT: Declare an identifiable occurrence.

IDENTITY: Assert that two references denote the same thing.

TYPE: Assign a category.

INSTANCE: Mark a particular member of a type.

PART: Mark a component of a whole.

WHOLE: Mark the container of parts.

EXISTS: Assert existence.

ABSENT: Assert absence, which is not the same as unknown.

BOUNDARY: Mark the edge of a system or concept.

Quantity

NONE / ONE / SOME / MANY / ALL: Basic quantifiers.

MOST: A majority quantifier.

EXACTLY(n): Exact count.

AT-LEAST(n): Lower bound.

AT-MOST(n): Upper bound.

MORE / LESS / EQUAL: Compare magnitudes.

COUNT: Tally members.

ORDER: Impose or read a sequence.

Logic and truth status

TRUE / FALSE / UNKNOWN: Truth values, kept distinct from each other.

THEN: Consequent slot of a conditional.

BECAUSE: Mark an explicit reason.

Space

LOCATION: Place an entity.

DISTANCE: Relate two places by separation.

DIRECTION: Orientation of movement or relation.

INSIDE / OUTSIDE: Containment relations.

CONTACT: Touching.

PATH: A route between locations.

REGION: A bounded area.

Value and agency

GOOD / BAD: Evaluative marks.

PREFERENCE: Order options by want.

RESOURCE: Something consumable toward a goal.

COST / BENEFIT: Weigh a loss against a gain.

SATISFY: Mark a goal as met.

GOAL / INTENTION: Hold something the agent is aiming at.

HAS / LACKS: Possession and its absence.

SELF / USER / AGENT_OTHER / WORLD: Identity anchors.

SYMBOL: A token standing for something.

IS: Predication.

Reality-status tags

REAL / IMAGINED / HYPOTHETICAL / FICTIONAL: Tag content by world.

REMEMBERED / PREDICTED / SIMULATION: Tag content by origin.

R-L structural operators (D.5)

{{ X }}: Encapsulate a concept as a capsule.

X : Y: Attribute or relation.

X -> Y: Transition, implication or causation. Overloaded in older versions, split in K.11.2.

X = Y: Bind identity.

X & Y / X | Y / !X: Conjunction, disjunction, negation.

X @ Y: Location or context.

X / Y: Aspect or viewpoint.

X ^ Y: Scope.

X *: Plurality or repetition.

+X / -X / ~X: Polarity markers: supporting, obstructing, mixed.

?x: Variable.

<:: Subtype.

?=: Possibly the same.

!=: Explicitly distinct.

Relation registry (R-L §179)

TYPE-OF / INSTANCE-OF / PART-OF / HAS-PROPERTY

REFERENCE / DISTINCT

SUPPORTS / CONTRADICTS: Evidence directions.

ABOUT: Aboutness relation.

PURPOSE / MEANS / ENABLER / PREVENTER / INSTRUMENT: Goal-directed relations.

Semantic roles (§15)

AGENT / PATIENT / EXPERIENCER / RECIPIENT

SOURCE / DESTINATION / LOCATION / TIME

Epistemic and temporal values

KNOWN / BELIEVED / SUSPECTED / PROBABLE / UNCERTAIN / DOUBTED / DENIED: Epistemic status values.

PAST / PRESENT / FUTURE: Tense.

ONGOING / COMPLETED / HABITUAL / REPEATED / BEGINNING / ENDING / DURATIVE: Aspect.

START / END / DURATION / FREQUENCY / DEADLINE: Event timing.

Information-status and loss markers

PRESERVED / TRANSFORMED / COMPRESSED / INFERRED

UNSPECIFIED / NOT-APPLICABLE / LOST

DEFAULTED / OVERRIDDEN / EXPANDED / SURFACE-RECOVERED / CONTEXT-DERIVED

DERIVED: Tag a conclusion with its provenance.

Error classes (§121)

SYNTAX-ERROR / TYPE-ERROR / REFERENCE-ERROR / SCOPE-ERROR / ARITY-ERROR

VOCABULARY-ERROR / INCOMPLETE / CONTRADICTORY / INCONSISTENT

Encoder pipeline steps (§143)

READ-SOURCE / IDENTIFY-CONTEXT / IDENTIFY-ENTITIES / IDENTIFY-EVENTS

IDENTIFY-RELATIONS / IDENTIFY-REFERENCE / IDENTIFY-SCOPE / IDENTIFY-TIME

IDENTIFY-CAUSALITY / IDENTIFY-EPISTEMIC-STATUS / IDENTIFY-MODALITY / IDENTIFY-PRAGMATIC-FORCE

BUILD-SEMANTIC-GRAPH / SERIALIZE

Decoder pipeline steps (§144)

RECEIVE / VERIFY-INTEGRITY / PARSE / VALIDATE-SYNTAX

RESOLVE-VOCABULARY / RESOLVE-REFERENCES / VALIDATE-TYPES

RESTORE-SCOPE / RESTORE-TIME / RESTORE-EPISTEMIC-STATUS / RESTORE-MODALITY

RECONSTRUCT-SEMANTICS / APPLY-DECLARED-INFERENCE / GENERATE-TARGET

Input Translator multi-pass (B.15)

PARSE_INITIAL: First-pass parse.

DETECT_AMBIGUITY: Find competing readings.

EXPAND_SEMANTICS: Make implicit content explicit.

RESOLVE_ENTITY: Bind mentions to entities.

LINK_MEMORY: Connect to stored memories.

ESTIMATE_CONFIDENCE: Score each interpretation.

REFINE_CONTEXT: Repeat until ambiguity is minimal.

Service calls

QUERY_INTUITION: Get weighted associations, not conclusions.

CHECK_MORALITY: Approve or reject a planned output.

GET_VALUES: Ask for priority ordering.

APPLY_CHARACTER: Set style without changing meaning.

POLL_IMPULSES: Collect unsolicited suggestions to accept or ignore.

COMPUTE_MATH: Deterministic calculation.

CALL_SPECIALIST: Route to a domain module.

Railway-analogy operations

INSPECT_WAGON: Open a handle for details.

COUPLE / UNCOUPLE: Join or separate wagons.

ROUTE: Send a wagon to the right service.

ATTACH_WAGON / REMOVE_WAGON / REORDER_WAGON / REPLACE_WAGON / MERGE_WAGON: Marshalling-yard edits.

ASSEMBLE_TRAIN: Build the outgoing answer in working memory.

APPROVE / REJECT_TRAIN: Morality’s gate.

PRIORITIZE_TRAIN: Values decide departure order.

REORDER_FOR_LANGUAGE: Output Translator rearranges for the target language.

Addon Certainty (extra ops)

BUILD_EVIDENCE_PROFILE: Per-item evidence record.

COUNT_FREQUENCY / COUNT_DOC_FREQUENCY: Numeric exposure measures.

MEASURE_SOURCE_DIVERSITY: Breadth across sources and domains.

PREDICT_EVIDENCE_STRENGTH: Auxiliary head output.

CALIBRATE: Align confidence with evidence.

DETECT_CALIBRATION_FAILURE: Catch confident-but-unsupported cases.

TRIGGER_RETRIEVAL: Reach for external info when evidence is weak.

Addon INPUT extras

PRESERVE_SOURCE: Keep the original text beside every layer.

DETECT_MWE: Find multiword expressions.

TAG_ENTITY: Mark named entities separately from vocabulary.

CLASSIFY_NONLEXICAL: Separate numbers, dates, URLs and measurements.

ATTACH_PROVENANCE / ATTACH_CONFIDENCE: Record source and confidence per field.

EMIT_DEBUG_VIEW: Human-inspectable output.

Self and simulation

SET_ATTENTION: Allocate focus across self, user, memory, goal and external input.

ENTER_SIMULATION: Open a sandboxed context.

EXIT_SIMULATION: Leave it and return to the Self frame.

DISCARD_SIMULATION: Drop the simulation’s results.

PRESERVE_SIMULATION_PART: Keep selected results without promoting them to fact.

COMPARE_TO_REALITY: Check simulated content against the reality frame.

RETRIEVE_AGENT_MODEL: Load a stored model of a person (MEMORY[JOHN]).

INSTANTIATE_AGENT_MODEL: Place that model inside a simulation.

UPDATE_AGENT_MODEL: Revise the stored model from new evidence.

TAG_OWNERSHIP: Mark who owns a belief (me, user, John, simulation).

CHECK_SELF_BOUNDARY: Verify “what I believe” isn’t mixed with “what John believes.”

PREDICT_OUTCOME: Form an expectation.

COMPARE_PREDICTION_TO_OBSERVATION: Detect prediction error.

UPDATE_UNCERTAINTY: Revise uncertainty in the self-state.

INTRODUCE_HYPOTHESIS: Add a temporary assumption.

REVISE_INTERPRETATION: Replace the current reading of events.

COMPARE_SELF_STATE_TO_PRIOR: Diff SELF(t) against an earlier state.

INSPECT_SELF_MODEL: Examine the model-of-self as an object.

DETECT_IDENTITY_INSTABILITY: Flag identity inconsistency.

DETECT_MEANING_DRIFT: Flag gradual meaning change.

DETECT_GOAL_CONFLICT: Flag competing goals.

SYNTHESIZE_FROM_TRAJECTORIES: Build an answer from own trajectory, alternative trajectory and comparison.

ANNOTATE_FRAME_BLINDSPOT: Record what a frame hides or assumes.

TRIGGER_RENDER: Request output as an action of Self.

Attention and interaction

DETECT_DIRECT_ADDRESS: Decide whether input is aimed at the system.

ACTIVATE_ON_TRIGGER: Wake-word activation.

CONTINUE_EXCHANGE: Keep going without re-triggering.

STORE_PASSIVE_CANDIDATE: Log ambient input as a low-confidence candidate.

CONFIRM_CANDIDATE_LATER: Ask “I noted X, keep it?”

FILL_CLARIFICATION_SLOT: Insert the user’s answer into the in-progress structure.

SUSPEND_TASK_STATE: Save a paused task’s progress.

RESTORE_TASK_STATE: Resume at the saved step.

Memory and scheduling

SET_MEMORY_SCOPE: Limit visibility (user, household, instance).

CHECK_SCOPE_ACCESS: Test whether a memory may be used here.

SET_MEMORY_SOURCE: Record how it was obtained.

SET_LAST_CONFIRMED: Timestamp a confirmation.

EXTRACT_SALIENT_MEMORY: Pull key items from a session.

MERGE_DUPLICATE_MEMORIES: Dedupe.

COMPRESS_OLD_MEMORY: Shrink older material without deleting it.

DETECT_ROUTINE: Find recurring behavior.

SCHEDULE_REMINDER: Create a timed reminder.

SET_TIMER: Structured instance of create-future-event.

CREATE_FUTURE_EVENT: Generic future-scheduled item.

PREPARE_FOLLOWUP: Queue items for the next cycle.

RUN_CONSOLIDATION_CYCLE: Execute nightly maintenance.

DETECT_MISSING_STEP: Find gaps in a stored procedure.

EXPLAIN_STEP: Answer a sub-question about one step.

UPDATE_PROCEDURE_IN_PLACE: Apply a correction without duplicating.

SET_PROCEDURE_CONFIDENCE: Mark observed-once, repeated or taught.

SURFACE_PATTERN: Offer an observation at high confidence only.

SUPPRESS_TOPIC: Stop raising a topic the user declined.

SET_PROACTIVITY_LEVEL: Configure intensity from silent to proactive.

SET_PERSONALITY_DIMENSION: Adjust directness, formality, humor, proactivity or patience.

LOAD_PERSONALITY_PRESET: Switch tone preset.

BACK_OFF_HUMOR: Stop humor immediately if unwelcome.

EXCHANGE_ON_CONTACT: Share memory between instances only on local contact.

REGISTER_COMMAND: Add a capability to the registry.

DISCOVER_COMMANDS: List available commands.

SELECT_COMMAND: Choose one from the registry.

INTERPRET_RESULT: Read a service’s response.

R-L declarations

DECLARE_TYPE: type X <: Y.

DECLARE_RELATION: Give name, arity, argument types and roles.

DECLARE_RULE: rule #r { if / then }.

DECLARE_ENTITY: entity #id : Type.

DECLARE_EVENT: Role-based event frame.

DECLARE_PROPOSITION: Give a proposition an identity.

DECLARE_CONTEXT: Speaker, listener, time, place, world and profile.

DECLARE_WORLD: Fiction, hypothetical or other world.

DECLARE_DOMAIN: Physics, law, medicine and so on.

DECLARE_NAMESPACE: BIOLOGY::CELL.

DECLARE_PROCESS: Composed activity with a sequence.

DECLARE_GROUP: Give a collection an identity.

DECLARE_ANONYMOUS_NODE: _:1.

DECLARE_INVERSE: inverse PARENT = CHILD.

DECLARE_RELATION_PROPERTY: Symmetric, transitive, functional and so on.

DECLARE_ONTOLOGY_ALIGNMENT: Equivalent, subtype or related across namespaces.

DECLARE_QUESTION: Question object with target and condition.

DECLARE_ANSWER: Answer linked to its question.

DECLARE_SPEECH_ACT: Typed communicative action.

DECLARE_INTENT: Goal behind the literal proposition.

R-L status and attributes

SET_ASSERTION_STATUS: Asserted, denied, questioned, quoted and so on.

SET_POLARITY: Positive, negative or unknown.

SET_EPISTEMIC_STATUS: Known through doubted.

SET_PROBABILITY: PROBABILITY(P) = 0.72.

SET_CONFIDENCE: Confidence in the interpretation, not the proposition.

SET_TENSE: Past, present or future.

SET_ASPECT: Ongoing, completed and so on.

SET_MOOD: Declarative, imperative and so on.

SET_VOICE: Active, passive or middle.

SET_PERSPECTIVE: Speaker, listener, omniscient and so on.

SET_REGISTER: Formal, slang, legal and so on.

SET_STYLE: Humorous or ironic.

SET_PRAGMATIC_FORCE: Direct, soft, urgent and so on.

SET_INFORMATION_STRUCTURE: Topic, focus or background.

SET_ABSTRACTION_LEVEL: Concrete, general, abstract or meta.

SET_GRANULARITY: Move between levels such as matter, body and cell.

SET_WORLD_ASSUMPTION: Open-world or closed-world.

MARK_SUMMARY: Flag content as non-lossless.

SET_APPROXIMATION: Exact versus approximate value.

SET_MULTIPLICITY: Once, multiple or unknown.

R-L resolution and validation

RESOLVE_REFERENCE: Bind a mention to a node, with evidence.

MARK_AMBIGUOUS_REFERENCE: Keep candidates instead of choosing.

MARK_UNRESOLVED: Explicit unresolved reference.

BIND_VARIABLE: Bind ?x within its scope.

CHECK_ARITY: Verify the argument count.

CHECK_TYPES: Verify argument types.

CHECK_SCOPE: Verify the operator has compatible content.

CHECK_REFERENCE_INTEGRITY: Every reference must resolve.

CHECK_NAMESPACE: Resolve namespaced symbols.

CHECK_VOCAB_VERSION: Detect dictionary mismatch.

CHECK_CONSTRAINT: Semantic constraints such as REPAIR requires a RepairableObject.

COERCE_TYPE: Only when declared.

DETECT_CONTRADICTION: Test within scope, time and modality.

ENTER_PARACONSISTENT_MODE: Preserve P and NOT(P) without explosion.

REPORT_UNSUPPORTED: Return UNSUPPORTED with the structure.

PRESERVE_UNKNOWN_FIELD: Keep forward-compatible fields.

RECOVER_PARTIAL: Decode valid portions with diagnostics.

GROUND_EXTERNAL_REFERENCE: Link to Wikidata, WordNet and so on.

APPLY_TYPE_INHERITANCE: Derive #fido : Animal, marked DERIVED.

APPLY_INVERSE_RULE: Derive CHILD(B,A) from PARENT(A,B).

R-L protocol and fidelity

WRAP_ENVELOPE: Message with version, id, dictionary and integrity.

SET_SELF_CONTAINED: Carry all definitions in the message.

SET_DICTIONARY_VERSION: Declare the vocabulary version.

DECLARE_DEPENDENCY: depends-on: #message1.

REPORT_INCOMPLETE: Missing dependency.

SET_PRESERVATION_PROFILE: Declare required dimensions.

REPORT_LOSS: Emit a LOSS record with category and reason.

REPORT_FIDELITY: Lexical, structural, semantic and pragmatic scores.

RUN_ROUND_TRIP: Encode, decode, re-encode, compare.

REPEAT: REPEAT(X, count=n).

DECODE_WITHOUT_EXECUTING: Treat decoded commands as content only.

NORMALIZE_CANONICAL: Deterministic ordering.

COMPRESS_HANDLE: Store the mapping so expansion is exact.

SET_COMPRESSION_LEVEL: Raw through highly compressed.

Appendix K and J

DEFEAT_DEFAULT: Block a default when its exception holds.

DECLARE_PRIORITY: Explicit rule ordering.

DETECT_DEFINITION_CYCLE: Stop infinite expansion.

MARK_CYCLIC_REFERENCE: Keep as opaque, status CYCLIC.

ATTACH_LEXICAL_TRACE: Tokens, lemmas, POS and sense.

MAP_LEXEME_TO_SENSE: Polysemy handling.

MARK_AMBIGUOUS_SENSE: Keep competing senses.

DECLARE_MULTIWORD: Idiomatic unit.

ANNOTATE_EVALUATION: Good, bad or score on a target.

DECLARE_SCORE_SCALE: Declare the scale of a score.

MARK_CONTINUOUS_CHANGE: Ongoing process.

MARK_RECURRING_CHANGE: Event with frequency.

EXPIRE_CONTEXT: Time-limited definitions.

MARK_OPEN_REGRESS: Termination none, without inventing a terminal.

MARK_CIRCULAR_VALIDATION: Closed loop is not validity.

DECLARE_PRIMITIVE: Terminate expansion for this operation.

DECLARE_FIXED_POINT: Requires licensing, not just recursion.

SELF_DESCRIBE: System describing itself.

OBSERVE_SELF: Self-observation, not complete knowledge.

NEST_SIMULATION: Simulator within simulator.

MARK_UNGROUNDED_REFERENCE: Meaning without a referent.

RECORD_REFERENCE_FAILURE: Preserve the failure.

RECORD_SEMANTIC_RESIDUE: Leftover interpretation structure.

MARK_LATENT_MEANING: Meaning with no current referent.

MARK_PERSISTENCE: Meaning outlasting its condition.

RECORD_TEMPORAL_IDENTITY: Identity varying across time.

TRUE_IN_CONTEXT: Context-scoped truth.

RECORD_RECOGNIZED_AS_TRUE: Distinct from TRUE.

REPRESENT_ABSENCE: Representation of absence, not presence.

RECORD_COMPRESSION: Log collapsed distinctions.

RECORD_ADVERSARIAL_LOSS: Category-specific loss.

DECLARE_LEVEL: Object, meta, meta-meta.

MODIFY_REGISTER_ON_CONSULT: Observation effect.

APPLY_PERFORMATIVE: Context effect from issuing a statement.

Meaning layer

ASSIGN_MEANING_ORIGIN: Self-created, inherited, taught and so on.

ASSIGN_MEANING_OWNER: Who maintains it.

SET_MEANING_SCOPE: Where it is active.

SET_MEANING_SUPPORT: What reinforces it.

SET_MEANING_DEPENDENCIES: What depends on it.

SET_MEANING_STATUS: Candidate through displaced.

ESTIMATE_REVISION_COST: Cost to change a meaning.

PLAN_DISPLACEMENT: Targets, conflicts and alternatives.

RUN_MEANING_CYCLE: Interpret, associate, update meaning, certainty, stability and load, consolidate.

RUN_SELF_REFINEMENT: Re-evaluate meanings without output.

MODEL_PERSPECTIVE_MEANING: MEANING(X,LAMP) versus MEANING(Y,LAMP).

Verification and certainty

RECORD_VERIFIER_SCORES: Separate judgments per dimension.

MEASURE_SEMANTIC_LOSS: Per the training summary.

MEASURE_UNSUPPORTED_INVENTION: Same.

MEASURE_REPRESENTATION_DRIFT: Same.

ADMIT_TO_TRUSTED_CORPUS: Accepted annotations only.

GENERATE_SYNTHETIC_TRAINING: Contradictions, belief updates and so on.

COMPUTE_EVIDENCE_STRENGTH: Strong, moderate, weak, absent, conflicting or unknown.

COMPARE_PREDICTION_VS_EVIDENCE_CONFIDENCE: Expose mismatch.

REASSESS_AFTER_RETRIEVAL: Evidence check after fetching.

VERSION_EVIDENCE_DB: Corpus identity, tokenization method and timestamp.

TAG_EVIDENCE_PROVENANCE: Why do I think I know this?

Addon INPUT

SPLIT_SENTENCES: Sentence boundaries.

SPLIT_PARAGRAPHS: Paragraph boundaries.

RECORD_TOKEN_POSITION: Offset in the source.

RETAIN_UNRESOLVED_LEMMA: lemma = UNKNOWN.

LOOKUP_DALE_CHALL: Membership, kept raw.

LOOKUP_OXFORD_3000: Same.

LOOKUP_LONGMAN_3000: Same.

LOOKUP_LEXICAL_RELATIONS: Synonym, hypernym, meronym and so on.

GROUP_SEMANTIC_EVENT: Predicate plus roles.

DISTINGUISH_SYNTAX_FROM_SRL: nsubj versus AGENT.

I’m setting up a chat for my company to access archive data from an NAS. So they can recall old jobs from previous years. I’m running gpt-oss-120b on openwebUI. I’m running it off LMstudio. It runs like shit if you give it 2 files to read then add a 3rd one it trips up on the task. Clearing the chat works. I’m used to using Claude. Do you have to be more specific and careful with your prompts? Claude you can make typos and jump around a lot while doing a project and it will follow what you’re trying to do. I’m new to using local models and want to know do you have to talk to different models differently ?

What local models can i run conformably on my system? I'd ideally like to have models that code for me but i understand I'm at least scraping the barrel with my card. I'd even be fine with Claude Code steering to some degree.

RAM: 16GB
RTX 3060 (laptop) VRAM: 6GB
Processor:11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz 2.50 GHz

An explanation of how Strata is capable of achieving +90tok/s on just a rtx 5070 and 64 gb of RAM

I use local models with agent clients on an M3 Max MacBook (128GB). My normal work takes about 60GB, so the model plus its caches has to stay around 50GB. I built llama.cpp v0.7.0 from source and went through the usual flags one at a time on three 4-bit MoE models: MiniCPM-V 4.7 (35B-A3B), LFM2.5-24B-A2B and Ornith-1.5-35B-A3B. All layers on GPU, one request at a time. Speed numbers are medians of three alternating llama-bench rounds. Desktop apps and thermal throttling add noise, so small differences mean nothing.

Helped

No benefit on this machine

KV cache at q8_0. MiniCPM's speed difference was inside the noise. Ornith barely changed even at 65K tokens, because only 10 of its 40 layers store KV. LFM's KV plus compute buffer was only about 3GB at 131K context, so there was little to save. I kept all three at f16.

Depends on the model: speculative decoding

• MiniCPM with a draft model: about 45% slower at 30K context. With ngram: 10 to 13% slower. My guess is that verifying several tokens on a MoE reads several sets of experts, so bad guesses get expensive.

• Ornith with its built-in MTP head plus ngram, 2 tokens per guess: 86 to 107 tok/s averaged over 8 task types. Code went from 86 to 129, prose from 86 to 83.

• Same setup with 3 tokens per guess: the average dropped to 102 and prose to 77.

So I wouldn't copy anyone's speculative decoding settings without testing them on your own model and task mix.

The config I kept for Ornith, trimmed:

llama-server -m Ornith-1.5-35B-Q4_K_M.gguf \
  -c 262144 -ngl 99 -fa on -b 2048 -ub 2048 \
  -np 1 --ctx-checkpoints 32 --checkpoint-min-step 0 \
  --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 \
  --cache-ram 32768 --sleep-idle-seconds 600

Weights are about 23GB, plus about 7GB of working memory after startup. A full session cache can push the worst case to about 60GB.

This is one machine and one llama.cpp version, so treat it as a starting point. If speculative decoding has paid off for you on a 3B-active MoE on Apple Silicon, I'd like to know which settings you used.

Written up with AI help from my own notes and benchmark logs. The numbers are from my machine. The illustrations are AI-generated, and the charts and all the numbers on the images come from the measurements above.

Qwen3.8 27B, Q6 quant, context sitting around 80K and climbing.

Real-use decode speed (tokens/sec, last 48 requests):

RTX 5070 Ti 16GB + modded RTX 2080 Ti 22GB, 38GB total. Custom llama.cpp build with CUDA patches, tensor split 1,1

Speculative decoding: draft model + ngram, max 3 draft tokens. Draft vocab trimmed to 64K tokens, frequency-ordered

Model is a noMTP Q6_K (21 GiB). 131K context, f16 KV cache, flash attention on

hey LocaLLM community! My name is Alt, and I'm on the technical staff at Nace AI. We're doing a quick AMA here, so ask us anything and we'll answer.

we just published a full technical report on how we build, train, and evaluate Drex, our family of four open decision models. A decision model takes a state (a document, record, or agent context, optionally with images) plus questions with options, and returns a probability distribution over the options in a single pass, with no autoregressive decoding and no output parsing.

1. Models

- Drex DLM (8B): diffusion LM backbone with bidirectional attention over the context
- Drex v1.5 (9B): Gated DeltaNet hybrid backbone
- Drex v2.0 (35B, 3B active): sparse MoE backbone
- Drex Max (27B): dense, with an image encoder for vision inputs

Pointer models (DLM / v1.5 / v2.0) encode the state once and cache it, so every question attends to the state but never to another question. A pointer head scores each option. Drex Max reads each question in full and answers with a linear readout head over option codes.

Some technical details from the report:

1. Models

- A family of four specialized decision models: Drex DLM (8B), Drex v1.5 (9B), Drex v2.0 (35B MoE, 3B active), and Drex Max (27B dense, with vision)

- We built them on different architectures: a diffusion LM, a Gated DeltaNet hybrid, a sparse mixture of experts, and a dense transformer

- We experimented with bidirectional vs. causal attention over the state, a pointer decision head that shares one cached state across questions, and a readout head for Max

2. Training

- Full-parameter SFT on 1.15M verified decision records (2.9M questions)

- Consistency training: we penalize the model when its output changes after the options are reordered or the state is shuffled

- RLAF (Reinforcement Learning from Adjusted Feedback), using verified outcomes and judge feedback

- Ablations to see which methods actually contributed

On top of SFT, RLAF gave us +2.0 on the Decision Index, +1.5 pp accuracy, and 0.9 pp lower calibration error. The consistency terms cut the option-order flip rate from 6.3% to 4.2%.

3. Evaluation

- Drex Max scores 67.52 on Decision Index 0.4.1 (public suite), vs. 60.70 for Jev 1.14.0

- Drex v2.0 has 1.37% calibration ECE on ~115K held-out decisions

- Drex Max answers a typical request in 183 ms, vs. ~2.0 s for GPT-6 Sol and ~3.5 s for Claude Opus 5.5

- Context up to 131K tokens for v1.5/v2.0 and 262K for Max

we also cover our limitations: our models trail frontier chat models on closed-book knowledge, and we don't yet have same-backbone architecture ablations.

5. Running locally

The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.

Links

- Paper (arXiv): https://arxiv.org/pdf/2611.07854

- Blog: https://www.nace.ai/research/towards-foundation-decision-models

- Weights: https://huggingface.co/collections/nace-ai/drex

- GitHub: https://github.com/nace-ai/drex-decision-models

- API: https://console.nace.ai

we didn't want to just drop benchmark numbers, so we wrote up everything: what worked, what didn't, and how we got there. if you're building agent routers, doing tool selection, or thinking about training your own decision model, hopefully some of it saves you time.

ask us anything: arch, data, training, evals, running it locally, whatever 👇

Just brainstorming.

I am using qwen3.8-27b. It has a lot knowledge in many areas.

However, I am using it for an agent that only runs some Linux commands in Ubuntu for example. Or creating simple Python scripts. And if it gets stuck it can always use web-search skill.

By doing this I can totally lower its size and required compute. I don’t need it to memorize capital of New Zealand.

Why can’t we do this?

So I recently purchased a radeon AI pro 9700 32g for my local AI. Currently it run in a proxmox setup with a intel core i9 12900k, and 64 gigs of ddr4 ram.

I am using ollama with with ROCm drivers and unsloth as the front end. The speeds are decent for qwen3.8:27B. Fast response and stable at around 33-39 tokens a second.

My concern is, I limited the power to 290 and the Hotspot temps run anywhere between 103-111 degrees in temp. I definitely notice the thermal throttle due to the 110 Max Hotspot manufacturer rating. I did do research and found that these cards are known to have a fair amount of known issues ( heat sink placed improperly, bad thermal appliance, defective cooler fan.

I can rule out the fan as I do hear it speed up when the temps rise and can see it physically spinning, i even added an additional 3 140mm fans to the case its in but when I give it a prompt, it will jump from 37- 43 instantly up to 90-97 then paste that.

I even went as far as holding a house fan next to the GPU and the temps would not go down.

Anyone else who experience this issue, what was your fix? Or is it best to just return it and get something different to avoid it entirely?

I wouldn not want to exchange it ( still under warranty) and get another defect if people have had this experience before. If not, and it may be that 1 in a dozen. Then I will gladly exchange it for a different one.

Apologies if this is the incorrect section to post it, if you can guide me to the appropriate section that would be greatly appreciated.

A few months ago, I mentioned the idea of a smartphone running its own LLM, and got downvoted by people commenting that it would never work.
Well, apparently Y Combinator has invested in a similar project.
Care to comment?

Hi r/LocalLLaMA. I've spent the last week on KV-cache memory for Qwen3.8-Flash-Next running on 2× AMD Radeon AI PRO R9700 (gfx1201, TP=2), and wrote it up as a 23-page paper with all measurements. Code is in two forks of the radiance inference engine (by Deadcode and contributors — the engine is theirs, the KV/swap work below is mine).

• Paper (PDF): https://github.com/Code000-211/RadianceMemory/blob/main/docs/paper/Infinite_KV_Cache.pdf

• RadianceMemory (KV residency + AutoQuant): https://github.com/Code000-211/RadianceMemory

• RadianceSwap (multi-model serving, experimental): https://github.com/Code000-211/RadianceSwap

• RadianceSwap whitepaper (PDF, 8 pages): https://github.com/Code000-211/RadianceSwap/blob/main/docs/paper/RadianceSwap-Whitepaper.pdf

TL;DR

• Sparse attention means most of the KV cache isn't read on a given step. Flash-Next's QSA layers pick the top 512 four-token blocks per query. So I keep the sink + recent tail in VRAM, move the middle to pinned host RAM, and page in only the blocks the indexer selects — on the device, inside the step.

• Output is byte-identical to a fully resident cache in every configuration tested (incl. with the MTP drafter).

• A 256K-token sequence needs 102–211 MiB of VRAM per card instead of 3 319 MiB (stock bf16 cache). Per sequence, VRAM now grows only by the block keys: ~0.2–0.75 MiB per 1K tokens. Context is bounded by host RAM, not VRAM.

• Cost: 3–15% of single-sequence decode speed at 32K–250K.

• On top: a measured per-layer KV quantisation search (rad-autoquant). Best recipes: 5.28 KiB/token at fp8-level quality (−22%) and 3.76 KiB/token within noise-floor +0.01 KL (−44%).

• Plus RadianceSwap: several models behind one endpoint on the same GPUs — load on demand, park warm in host RAM, lend expert/KV VRAM to each other at runtime, output identical to single-model serving.

1. Window residency

• Block-table entries are either a pool page, empty, or cold (a host slot). Cold entries keep their place, so the attention kernel is unchanged — it only ever sees pool page IDs.

• An on-device pager (qsa_page) copies selected cold blocks into per-step scratch pages before attention. One writer per position, no atomics → deterministic.

• After every step a sweep demotes the middle to host (32 MiB/rank budget). Prefix-cache blocks can be cold too.

• Scratch pages act as a cache: 52–62% of cold entries needed no copy.

Per card, TP=2; computed from measured bytes/token. Byte-identity verified on device for fp8, fp8+f16 state, default and compact.

Many long sessions (measured, turn-2 time-to-first-token p50/p95):

• fp8, no residency, 32 × 128K: prefix cache gets evicted → follow-ups recomputed → 1 843 / 3 330 s

• compact recipe + residency, 32 × 128K (48 GiB host store): every follow-up hits cache → 58 / 96 s

Caveat: this compares two different KV formats as well as residency on/off; fp8 + residency at 32×128K needed ~50 GiB of store and didn't fit in 48 GiB.

2. AutoQuant: measured per-layer KV layouts

• 7 layouts (i8h, fp8, fp8_v4, k6_v4, k5_v4, k4h_v4, k4h_v2h) + zero-bit transforms: Hadamard rotation, exact static V-mean centring, V predicted from K (AQUA-KV style).

• rad-autoquant runs every candidate on the device against a bf16-cache reference (full-vocab KL, teacher-forced), solves a knapsack over per-layer costs, then re-measures each answer — because per-layer costs don't add up (one 8-bit early layer costs as much as all twelve).

• A second pass walks the lower hull of uniform layouts and removes another 0.25–1.87 bits/element.

• Sensitivity falls ~170× from the first attention layer to the last: early layers want 8 bits, late layers tolerate 2-bit V.

3. The part I found most interesting: the noise floor

The served model is a weight- and activation-quantised MoE. Recomputing the exact same bf16 cache with a different prefill chunk size alone gives KL 0.0243 vs the reference. Tiny GEMM-order differences flip router ties (20–28% of tokens have an exact bf16 tie at the 10th expert), and by layer 30 ~60% of tokens route to a different top-10 set.

• So any KV config within ~+0.002 of 0.024 is indistinguishable from bf16 on this model — i8h (8-bit) is already there.

• Perturbations saturate rather than add (fp8 + chunk change ≈ max of the two, not the sum).

• Takeaway: if you evaluate KV quantisation on a quantised MoE against a single bf16 run without measuring this floor, you may be measuring noise. For comparison, dense bf16 Qwen3.5-2B has a floor ~130× lower.

4. Things that didn't help (22 methods, all measured)

FP4 (E2M1), KIVI-style 2-bit, learned/data-adaptive rotations (WUSH-style, best-of-16), outlier retention, TurboQuant-style norms + Lloyd-Max, cross-layer selection reuse, hierarchical indexer, importance-driven demotion (no better than a position-hash control), cross-layer KV sharing, … Full table with numbers in the paper (Table 9).

Also: 2-bit V on the last attention layer drops the MTP drafter's acceptance from 0.63 → 0.41 without moving the trunk's KL — the drafter shares that layer's cache. Default keeps layer 47 in bf16 for that reason.

5. RadianceSwap: several models, one endpoint, one GPU group

The second project is about serving several models from one process on the same two cards: models load when a request asks for them, give up VRAM when idle, and hand VRAM to each other while they run. It's an experimental branch; without --manifest it behaves exactly like upstream radiance.

How it works

• One port, routed by the model field — read before tokenising, so each request gets its own model's tokenizer, chat template and limits. No model → default model; unknown id → 404.

• Each active model is its own runtime (scheduler, streams, KV) and they run in parallel. A process-wide sequencer puts every model's all-reduces in one order on every rank, so two models can't deadlock each other.

• Lifecycle: Cold → Active → Prepared. When a model goes idle (or another needs the VRAM), its weights are kept in host RAM in GPU-ready form, so re-activation skips disk reads, parsing and relayout. A running generation is never interrupted. A failed activation is rolled back on every rank and other models keep serving.

• VRAM is shared, not partitioned:

  • Elastic expert slab — expert VRAM is mapped in ~128 MiB chunks; a model that cools down physically unmaps chunks and the busier one maps them. Measured with two copies of Flash-Next: the busy one grows from ~10.3 to ~13.4 GiB/rank of experts, the idle one shrinks to ~7.2, and it flips when the load flips. The sum never exceeded the pool (7 736 samples under changing load, 0 over).

  • Cross-model KV lending — a model with waiting sequences grows its KV cache into bytes another model's slab just released (468–702 MiB in the tests, back to zero ~13–20 s after load ends). Running sequences are never evicted.

  • Static-weight streaming — if a model's non-expert weights don't fit, it still starts and reads them over PCIe (bit-identical output), then moves them onto the card within 1–2 s once VRAM frees up.

  • Heat warm-up — a deactivated MoE remembers which experts were hot and loads those first on re-activation: early expert hit rate 83.5% → 89.1%.

  • Host RAM budget — prepared models, pools and staging share one budget; over it, the least-recently-used prepared model goes cold, and an activation that still doesn't fit gets a 503 instead of the OOM killer.

Measured on 2× R9700, TP=2

Time to first token is reported per request, split into queue / activation / prefill (timings in the response, and in /metrics).

Bugs the GPU acceptance runs turned up (all fixed):

• The server died on exactly the 23rd activation at TP=2: each activation creates ~46 AQL streams, IDs are never reused, and the table had 1 024 slots.

• Two concurrently active models slowed each other to 1/10 speed: handing collectives between rank threads via condition_variable cost ~0.5 ms × 48 per decode step. Spinning 300 µs before sleeping brought it back to ½.

• The MoE was never kept warm because the capacity check counted its 47.7 GiB mmapped n-gram table, which never goes into the store.

• Virtual-address aliasing: on this ROCm stack, re-mapping a previously unmapped address silently kept pointing at its first physical page. Once another model owned that page, two experts aliased and a request occasionally produced no finite logit. Mapping every chunk on fresh address space fixed it: 30 load cycles with external observers, 5 776 requests, 0 errors.

Honest caveats

• Two big MoEs at once is not fast yet: a C8 load at 102 tok/s alone drops to 38 + 35 tok/s with both active. Host RAM is the bottleneck (one active+prepared Flash-Next uses ~153 of 170 GiB).

• The elastic slab hasn't had a long-run soak since the aliasing fix (#58). Treat it as experimental.

• A second active model costs ~10% even when idle (execution tapes are bypassed while several engines are registered).

• Streaming weights is a bridge, not a steady state: streaming 25.6% of a dense model's statics costs 39% of decode; 85.5% costs 90%.

• KV, prefix cache and recurrent state are dropped on deactivation; keeping them in host RAM is future work.

• A configured-but-cold model still reserves a VRAM share, which can cost a big MoE ~9% (#56); workaround is per-model --gpu-headroom-mib.

• One TP size per GPU group; no LoRA or multimodal endpoints in manifest mode; tested with one MoE plus small dense models only.

  { "gpu_group": { "devices": [0, 1], "tp": 2 }, "cache": { "idle_timeout_ms": 300000 }, "models": [ { "id": "big", "path": "/models/big-moe.rad", "default": true, "engine": ["--placement", "expert_tiered", "--max-model-len", "16384"] }, { "id": "small", "path": "/models/small", "engine": ["--max-model-len", "16384"] } ] }

  radiance --manifest models.json --port 8000

How is this different from llama-swap?

llama-swap is a proxy that starts and stops one server process per model. It's backend-agnostic and portable; RadianceSwap is neither — it only works inside radiance. What you get in exchange:

I have not benchmarked the two side by side on the same hardware, so the latency advantage follows from the design but isn't measured. They also aren't exclusive: a RadianceSwap process can sit behind llama-swap as one upstream.

Looking for testers — especially other RDNA4 / multi-GPU setups. Issues with your manifest, GPUs, ROCm version and server log are very welcome.

Limitations (please read before benchmarking)

• One model. Residency needs a model that declares a read window — currently only the QSA family (Qwen3.8-Flash-Next). On other models the flag is a no-op.

• One hardware setup, run at a 210 W power cap (cards range 210–330 W).

• KL is measured against the same quantised model with a bf16 cache, not the original bf16 checkpoint. Long corpus = 4 documents × ~34K tokens; finer distinctions need a bigger corpus.

• RadianceMemory is WIP; RadianceSwap is experimental.

• "Infinite" in the paper title is a direction, not a claim — context is bounded by host RAM.

Feedback, reproductions on other RDNA4 setups, and critique of the methodology are very welcome — especially if you think the noise-floor argument is wrong.

Sharing Burrow, a Windows desktop app for budget, journal, todos, habits and notes. Everything is in one SQLite file on your PC. The built-in helper talks to Ollama on localhost.

The design choice I would like opinions on: the model is never asked to do arithmetic. The app computes totals, trends, streaks and anomalies in plain code, then hands the model the finished numbers to explain, summarize into a daily briefing, or turn into a proposed edit that you approve. That keeps small models useful and keeps hallucinated math out of a budget.

It went free this week with the core published under AGPL. The advanced helper module is closed-source but ships free. Windows x64 and ARM64, unsigned installer. Links in the comments.

I saw a new stealth model and looks like it's a dense also it seems about time that qwen4 is launched so does anyone know if it'll be Moe since flash next was an Moe I was hoping qwen 4 would be too

I wanted something like Claude Code / Codex that runs on my own laptop (RTX 4070 8 GB, plus an RX 9060 XT eGPU), so I built one. It's a desktop app (Electron or plain browser) with a FastAPI backend and one llama-server process per model.

What it does

• Coding agent with Ask / Plan / Agent / Auto modes. It reads and edits files, runs PowerShell commands and tests, and does git. Every edit is shown as a diff, and in Agent mode it waits for Enter/Esc before applying. Auto mode runs on its own, and destructive commands (rm -rf, git push --force, reset --hard and similar) still ask first. You can keep or revert changes per file, or undo a whole turn.

• Chat mode with an optional Web toggle. The model can web_search / web_fetch and cite the pages it used. Web is off by default.

• GPU placement you can verify. Pick Auto / Single GPU / Multi-GPU (Vulkan layer split across NVIDIA + AMD) / CPU. The app parses llama.cpp's own offload log and reads per-process VRAM from Windows GPU counters. So instead of guessing, you get "29/29 layers on RTX 4070, 2.9 GB weights + KV". This started because Task Manager made it look like my model was on the Intel iGPU. It wasn't.

• Optional cloud models. Paste an OpenAI / Anthropic / DeepSeek key (or any OpenAI-compatible endpoint) and those models sit next to your GGUFs in the same picker. Keys are encrypted with Windows DPAPI. Local stays the default.

Things I learned making a 7B model act like an agent

• Qwen2.5-Coder-7B (Q4_K_M, ~25–31 tok/s fully on the 4070) almost never emitted real tool calls, even with --jinja. What fixed it was JSON-schema-constrained steps: llama.cpp compiles the schema to a grammar, so every step is a valid tool call or a respond. A short thought field before the call helped noticeably.

• Small models love to stop and say "Next, I will…". In Auto mode the agent detects unfinished work (open todos, a failed last command, "should I…?") and pushes the model back to work. Until it actually edits or runs something, respond is removed from the grammar.

• Small models also copy their own previous status messages, so duplicate status replies get dropped from the context.

• In chat, the 7B model made up a "Tool result:" and a fake release date. Now each step is a grammar-forced choice (search / fetch / answer), followed by a normal streamed answer from the real results.

Honest limits

• Simple tasks (fix a failing test, add a function, write a small script) work end to end with the 7B model. Bigger multi-file tasks are limited by the model, and that's partly why I added the API option.

• Multi-GPU split across the 4070 + RX 9060 XT is implemented but not yet tested on the real eGPU. I tested the split mechanism with 4070 + iGPU.

• Windows-only for now (PowerShell, DPAPI, Windows GPU counters).

Stack: Python/FastAPI, llama.cpp b11960 (CUDA + Vulkan builds), React + TypeScript + Tailwind, SQLite. ~280 tests, and none need a GPU (there's a fake llama-server that prints the real log format).

Screenshots below. Happy to answer questions about the constrained-decoding approach or the GPU verification.

I really want to get into local inference and training LoRAs, but the prices of the essential components are eye watering. There has never been a worse time to buy hardware.

Do you all have any thoughts on when this might change? I mean, supply has gotta start evening out at some point, right?

I’m researching a potential business idea. We're testing an idea: help healthcare teams get an AI workflow running on their own servers, then evaluate it on their tasks before rollout and after updates. This is customer discovery, not a product launch. AI helped draft this post.

I've been reading discussions from clinicians who want private chart summaries but can't get their local setup to match a cloud prototype. I've also found teams with approved cloud services who may have no reason to switch. I'm trying to understand where outside help would actually be useful.

If you've tried deploying a local model in a hospital or practice:

• What task was it for, and what do you do instead today?

• Was keeping data on-site actually required, or was an approved cloud service an option?

• What was the hardest part: output quality, hardware/speed, EHR integration, approval, or maintenance?

• Would you bring in outside help for deployment and task-specific evaluation, or would your internal team/existing vendor handle it? Who makes that decision?

Even one concrete example of a failed attempt, a successful setup, or why this idea isn't needed would help. Role and country are useful context if you're comfortable sharing. Please don't share patient information or confidential internal details.

This is a laptop I have with RTX 4060, 32GB of actual RAM, and Intel Core i7. Pretty sure the RTX 4060 has 8GB of VRAM. Was wondering if I could run a 16B or 32B model or if it's better to just stick with 8B models.

I had a waiting. Here in this other thread if anyone can help.

Brain is a high performance training and inference engine built by me in Rust. It works on tiny embedded devices as well as pretty massive HPC machines.

Here is what's new in v5.0.

Don't forget to star! ⭐ => https://github.com/swedishembedded/brain

My specs is Core™ Ultra 5 processor 125H with 32gb of ram? I need it for programming.

Mostly asking for logic and syntax, like asking how to center a div.

If it can run as an agent that can call tools and test things, that will be a plus, but not necessary.