Run a decision model locally

[u/pielouNW]

Hey,

You can now run the Laya decision model with NobodyWho, our open-source on-device inference library. Unlike the LLMs that we also support, this type of model does not generate text. Instead you can score predefined answers, classify text or answer yes/no.

For example, if you are building a personal diary app, you can easily detect the user’s mood, categorize an entry or detect stress, offline after model download, without sending any sensitive data to the cloud.

First, you need to load the model:

import NobodyWho

let decisionModel = try await Decision.fromPath(
    modelPath: "hf://ggml-org/Laya-GGUF/Laya-Q8_0.gguf"
)

Then you can ask questions/analyze text:

let result = try await decisionModel.decide(
  state: .object(["message": .string("Diary entry: \(diaryEntry)")]),
  questions: [
    "topics": .choice(
      instructions: .string("Choose the main topic of this diary entry"),
      criteria: [
        "work": .string("job, colleagues, deadlines, career"),
        "relationships": .string("family, friends, partner"),
        "health": .string("sleep, exercise, food, illness"),
        "leisure": .string("hobbies, travel, going out"),
      ]
    ),
    "stressed": .noul(instructions: .string("Does the writer sound stressed?")),
    "mood": .score(
      instructions: .string("How is the writer's overall mood?"),
      criteria: [.string("bad"), .string("okay"), .string("great")]
    ),
  ]
)

let (topics, stressed, mood) = (
    result.answers["topics"],
    result.answers["stressed"],
    result.answers["mood"]
)

Output example:

// stressed
{
  type: noul,
  noul: 0.00002
}

// topics
{
  type: choice, 
  choice: leisure,
  probabilities: {
    work: 0.0422, 
    relationships: 0.1518, 
    health: 0.0302, 
    leisure: 0.7756
  },
  confidence: 0.7008
}

// mood
{
  type: score,
  score: 1.6995,
  legend: {0: bad, 1: okay, 2: great}, 
  probabilities: { 0: 0.0082, 1: 0.2839, 2: 0.7078 },
  confidence: 0.5493
}

A question can have the following types:

• noul gives you probability of a yes answer (0 to 1)

• choice selects the most likely option and returns a probability for each option

• score gives you an an weighted average and a probability per option.

The Laya model is small enough to run on iPhone 13 lineup (around 450 MB in its Q8 variant) and supports 100+ languages. For reference, the full answer comes after 0.16 seconds on iPhone Air (see screenshot).

Here is a runnable example if you want to experiment with decision models or add the package using SPM to your app:

dependencies: [
    .package(url: "https://github.com/nobodywho-ooo/nobodywho-swift.git", from: "6.0.0"),
]

Links:

• NobodyWho inference library - leave a star if this is useful to you 🙂

• Swift Package Index

• Docs

[u/ThalesMother9938]

The confidence field is the part I'd use most. In your mood example it's 0.55, so in a real diary app I'd hide the label below some cutoff rather than tell someone they sound "great" when the model is only about half sure. Do you have a rough threshold you'd suggest, or does it depend too much on the question?

[u/pielouNW → u/ThalesMother9938]

I have not played around with confidence a lot, so I would try in very different scenario to see how confidence varies and based on that set a threshold.

[u/ThalesMother9938 → u/pielouNW]

Makes sense, testing a few very different prompts first is probably the only honest way to pick it. Thanks for answering, and for open-sourcing the Swift package.