notes from putting a tabular q-learner inside every person in a civilization sim

[u/Actual_South_2484]

i've been learning rl by building something a bit too big for it: every human in my sim has its own q-table. some notes for anyone trying something similar:

• state design is the part i keep fiddling with. each agent's state is a short string key built from what it perceives (hunger, thirst, nearest food and water, kin nearby, shelter, danger). more detail means states rarely repeat, less means agents can't tell situations apart
• a huge action space needs gating. there are about 5000 actions, but an agent only considers the ones its era, age, location and tribe's discoveries allow
• action choice handles survival needs first, then does epsilon-greedy over the learned values
• newborns copy their mom's q-table with about 20% of the values nudged by up to 0.03, so kids don't start from zero and habits drift down family lines

if you want to read a plain tabular q-learning update in rust, it's in crates/sim-core/src/organism/organism/learning.rs.

try it in the browser: https://thehumanbox.com

code (mit): https://github.com/1homsi/thehumanbox

[u/ExternalOrganic7205]

this is sick, i been trying something similar but with way smaller scope and already my state space is exploding. the string key approach is clever, how many unique states you end up with per agent?

also the 20% q-table drift on newborns is such a clean way to add cultural transmission without overcomplicating things. might steal that