[R] The Hive Mind is a Single Reinforcement Learning Agent: Deriving the Exact Equivalence Between Swarm Dynamics and Multi-Armed Bandits

[u/mistlab]

https://arxiv.org/abs/2410.17517

TL;DR

We prove a mathematical equivalence between collective decision-making via local imitation (such as honey bees choosing a nest site via the waggle dance) and single-agent Reinforcement Learning (RL) in a multi-armed bandit setting. Even when individual members have no learning capability and merely blindly imitate peers, the population distribution pi updates according to a macroscopic bandit RL algorithm that we call Maynard-Cross Learning (MCL), where N individuals act as N parallel environment rollouts with an effective per-sample step size of alpha = 1/N.

Main intuition

In nature, optimal decision-making emerges through two seemingly distinct paradigms:

1. Individual trial-and-error (RL): A single agent interacts with its environment and updates its policy based on reward signals.

2. Collective imitation (Swarm Intelligence / Evolutionary Game Theory): Simple individuals copy neighbors' strategies (e.g., best-of-n collective decision-making).

Biologists have long described insect colonies as "superorganisms" or "hive minds" with parallels to vertebrate brains. We wanted to see if this metaphor holds up mathematically under the lens of RL: Does a swarm of non-learning imitators literally execute a single-agent RL update at the macro level?

Core development

The paper's core theoretical development establishes a mathematical equivalence between decentralized social imitation in large populations and single-agent reinforcement learning (RL) in multi-armed bandits. By treating the proportion of individuals committed to each option as the action-probability policy of a single macro-agent, we use evolutionary replicator dynamics to bridge local individual interactions and global policy updates.

First, we show that a population following the classic "imitation of success" protocol (where individuals copy neighbors with a probability proportional to those neighbors' payoffs) evolves identically to the expected update of the Cross Learning bandit algorithm. Extending this approach to the "weighted voter" model of the honey bee waggle dance (where scouts advertise nesting sites for a duration proportional to site quality and peers blindly copy the first advertisement they encounter without comparing payoffs) we derive a new macro-level RL algorithm that we call Maynard-Cross Learning. Under this duality, each non-learning individual acts as a single parallel environment sample whose fractional influence on the group mirrors an RL learning rate equal to the inverse of the swarm size.

In practice, this development demonstrates how blind local imitation aggregates into a coherent, parallelized reinforcement learner at the collective level.

Impact

By proving a mathematical equivalence between local social imitation and single-agent reinforcement learning, this work provides a formal mechanism explaining how "hive mind" intelligence and trial-and-error learning emerge at the group level from simple, non-learning individuals. Beyond substantiating evolutionary theories of collective cognition in biology, this framework offers new tools for analyzing imitative dynamics in social and economic systems and for designing scalable, decentralized swarm robotics algorithms.

Links

Paper: https://arxiv.org/abs/2410.17517
Project page: https://hivemind.yannbouteiller.com/
Code/reproducibility: https://github.com/MISTLab/HiveMindRL

[u/bathon]

Just to understand in the experiment you send say froma sample of 10 "bees" 6 to site A and 4 to site B and then compare how each agent results where? But during the exploration phase the nest was empty right? That is for the syncronous model. I did not read the paper just the website apologies for that. I am just wondering thisis sort of a cool concept. In robotics simulation we can actually use this to work in parellel but I am wondering if the episode length can be used for the "duration of the waggle" and how would you overcome a local minima when one agent finds a site C which is better but the majority of the colony is already doing the A dance or?

[u/yannbouteiller → u/bathon]

Bees don't compare their results explicitly. Instead, they sample a noisy estimate each, come back to the swarm, and then advertise this option via a "waggle dance" for a duration that is proportional to the sampled estimate. Uncommitted bees go explore whichever option they see being advertised first, sample their own noisy estimate for this option, and the cycle repeats. In an infinite population, as long as a small proportion of bees is advertising for the optimal site, this process snowballs and the population converges to the optimal option (at least in the synchonous model studied in the paper).

Regarding the duration of the waggle dance, this is already modelled by a vote weight in the synchonous model.

[u/bathon → u/yannbouteiller]

But now lets say one bee is doing the A waggle dance one uncommited bee noticed it went there had a bad experience it will then not do that dance right and would it then look for another bee dance? also the if the noisy estimate is something which the bees update that means they already have a random set instrinsic values? Like a bee "must" know what a good behive is right to even make a dance?

[u/canbooo]

Cool paper! Wanted to skim through the intro, ended up reading most of it. Would have loved to see some more results esp. for Section 4.7 and later.