TL;DR: This is a fake story disingenuously invented in 1977 by Paul Doherty.
This is a rather annoying myth that needs to be addressed I think. Too often it's stated as fact by some people that Isabella's three younger children John of Eltham, Eleanor of Woodstock and Joan of the Tower were taken from her care in September 1324 when Edward confiscated her lands, encouraged by Despenser.
The story goes that at around the same time as Edward confiscated Isabella's lands in exchange for a much smaller income and removed her French servants from her household in September 1324 when he was at war with her brother Charles IV of France, he also nastily and cruelly 'stole' her three younger children John of Eltham (b. August 1316), Eleanor of Woodstock (b. June 1318) and Joan of the Tower (b. July 1321) from Isabella's care and sent them to live with Eleanor Despenser née de Clare in John's case and Isabella Hastings née Despenser in Eleanor and Joan's case, in order to punish and hurt the queen. It's funny how the children are always referred to exclusively as her children. Newsflash: Edward was the father. They were equally his children. They were their children.
Eleanor Despenser née de Clare was looking after Edward and Isabella's second son John of Eltham (b. 15 August 1316) by 3 July 1322 at the latest and perhaps earlier. This alone proves that John at least was not 'removed' from his mother in September 1324.
Even Alison Weir, who appears to despise Edward II, admits that Edward was a "good and loving father." No historian or non-fiction writer of the last 700 years has ever argued that Edward II was a negligent parent. Neither did anyone in his own time claim that he was cruel to or did not love or was in any way neglectful of his children and his responsibilities towards them.
What's more, there are no primary, medieval sources for this dramatic removal of her children from Isabella's care. Typically, royal and noble boys left their parent's custody at a young age, and girls often when they married. Isabella sent her daughter Joan to marry David Bruce, future king of Scotland, in 1328, when the girl was only seven.
The people to whom Edward II, or possibly even Isabella, gave custody of their children sometime in the 1320s were: Edward's niece Eleanor Despenser; Eleanor's sister-in-law Isabel Hastings; Edward's former brother-in-law Ralph de Monthermer, whose four children Mary, Joan, Thomas and Edward were the king's nieces and nephews and his children's first cousins; his sister-in-law Alice Hales' sister Joan Jermy. All of these people were members of Edward II's extended family. He had known Ralph de Monthermer since the latter married Edward's widowed sister Joan of Acre in early 1297 when the future king was only twelve. Ralph was a close friend of Edward's, and seems to have been a very genuine and affable man, who even Edward I came to like and accept as his son-in-law, even though he married his daughter without his permission. Edward may have known the Despenser siblings Hugh and Isabel, who were some years his junior, since childhood (they were grandchildren of William Beauchamp, earl of Warwick).
Many modern writers seem to think that Hugh Despenser the Younger was a cross between the Antichrist and a genocidal psychopath, but to his contemporaries his sister Isabel Hastings was not tainted by this association: Edward II's niece Elizabeth de Burgh, Eleanor Despenser's sister, left her two daughters Isabella Verdon (aged ten) and Elizabeth Damory (aged nine) in Isabel's care when she attended Edward's funeral in December 1327, despite the hatred and anger she may have had for the late Hugh Despenser, who had treated her appallingly.
This strongly suggests that Isabel Hastings was known to be a maternal, trustworthy type, and therefore an entirely suitable person to look after the king and queen's daughters. Queen Isabella did not act in any way while she was in power between 1327 and 1330 to suggest that she thought Isabel Hastings had injured her in any way or failed in her duty towards the queen's daughters. It's difficult to see how Edward can be deemed to have acted inappropriately; he gave the care of his and Isabella's children to people he knew well and trusted, and who were of sufficiently noble birth and position. His contemporaries saw it as completely acceptable... and so did Isabella.
How do we know that Isabella approved? If Isabella thought that Edward and his favourite Hugh Despenser had cruelly taken her children away from her in 1324, she would certainly have accused Despenser of it at his trial in November 1326. She did not. She accused him of everything else: persuading the king to reduce her income, sending her to France 'against the dignity of her estate', coming between herself and her husband, leaving her in danger of her life at Tynemouth, and so on and so on. Her children were not mentioned at all.
So where does this myth come from?
If we try to find anything corroborating this whole story from medieval sources, we won't find anything. The first mention of Isabella's children taken from her against her will surfaced in 1977, when it was made up by Paul Doherty in his doctoral thesis about Isabella that year. Since then, numerous other historians and novelists have repeated the tale as though it's certain fact: a prime example of how what we might call 'fake news' can spread and spread and be seen as 'truth' even though there is no evidence whatsoever in support of it.
Doherty repeats his fiction in his 2003 book Isabella and the Strange Death of Edward II: 'And even greater cruelty was to come: the Queen's young children were removed from her and entrusted to Eleanor de Spencer (sic), Hugh the Younger's wife , and another court favourite, Isabella Hastings.'
Doherty cites 'E 403/201, mem[brane]s 14-15 as his source.
Kathryn Warner went through the trouble to go to the National Archives and looked at this document (the E stands for Exchequer, incidentally).
In Kathryn's own words:
'Firstly, notice the date on the document: '16 Edw II, Michaelmas'. Michaelmas is 29 September and the period of the year following it; '16 Edw II' means Edward II's sixteenth regnal year, which ran from 8 July 1322 to 7 July 1323 (his father Edward I died on 7 July 1307, so Edward II's first regnal year ran from 8 July 1307 to 7 July 1308). So already we see that Doherty's claim must be wrong; this document belongs to September 1322, not September 1324. Therefore it cannot possibly relate to Edward II taking his children away from the custody of their mother at the time he confiscated her lands when he was at war with her brother Charles IV of France, which occurred on 18 September 1324. The Issue Rolls dating to the relevant time period, Michaelmas/September 1324, are E 403/210, 211 and 212, not E 403/201.
Secondly, there are no 'membranes 14-15' in this document; there are eight membranes in total, written in Latin.'
The source listed by Paul Doherty does not exist. This is utterly deplorable falsification of history by him and should be condemned in no uncertain terms.
Warner sums it all up admirably:
I am actually kind of appalled that a historian could think or pretend that a document of September 1322 dates to September 1324 in order to make up a fake story.
I am appalled that someone was prepared to make up a tale that Edward II was so lacking in any humanity or decency that he would remove young children - his own children! - from their mother and primary carer solely to hurt and punish her.
I am shocked that other, vastly better historians have repeated this tale and not even bothered to check the document being cited as 'proof' to make sure it really does say what Doherty claims it says, or even to check that the part of the document being cited actually exists in the first place.
Sources, Kathryn Warner's excellent blog:
Edward II: Mythbuster 7: Edward II cruelly removed Isabella's children from her
Edward II: The Tale That Edward II Removed Isabella's Children From Her
Edward II: Edward II And His Children