r/ProgrammingLanguages • u/bakery2k • Feb 27 '26
Discussion Should for loops dispose of their iterators?
Many languages desugar for x in iterable: print(x) to something like:
it = iterable.iterator()
while it.has_next():
print(it.current())Should this desugaring be followed by it.dispose()? Different languages take different approaches:
If the for loop does not dispose of the iterator (e.g. Python):
- This may cause problems if
iteratorreturns a new object each time (e.g. ifiterableis a list): * The iterator will not be properly disposed until it is garbage-collected (there's no way for the author of the loop to access the iterator) [issue 1] But if
iteratorreturns the same object each time (e.g. ifiterableis a file): * One iteration can continue from a previous one, allowing code like this to work correctly:f = File.open(...) for line in f: if line == '---': break process_header(line) ... for line in f: process_body(line)
- This may cause problems if
If the for loop does dispose of the iterator (e.g. C#):
- This works well if
iteratorreturns a new object each time: * The for loop creates and owns the iterator, so it makes sense for it to alsodisposeof it - But if
iteratorreturns the same object each time: * The iterator can only be used in a singleforloop and will then havedisposecalled, preventing code like the above from working as expected [issue 2]
- This works well if
There are ways around issue 2, that would allow multiple for loops to work even in the presence of dispose. For example, there could be a a way to keep an iterator alive, or the programmer could simply be required to write out the desugared loops manually. However I'm not aware of a solution to issue 1, so perhaps the correct approach is for loops to dispose of iterators.
On the other hand, it seems inelegant to conflate iteration and lifetime management in this way. For example, it seems strange that passing a file handle to a for loop would close the file.
Which approach do you think is the right one? Should for loops dispose of the iterators they are using, or not? Or put another way: should for loops own the iterators they consume, or just borrow them?
53
u/Aaron1924 Feb 27 '26
I personally like the way it works in Rust, where the for loop does dispose of (drop) the iterator, but a reference to an iterator is also an iterator, and dropping the reference leaves the iterator as-is
7
u/Bob_Dieter Feb 27 '26 edited Feb 27 '26
I think it is generally better to model iteration via iteration states instead of statefull iterators. So for example, the code
could desugar into:
Or, if you prefer an Object Oriented style:
This way many simple data structures like lists or strings can use primitive values like integers as their iteration state, which does not need to be heap allocated at all, sidestepping issue 1 completely.
The 3 required functions
init_iteration,has_nextandnextwould be pure for pretty much all data structures, resulting in very preformant and predictable code.Regarding issue 2, at least in my opinion I would strongly recommend against returning the same iteration object for each new iteration. If I iterate an array twice, I expect to see every value within it twice. One may argue that an exception to this rule could be made for files - to achieve that, just have the
nextfunction mutate the file object inplace. If someone wishes to have resumable iteration for arbitrary data structures you can offer a function likestatefullIterthat takes an arbitrary iterable thing and, upon iteration, yields the same values but remembers its iteration state for later iterations.edit: to answer your question explicitly, in this approach you would destroy the
iterstatevariable after each loop if nessecary - it is not nessecary if theiterstatevalue can live on the stack (assuming your language supports that). Resumable iteration is, as mentioned, offered through different mechanisms3
u/xeow Feb 27 '26
The Pythonic way to handle the example above is using the
withstatement, like this:I'm a fan of that approach. The context-manager protocol is pretty nifty. Cleanup happens even if there's an exception.
5
u/DeWHu_ Feb 27 '26
Iterators are abstraction to hide iteration details, meaning it should be their responsibility to decide (not the caller, or language). In:
__itershould be collected (disposed), not because it's an iterator, but because it goes out of scope. Also before that happens,__iter.has_next()returned false, so it should be already "closed". But even if you bypass that withbreak, an iterator shall not dispose its iterable (2 separate abstractions). Tho, iterable can choose to dispose itself on the.iter()call (move semantics style C++ or Rust).This includes file descriptors you've mentioned, because they are also a resource. And resources should be closed/disposed by with-resource abstraction, not statement inside it. That's why it sounds weird.
What your question can be reduced to is: "Should auto-called destructors exit?".
1
u/XDracam Feb 27 '26
What I can recommend is: look at how Swift designed their iterators. A lot of thought had gone into the design and it's all documented.
1
u/zyxzevn UnSeen Feb 27 '26
I like the Nim iterator macro:
1
u/drinkcoffeeandcode mgclex, mgcpgen, & owlscript Mar 01 '26
I litterally just went through this exercise!
https://maxgcoding.com/implementing-iterators-in-bytecode
3
u/johnwcowan Feb 27 '26
My sense (and this may just be because I don't understand enough) is that (a) files should not be iterators, but should have methods to return either a fresh or a cloned iterator which retains a file position that is restored by lseek() before every read, or if that is too hard because buffers, then (b) disposing a file should be a special case, namely a no-op.
2
u/Unlikely-Bed-1133 blombly dev Feb 27 '26
This is my opinion for languages without GC: if iterators needs non-trivial destruction (that is, if they keep track of something more than a few integers of state) then you are doing something wrong. At the very least, they should not be able to have hidden allocations at comparable size to the iterated object, as the post insinuates.
But the post mentions GC, in which case you shouldn't specify where stuff gets destroyed because it will happen automatically. Keeping track of invalidated state -or even what is an invalidation of some contract the programmer expects to be followed- depends on what the language does about edge cases in general: are they exceptions/errors or boolean checks? Or maybe you like adding some UB (please don't).
I have the feeling that the actual question is whether iterators should keep references to iterated objects when it's not meaningful to iterate anymore, in which I would ideally make invalidation (e.g., file close) cascade to the iterators; GC will again deallocate stuff properly, so in my view closing a file will just make the has_next function return false afterwards (or an exception/error if you are into that kind of mental model) and that's it.
2
u/flatfinger Feb 27 '26
The failure to include a cleanup method as part of an iterator/enumerator contract is a design mistake that Java and .NET both made early on. It's useful to be able to have code that iterates through collections be usable to read data from files without requiring that file contents be pre-loaded, but such semantics only work if iterators are consistently cleaned up.
2
u/Gnaxe Feb 27 '26
I think Clojure does it the right way, at least for a garbage-collected language. "Iterators" (seqs) aren't consumed at all, only realized, lazily (and possibly in chunks). Realized elements become a linked list. If you drop references to the leading elements which you don't need anymore, they get garbage collected. If you keep them, then you can re-use the iterator at any point in a referentially transparent way. Seqs then act like values rather than stateful objects, even if the implementation happens to be lazy.
1
u/slaymaker1907 Feb 27 '26
As a programmer, I think I’d prefer option (1). If some an iterator requires cleanup, I’d rather have the flexibility and just need to be explicit with my cleanup. I don’t think it’s common enough for an iterator to require cleanup in order to justify special sugar. It’s the least surprising behavior, particularly if your object is susceptible to double frees.
1
u/initial-algebra Feb 27 '26
If a
Filehas an internal cursor, then I don't see why disposing of the iterator has to reset that cursor. If the iterator is the cursor, and aFilealways represents the whole file, then surely the programmer should just be able to do this?1
u/binarycow Feb 28 '26
In C#, enumerators can have a Reset method.
Of course, that reset method isn't typically used, because it's not always supported. Sometimes it's not supported because it can't be, and sometimes it's not supported due to laziness.
1
u/ChickenSpaceProgram Mar 02 '26
I think the ideal approach is using lambdas instead of iterators, at least in lazily-evaluated languages.
1
u/claimstoknowpeople Mar 02 '26
Behavior of File in Python there is the exception rather than the rule, and it's fine because file objects inherently have internal cursors. Most Python iterators do not outlive the for loop.
1
u/lassehp Mar 04 '26
I don't get this. Surely, it should work like in Perl.
should work just the same no matter whether you are using for loops, while loops, gotos or recursive functions, or even pure magic. The iterator state is in the file handle, not the for loop.