IMO the documentation isn't *wrong*, just misleading :) This will lead to pathologically inefficient behavior when any individual directory has a large number of files in it.Īuthor: R. Internally, it iterates over the directory tree, and stores all the names from each directory at once. Note: The current iglob() implementation is optimized for the case of many files distributed in a large directory tree. We're shooting for principle of least astonishment here. If I ran iglob() on that and discovered that it had internally generated a list of all 200k names in memory at the same time, I would be pretty darn surprised, based on what the docs say now. The thread that led to this started out with the use case of a directory that had 200k files in it. I can't think of a clean way to explain that, so a doc patch is welcome. I think the documentation is trying to say that iglob doesn't *always* store all the matching files, without implying that it *never* stores all the matching files. (3) append the files matching 'spam/c/*.txt' (2) append the files matching 'spam/b/*.txt' (1) generate the list of files matching 'spam/a/*.txt' (3) generate the list of files matching 'spam/c/*.txt' and yield them (2) generate the list of files matching 'spam/b/*.txt' and yield them (1) generate the list of files matching 'spam/a/*.txt' and yield them What iglob does is conceptually closer to: I agree that the documentation could be improved, but it's not really *wrong*. Or rather, it doesn't really not do what it says it doesn't :-)Īuthor: Steven D'Aprano (steven.daprano) * I recognize that iglob() is not going to get fixed in 2.7, but at least the documentation should be updated to point out that it doesn't really do what it says it does. In a nutshell, the iglob() docs say, 'Return an iterator which yields the same values as glob() without actually storing them all simultaneously.' The problem is, internally, it calls os.listdir(), which apparently *does* store the entire list internally, defeating the whole purpose of iglob()

Gumnos, Roger Erens, andrei.avk, gvanrossum, r.david.murray, roysmith, serhiy.storchaka, steven.dapranoĬreated on 01:09 by roysmith, last changed 19:36 by andrei.avk. Iglob() has misleading documentation (does indeed store names internally)

Python glob documentation