Repository navigation
pkgutil.walk_packages "prefix" option docs are misleading #77391
Description
| BPO | 33210 |
|---|---|
| Nosy | @ncoghlan, @ericsnowcurrently, @cykerway |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-04-03.02:25:38.194>
labels = ['type-bug', '3.7', 'docs']
title = 'pkgutil.walk_packages "prefix" option docs are misleading'
updated_at = <Date 2018-04-06.14:30:40.552>
user = 'https://github.com/cykerway'bugs.python.org fields:
activity = <Date 2018-04-06.14:30:40.552>
actor = 'ncoghlan'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2018-04-03.02:25:38.194>
creator = 'cykerway'
dependencies = []
files = ['47519']
hgrepos = []
issue_num = 33210
keywords = []
message_count = 4.0
messages = ['314850', '314960', '315017', '315018']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'docs@python', 'eric.snow', 'cykerway']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33210'
versions = ['Python 3.6', 'Python 3.7']
Activity
cykerway commentedon Apr 3, 2018
The current implementation of
pkgutil.walk_packages()is confusing. Users may be given incomplete results while no exception is raised if they don't explicitly provide theprefixparameter. The doc says:But the fact is,
prefixis not merely an output formatter at all. This function cannot handle an empty prefix (which is the default) and will often encounter import errors which are then simply ignored without anonerrorfunction (which is default again).See test program for details.
Doc:
https://docs.python.org/3.6/library/pkgutil.html#pkgutil.walk_packages
Source:
https://github.com/python/cpython/blob/3.6/Lib/pkgutil.py
cykerway commentedon Apr 4, 2018
Update test program.
ncoghlan commentedon Apr 6, 2018
I think this is actually two distinct problems, one documentation one (which should be addressed in the online docs for all currently maintained versions), and one actual functional issue.
The documentation issue is the one you've reported: in order for the recursive descent to work in walk_packages given the current algorithm, then the combination of the given prefix, and the current global import configuration must allow that package to actually be imported. While there is a note about that limitation, it's currently thoroughly unclear.
The functional issue is two-fold:
pkgutil.iter_modules() doesn't identify PEP-420 namespace packages correctly (it ignores them as not being potential packages)
The recursive import to check pkg.__path__ uses a name based global __import__ rather than the more state independent https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly technique
It's that second problem that introduces the "prefix must be set to get useful output" behaviour that you're currently seeing.
ncoghlan commentedon Apr 6, 2018
https://bugs.python.org/issue29258 is an existing issue for the PEP-420 limitation (it also notes why fixing that implicitly is a potential problem)
I've retitled this issue to be specifically about the misleading docs for the "prefix" option (and updated the metadata accordingly).
"Recursive descent in pkgutil.walk_packages depends on sys.path" would be a reasonable title for an issue to actually fix the operation to use a better algorithm based on newer importlib APIs, but I haven't filed that myself.
[-]pkgutil.walk_packages gives incomplete results[/-][+]pkgutil.walk_packages "prefix" option docs are misleading[/+]