5

I am in the middle of refactoring a huge py module to packages - to not break existing code I moved its contents to package/__init__.py module (Adding code to __init__.py) and went on splitting it from there. I noticed at some point that in my tracebacks I get:

Traceback (most recent call last): File "<string>", line 656, in DoItemMenu File "bash\balt.py", line 2109, in PopupMenu link.AppendToMenu(menu,parent,*args) File "bash\balt.py", line 2225, in AppendToMenu for link in self.links: link.AppendToMenu(subMenu,window,data) ...

where the lines in File "<string>" correspond to the particular package/__init__.py module. Moreover PyCharm's debugger displays a "frame not available" line and does not step into the lines in the __init__.py. Why? Is it related to the import pattern?

The code is imported by a launcher class:

class UnicodeImporter(object): def find_module(self,fullname,path=None): if isinstance(fullname,unicode): fullname = fullname.replace(u'.',u'\\') exts = (u'.pyc',u'.pyo',u'.py') else: fullname = fullname.replace('.','\\') exts = ('.pyc','.pyo','.py') if os.path.exists(fullname) and os.path.isdir(fullname): return self for ext in exts: if os.path.exists(fullname+ext): return self def load_module(self,fullname): if fullname in sys.modules: return sys.modules[fullname] else: sys.modules[fullname] = imp.new_module(fullname) if isinstance(fullname,unicode): filename = fullname.replace(u'.',u'\\') ext = u'.py' initfile = u'__init__' else: filename = fullname.replace('.','\\') ext = '.py' initfile = '__init__' if os.path.exists(filename+ext): try: with open(filename+ext,'U') as fp: mod = imp.load_source(fullname,filename+ext,fp) sys.modules[fullname] = mod mod.__loader__ = self return mod except: print 'fail', filename+ext raise mod = sys.modules[fullname] mod.__loader__ = self mod.__file__ = os.path.join(os.getcwd(),filename) mod.__path__ = [filename] #init file initfile = os.path.join(filename,initfile+ext) if os.path.exists(initfile): with open(initfile,'U') as fp: code = fp.read() exec code in mod.__dict__ return mod
Improve this question
8
  • 2
    It means that Python was told to compile that module from a string; that probably is PyCharm's fault, not Python. Commented Nov 28, 2014 at 17:38
  • How exactly are you running this code at the moment? Commented Nov 28, 2014 at 17:40
  • @MartijnPieters: here is the launcher - I am almost certain that before I copy pasted the file in basher/__init__.py it did not load a string though... Commented Nov 28, 2014 at 21:50
  • No, I meant that PyCharm is instructing the Python interpreter to run that file from a string rather than asking Python to import it as a module. I didn't say anything about the module itself doing anything. Commented Nov 28, 2014 at 21:52
  • @MartijnPieters: It happens also when I directly run the launcher - I mentioned Pycharm cause it seems to me that it's the same reason it can't step through the code in __init__.py Commented Nov 28, 2014 at 22:38

1 Answer 1

Reset to default
5

The code is not imported in a traditional manner; instead the launcher code uses an exec statement for loading __init__.py files.

Paring down the flow in the launcher load_module() function for a package (so not a path to a module) you get this:

# the fullname module isn't yet loaded sys.modules[fullname] = imp.new_module(fullname) initfile = '__init__' # or u'__init__' if a unicode path was used # if no .py file was found, so not a module mod = sys.modules[fullname] mod.__loader__ = self mod.__file__ = os.path.join(os.getcwd(),filename) mod.__path__ = [filename] #init file initfile = os.path.join(filename,initfile+ext) if os.path.exists(initfile): with open(initfile,'U') as fp: code = fp.read() exec code in mod.__dict__ return mod

This creates an empty module object, loads the source manually and executes it as a string, passing in the module namespace as the globals for the executed code. The resulting code object is always going to list <string> in tracebacks:

>>> import imp >>> mod = imp.new_module('foo.bar') >>> mod.__file__ = r'C:\some\location\foo\bar' >>> mod.__path__ = [r'foo\bar'] >>> exec 'raise ValueError("oops")' in mod.__dict__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> ValueError: oops

Because there is no filename associated with the code, PyCharm cannot find the original source either.

The work-around is to use the compile() function to create a code object first, and attaching a filename to that:

>>> exec compile('raise ValueError("oops")', r'C:\some\location\foo\bar\__init__.py', 'exec') in mod.__dict__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\some\location\foo\bar\__init__.py", line 1, in <module> ValueError: oops

Note that I included __init__.py in the filename; translating that back to the launcher you'd use:

if os.path.exists(initfile): with open(initfile,'U') as fp: code = fp.read() exec compile(code, initfile, 'exec') in mod.__dict__
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Yep the launcher code was beyond me - will go through and digest all this and get back to you - thanks :)
Just to be sure I got it - using the compile() function will only do good in my scenario (remember I am in the middle of refactoring so these files are huge) ? Btw, I had noticed that there is no pyc file for the __init__.py, just did not mention this in the question - now I know why.
exec has to compile anyway, but by using an explicit compile() call you get to give the resulting code object a filename. It'll be helpful for all code launched by the launcher.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.