855

Python gives us the ability to create 'private' methods and variables within a class by prepending double underscores to the name, like this: __myPrivateMethod(). How, then, can one explain this

>>>> class MyClass: ... def myPublicMethod(self): ... print 'public method' ... def __myPrivateMethod(self): ... print 'this is private!!' ... >>> obj = MyClass() >>> obj.myPublicMethod() public method >>> obj.__myPrivateMethod() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: MyClass instance has no attribute '__myPrivateMethod' >>> dir(obj) ['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod'] >>> obj._MyClass__myPrivateMethod() this is private!!

What's the deal?!

I'll explain this a little for those who didn't quite get that.

>>> class MyClass: ... def myPublicMethod(self): ... print 'public method' ... def __myPrivateMethod(self): ... print 'this is private!!' ... >>> obj = MyClass()

I create a class with a public method and a private method and instantiate it.

Next, I call its public method.

>>> obj.myPublicMethod() public method

Next, I try and call its private method.

>>> obj.__myPrivateMethod() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: MyClass instance has no attribute '__myPrivateMethod'

Everything looks good here; we're unable to call it. It is, in fact, 'private'. Well, actually it isn't. Running dir() on the object reveals a new magical method that Python creates magically for all of your 'private' methods.

>>> dir(obj) ['_MyClass__myPrivateMethod', '__doc__', '__module__', 'myPublicMethod']

This new method's name is always an underscore, followed by the class name, followed by the method name.

>>> obj._MyClass__myPrivateMethod() this is private!!

So much for encapsulation, eh?

In any case, I'd always heard Python doesn't support encapsulation, so why even try? What gives?

Improve this question
11
  • 44
    Same is true for Java or C# if you use reflection (which is somehow what you doing there). Commented Feb 22, 2015 at 10:24
  • 8
    It was build for Unit Testing purpose, so you can use that "hack" in order to unit test the private methods of your class from outside. Commented Apr 12, 2016 at 16:10
  • 39
    Isn't testing private methods an anti-pattern? Private methods will be used in some public method for sure else it's just unused forever. And the right way to test private methods (based on my learning so far from ThoughtWorks) is that you write tests for public methods only that covers all cases. If that works fine, you don't need to test private methods from outside at all. Commented Jan 22, 2017 at 23:19
  • 18
    @VishnuNarang: Yeah, that's whats often teached. But As always, an almost "religious" approach of "always do this, never do that" is the only thing that "never" is good. If unit tests are "only" used for regression tests or testing public API, you don't need to test privates. But if you do unit test driven development, there are good reasons to test privat methods during development (for example when it's hard to mock certain unusual / extreme parameters through the public interface). Some languages / unit test environments don't let you do this, which IMHO is not good. Commented Feb 13, 2017 at 14:25
  • 10
    @MarcoFreudenberger I see your point. I do have experience in unit test driven development. Often when it becomes difficult to mock parameters, most often it's resolved by changing and improving the design. I'm yet to come across a scenario where the design is perfect and still unit testing is extremely difficult to avoid testing private methods. I'll look out for such cases. Thanks. I'd appreciate if you could maybe share one scenario off the top of your head to help me understand. Commented Feb 13, 2017 at 18:56

12 Answers 12

Reset to default
716

The name scrambling is used to ensure that subclasses don't accidentally override the private methods and attributes of their superclasses. It's not designed to prevent deliberate access from outside.

For example:

>>> class Foo(object): ... def __init__(self): ... self.__baz = 42 ... def foo(self): ... print self.__baz ... >>> class Bar(Foo): ... def __init__(self): ... super(Bar, self).__init__() ... self.__baz = 21 ... def bar(self): ... print self.__baz ... >>> x = Bar() >>> x.foo() 42 >>> x.bar() 21 >>> print x.__dict__ {'_Bar__baz': 21, '_Foo__baz': 42}

Of course, it breaks down if two different classes have the same name.

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

docs.python.org/2/tutorial/classes.html. Section:9.6 on Private variables and class-local references.
For those of us too lazy to scroll/search: Section 9.6 direct link
You should put a single underscore to specify that the variable should be considered as private. Again, this does not prevent someone from actually accessing that.
Guido answered this question - "main reason for making (nearly) everything discoverable was debugging: when debugging you often need to break through the abstractions" - I added it as comment because it's too late -- too many answers.
If you go by the "prevent deliberate access" criterion, most OOP languages don't support truly private members. For example in C++ you have raw access to memory and in C# trusted code can use private reflection.
|
235

When I first came from Java to Python I hated this. It scared me to death.

Today it might just be the one thing I love most about Python.

I love being on a platform, where people trust each other and don't feel like they need to build impenetrable walls around their code. In strongly encapsulated languages, if an API has a bug, and you have figured out what goes wrong, you may still be unable to work around it because the needed method is private. In Python the attitude is: "sure". If you think you understand the situation, perhaps you have even read it, then all we can say is "good luck!".

Remember, encapsulation is not even weakly related to "security", or keeping the kids off the lawn. It is just another pattern that should be used to make a code base easier to understand.

Improve this answer

11 Comments

APIs are actually a really good example of why encapsulation matters and when private methods would be preferred. A method intended to be private may go away, change signature, or worst of all change behavior -- all without warning -- on any subsequent new version. Will your smart, adult team member really remember that she accessed an intended-to-be-private method a year from now when you update? Will she even be working there anymore?
I disagree with the argument. In production code, I would most probably never use an API that has a bug that makes me change public members to make it "work". An API should work. If it doesn't, I would file a bug report or make the same API myself. I don't like the philosophy and I am not very fond of Python, although its syntax makes it fun to write smaller scripts in...
Java has Method.setAccessible and Field.setAccessible. Also scary?
The enforcement in Java and C++ is not because Java distrusts the user whilst Python does. Its because the compiler and/or vm can make various assumptions when dealing with how it goes about its business if it knows this information, for instance C++ can skip an entire layer of indirection by using regular old C calls instead of virtual calls, and that matters when your working on high performance or high precision stuff. Python by its nature can't really make good use of the information without compromising its dynamism. Both languages are aiming at different things, so neither are "wrong"
We dont make methods private and public for security's sake. We do them to prevent users from fatally injuring state and writing programs using volatile methods.. A public method represents a contract, if I give you A you will output B. Private methods do not represent a contract. I can change the logic that turns A into B as much as I need to so long as the interface to the interaction still fulfills the contract. If I give you access to private methods, you will use them. Then when I change implementation, your program breaks. Doesn't happen with public methods that form a contract.
|
220

Example of a private function

import re import inspect class MyClass: def __init__(self): pass def private_function(self): try: function_call = inspect.stack()[1][4][0].strip() # See if the function_call has "self." in the beginning matched = re.match( '^self\.', function_call) if not matched: print 'This is a private function. Go away.' return except: print 'This is a private function. Go away.' return # This is the real function, only accessible inside the class # print 'Hey, welcome in to the function.' def public_function(self): # I can call a private function from inside the class self.private_function() ### End ###
Improve this answer

15 Comments

self = MyClass() self.private_function(). :D Of course it doesn't work in classes, but you just have to define a custom function: def foo(self): self.private_function()
Just in case it wasn't clear: never do this in real code ;)
@ThorSummoner Or just function_call.startswith('self.').
inspect.stack()[1][4][0].strip() <- what are those 1, 4, and 0 magic numbers?
This can be defeated quite easily by doing self = MyClass(); self.private_function() and it fails when called using x = self.private_function() inside a method.
|
166

From Dive Into Python, 3.9. Private functions:

Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them seem inaccessible by their given names. You can access the __parse method of the MP3FileInfo class by the name _MP3FileInfo__parse. Acknowledge that this is interesting, then promise to never, ever do it in real code. Private methods are private for a reason, but like many other things in Python, their privateness is ultimately a matter of convention, not force.

Improve this answer

4 Comments

or as Guido van Rossum put it: "we are all adults."
-1: this is just wrong. Double underscores is never meant to be used as private in first place. The answer from Alya below tells the real intention of name mangling syntax. The real convention is a single underscore.
Try with one underscore only and you will see the result you get. @nosklo
Single underscore is the default convention for "private" attributes. Double underscore is when you specifically want to avoid collisions in class inheritance.
118

The phrase commonly used is "we're all consenting adults here". By prepending a single underscore (don't expose) or double underscore (hide), you're telling the user of your class that you intend the member to be 'private' in some way. However, you're trusting everyone else to behave responsibly and respect that, unless they have a compelling reason not to (e.g., debuggers and code completion).

If you truly must have something that is private, then you can implement it in an extension (e.g., in C for CPython). In most cases, however, you simply learn the Pythonic way of doing things.

Improve this answer

2 Comments

so is there some sort of wrapper protocol that I'm supposed to use to access a protected variable?
There aren't "protected" variables any more than there are "private". If you want to access an attribute that starts with an underscore, you can just do it (but note that the author discourages this). If you must access an attribute that starts with a double underscore, you can do the name mangling yourself, but you almost certainly do not want to do this.
39

It's not like you absolutely can't get around privateness of members in any language (pointer arithmetics in C++ and reflections in .NET/Java).

The point is that you get an error if you try to call the private method by accident. But if you want to shoot yourself in the foot, go ahead and do it.

You don't try to secure your stuff by OO-encapsulation, do you?

Improve this answer

3 Comments

Not at all. I'm simply making the point that it's odd to give the developer an easy, and in my opinion way to magical, way of accessing 'private' properties.
Yeah, I just tried to illustrate the point. Making it private just says "you shouldn't access this directly" by making the compiler complain. But one wants to really really do it he can. But yes, it's easier in Python than in most other languages.
In Java, you actually can secure stuff via encapsulation, but that requires you to be smart and run the untrusted code in a SecurityManager, and be very careful. Even Oracle gets it wrong sometimes.
23

Important note:

Any identifier of the form __name (at least two leading underscores, at most one trailing underscore) is publicly replaced with _classname__name, where classname is the current class name with a leading underscore(s) stripped.

Therefore, __name is not accessible directly, but can be accessed as_classname__name.

This does not mean that you can protect your private data as it is easily accessible by changing the name of the variable.

Source:

"Private Variables" section in official documentation: https://docs.python.org/3/tutorial/classes.html#tut-private

Example

class Cat: def __init__(self, name='unnamed'): self.name = name def __print_my_name(self): print(self.name) tom = Cat() tom.__print_my_name() #Error tom._Cat__print_my_name() #Prints name
Improve this answer

3 Comments

This is not an important update, it has always been true. Replacing the 3 with a 2 in your link, docs.python.org/2/tutorial/classes.html#tut-private, reveals almost exactly the same text that you made your answer. NOT an update.
This is also untrue: __name is not "private", since it is accessible using _classname__name. Something accessible is NOT "private". There is no such thing as a "private" attribute, in any major computer language (AFAIK), it's as simple as that. You can say that local variables in a method are "private", but only if they are not accessible from outside the method in any way.
That is already explained in the answer. However, with that definition, and saying that "There is no such thing as a private attribute, in any major computer language", then we have to re-define "private" in programming languages. But until then, we can use what we have and call it private, since the official documents are calling that "private".
13

Similar behavior exists when module attribute names begin with a single underscore (e.g. _foo).

Module attributes named as such will not be copied into an importing module when using the from* method, e.g.:

from bar import *

However, this is a convention and not a language constraint. These are not private attributes; they can be referenced and manipulated by any importer. Some argue that because of this, Python can not implement true encapsulation.

Improve this answer

Comments

12

It's just one of those language design choices. On some level they are justified. They make it so you need to go pretty far out of your way to try and call the method, and if you really need it that badly, you must have a pretty good reason!

Debugging hooks and testing come to mind as possible applications, used responsibly of course.

Improve this answer

Comments

5

The most important concern about private methods and attributes is to tell developers not to call it outside the class and this is encapsulation. One may misunderstand security from encapsulation. When one deliberately uses syntax like that (below) you mentioned, you do not want encapsulation.

obj._MyClass__myPrivateMethod()

I have migrated from C# and at first it was weird for me too but after a while I came to the idea that only the way that Python code designers think about OOP is different.

Improve this answer

Comments

4

With Python 3.4, this is the behaviour:

>>> class Foo: def __init__(self): pass def __privateMethod(self): return 3 def invoke(self): return self.__privateMethod() >>> help(Foo) Help on class Foo in module __main__: class Foo(builtins.object) | Methods defined here: | | __init__(self) | | invoke(self) | | ---------------------------------------------------------------------- | Data descriptors defined here: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) >>> f = Foo() >>> f.invoke() 3 >>> f.__privateMethod() Traceback (most recent call last): File "<pyshell#47>", line 1, in <module> f.__privateMethod() AttributeError: 'Foo' object has no attribute '__privateMethod'

From 9.6. Private Variables:

Note that the mangling rules are designed mostly to avoid accidents; it still is possible to access or modify a variable that is considered private. This can even be useful in special circumstances, such as in the debugger.

Improve this answer

Comments

2

Why are Python's 'private' methods not actually private?

As I understand it, they can't be private. How could privacy be enforced?

The obvious answer is "private members can only be accessed through self", but that wouldn't work - self is not special in Python. It is nothing more than a commonly-used name for the first parameter of a function.

Improve this answer

1 Comment

You can enforce encapsulation by having your data-members live in a c-struct and have your python-class only hold a pointer to it. Many low-level-libraries work like that. Though that is not really a 'private method in python', more like 'method outside of the interpreter'.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.