45

Update: dicts retaining insertion order is guaranteed for Python 3.7+

I want to use a .py file like a config file. So using the {...} notation I can create a dictionary using strings as keys but the definition order is lost in a standard python dictionary.

My question: is it possible to override the {...} notation so that I get an OrderedDict() instead of a dict()?

I was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work, but it doesn't.

Eg:

dict = OrderedDict dictname = { 'B key': 'value1', 'A key': 'value2', 'C key': 'value3' } print dictname.items() 

Output:

[('B key', 'value1'), ('A key', 'value2'), ('C key', 'value3')] 
6
  • I assume that the Output mentioned here is what you would like - not what happens ? Commented Aug 16, 2016 at 18:53
  • 6
    FYI to people stumbling on this 5 year old question in 2016: as of python 3.6 all dicts retain insertion order, so going forward none of these hacks will be needed. Commented Dec 17, 2016 at 4:39
  • @NickSweeting docs.python.org/3/whatsnew/3.6.html#new-dict-implementation says "The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon". Commented Jun 20, 2017 at 22:35
  • @Samuel Santana the way I read the rest of the sentence that you quoted the start of suggests that this new ordering-preservation is the long term desired semantics for the language, without committing to it, right now. Commented Jan 9, 2018 at 16:37
  • 3
    As of 3.7, those semantics can be relied upon Commented Aug 25, 2018 at 16:35

7 Answers 7

80

Here's a hack that almost gives you the syntax you want:

class _OrderedDictMaker(object): def __getitem__(self, keys): if not isinstance(keys, tuple): keys = (keys,) assert all(isinstance(key, slice) for key in keys) return OrderedDict([(k.start, k.stop) for k in keys]) ordereddict = _OrderedDictMaker() 
from nastyhacks import ordereddict menu = ordereddict[ "about" : "about", "login" : "login", 'signup': "signup" ] 

Edit: Someone else discovered this independently, and has published the odictliteral package on PyPI that provides a slightly more thorough implementation - use that package instead

Sign up to request clarification or add additional context in comments.

8 Comments

shudder - i can see why you call that a hack - please - don't use that in production
This is genius. Evil genius.
@Eric I came here from reddit reddit.com/r/Python/comments/4xyfh7/…
To help decipher how this works, see stackoverflow.com/questions/2936863/…
@Eric it doesn't work if one key: menu = ordereddict["about" : "about"]; but it seems easy to fix it.
|
40

To literally get what you are asking for, you have to fiddle with the syntax tree of your file. I don't think it is advisable to do so, but I couldn't resist the temptation to try. So here we go.

First, we create a module with a function my_execfile() that works like the built-in execfile(), except that all occurrences of dictionary displays, e.g. {3: 4, "a": 2} are replaced by explicit calls to the dict() constructor, e.g. dict([(3, 4), ('a', 2)]). (Of course we could directly replace them by calls to collections.OrderedDict(), but we don't want to be too intrusive.) Here's the code:

import ast class DictDisplayTransformer(ast.NodeTransformer): def visit_Dict(self, node): self.generic_visit(node) list_node = ast.List( [ast.copy_location(ast.Tuple(list(x), ast.Load()), x[0]) for x in zip(node.keys, node.values)], ast.Load()) name_node = ast.Name("dict", ast.Load()) new_node = ast.Call(ast.copy_location(name_node, node), [ast.copy_location(list_node, node)], [], None, None) return ast.copy_location(new_node, node) def my_execfile(filename, globals=None, locals=None): if globals is None: globals = {} if locals is None: locals = globals node = ast.parse(open(filename).read()) transformed = DictDisplayTransformer().visit(node) exec compile(transformed, filename, "exec") in globals, locals 

With this modification in place, we can modify the behaviour of dictionary displays by overwriting dict. Here is an example:

# test.py from collections import OrderedDict print {3: 4, "a": 2} dict = OrderedDict print {3: 4, "a": 2} 

Now we can run this file using my_execfile("test.py"), yielding the output

{'a': 2, 3: 4} OrderedDict([(3, 4), ('a', 2)]) 

Note that for simplicity, the above code doesn't touch dictionary comprehensions, which should be transformed to generator expressions passed to the dict() constructor. You'd need to add a visit_DictComp() method to the DictDisplayTransformer class. Given the above example code, this should be straight-forward.

Again, I don't recommend this kind of messing around with the language semantics. Did you have a look into the ConfigParser module?

2 Comments

Yes I will use ConfigParser...but your solution is illuminating. Thank you very much.
@fdb - before you think of changing the language semantics - think about the principle 'Explicit is better than implicit' - if you try to override '{}' or hide to avoid having to type 'OrderedDict' - you will end up making your code far more difficult to read for others - of for yourself 6 months down the line. Just type 'OrderedDict' - it is understood, and does what you want - more typing, but improved readability.
13

OrderedDict is not "standard python syntax", however, an ordered set of key-value pairs (in standard python syntax) is simply:

[('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')] 

To explicitly get an OrderedDict:

OrderedDict([('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')]) 

Another alternative, is to sort dictname.items(), if that's all you need:

sorted(dictname.items()) 

6 Comments

my question isn't if OrderedDict is "standard python syntax" but if is possible to override the {...} notation
@fdb: In Python {} creates a dict object, which is unordered by definition. You can of course define your own language with {} denoting an orderd dictionary. You can even write a small wrapper that translates your new language to Python. Is this what you actually want?
@SvenMarnach: yes! but was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work.
@fdb: That only works if you make your dictionary by calling dict()
before you think of changing the language semantics - think about the principle 'Explicit is better than implicit' - if you try to override '{}' or hide to avoid having to type 'OrderedDict' - you will end up making your code far more difficult to read for others - of for yourself 6 months down the line. Just type 'OrderedDict' - it is understood, and does what you want - more typing, but improved readability.
|
6

As of python 3.6, all dictionaries will be ordered by default.

Insertion order is always preserved in the new dict implementation:

>>>x = {'a': 1, 'b':2, 'c':3 } >>>list(x.keys()) ['a', 'b', 'c'] 

As of python 3.6 **kwargs order [PEP468] and class attribute order [PEP520] are preserved. The new compact, ordered dictionary implementation is used to implement the ordering for both of these.

2 Comments

Maybe it has to do with what docs.python.org/3/whatsnew/3.6.html#new-dict-implementation says: "The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon". Still, I found the information interesting, so here's an upvote!
Update: Insert order being preserved is now standard in 3.7 and can be relied upon.
5

What you are asking for is impossible, but if a config file in JSON syntax is sufficient you can do something similar with the json module:

>>> import json, collections >>> d = json.JSONDecoder(object_pairs_hook = collections.OrderedDict) >>> d.decode('{"a":5,"b":6}') OrderedDict([(u'a', 5), (u'b', 6)]) 

3 Comments

"Impossible" might be a bit too strong a word -- see my answer.
@Sven: Yes, I totally enjoyed your answer! :) I think I will let my wording stand, though. Please adjust your understanding of "impossible" in this context to match reality ;)
json.loads and json.load have been also updated since Python 3.1 with support for object_pairs_hook docs.python.org/3.4/library/json.html#json.load
5

The one solution I found is to patch python itself, making the dict object remember the order of insertion.

This then works for all kind of syntaxes:

x = {'a': 1, 'b':2, 'c':3 } y = dict(a=1, b=2, c=3) 

etc.

I have taken the ordereddict C implementation from https://pypi.python.org/pypi/ruamel.ordereddict/ and merged back into the main python code.

If you do not mind re-building the python interpreter, here is a patch for Python 2.7.8: https://github.com/fwyzard/cpython/compare/2.7.8...ordereddict-2.7.8.diff .A

1 Comment

As of 2016/12, the Pypy implementation will become the standard python dict implementation, nice job predicting this 2 years in advance!
0

If what you are looking for is a way to get easy-to-use initialization syntax - consider creating a subclass of OrderedDict and adding operators to it that update the dict, for example:

from collections import OrderedDict class OrderedMap(OrderedDict): def __add__(self,other): self.update(other) return self d = OrderedMap()+{1:2}+{4:3}+{"key":"value"} 

d will be- OrderedMap([(1, 2), (4, 3), ('key','value')])


Another possible syntactic-sugar example using the slicing syntax:

class OrderedMap(OrderedDict): def __getitem__(self, index): if isinstance(index, slice): self[index.start] = index.stop return self else: return OrderedDict.__getitem__(self, index) d = OrderedMap()[1:2][6:4][4:7]["a":"H"] 

1 Comment

Note: Both of these violate the expectations of their operators in extreme ways. Both __add__ and __getitem__ are intended to be non-mutating, and slicing support is expected to be an aggregate form of indexing support, not a completely unrelated behavior. Violating those expectations is asking for maintainability nightmare. The slice hack is much better used to achieve the result given in the accepted answer, where it's a factory object that makes a normal OrderedDict, not an OrderedDict replacement with ongoing weird behaviors.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.