Better ways to organize data

Question

So I have the following loop:

for temperature in [-40,25,85]: for voltage in [24, 28, 32]: for power in [50,100,200]: #measurement 1 ... #measurement 2 ...

There is a question of how to organize the measured data. Up to now I've been creating a separate folder at each loop for each loop variable, so if I want to grab the data for measurement 1 at -40°C, 28V and 100W, I'll just go to ./-40/28/100. It's been working reasonably well but I cannot help wondering whether there is a better way.

I would think of using a database (if there are more than those 27 possible combinations) or, using just one folder whose name is composed of the 3 components, for example: ./-40,28,100/ — DWilches
– DWilches, Commented Nov 20, 2013 at 21:07
At most there could be 3 temperature points, 5 voltage points and maybe 10 power points, leading to 150 combinations. Never done any work on database before, where to begin then, assuming the convenience it brings outweigh the effort to set it up. — Tian He
– Tian He, Commented Nov 20, 2013 at 21:10
The easiest database to setup (IMO) is SQLite, you can try it but that will mean learning the query language: SQL. That is the best solution when you want to organize and retrieve data. Otherwise you'll end up with 150 folders. — DWilches
– DWilches, Commented Nov 20, 2013 at 21:16
How's PyTables for this application? I'm browsing through its main features now and it seems promising. — Tian He
– Tian He, Commented Nov 20, 2013 at 21:32

tmj · Accepted Answer · 2013-11-20 21:29:49Z

3

import json dictionary = {} for temperature in [-40,25,85]: for voltage in [24, 28, 32]: for power in [50,100,200]: #measurement 1 #measurement 2.. dictionary[str((temperature,voltage,power))] = (measurement1,measurement2,...) f = open(".../somefile.txt",'w') f.write(json.dumps(dictionary,indent=1,sort_keys=True)) f.close()

Maybe JSON can help you.

edited Nov 20, 2013 at 21:29

answered Nov 20, 2013 at 21:13

tmj

1,8982 gold badges22 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

DWilches Over a year ago

But if the amount of data is big the memory usage could be a problem. Also file reading can be a performance problem depending on how often the data will be read from the file and its size.

tmj Over a year ago

The data isn't that big, is it? The OP said 150 or so combinations, so I'm guessing pretty small files.

DWilches Over a year ago

He said 150 combination (or folders), but made no statement about the data size. Maybe he can clarify this to us?

Tian He Over a year ago

Each measurement gives a numerical array the dimension of which normally doesn't exceed 10X10, which will then be saved to a file. So yes, those would be pretty small files.

DWilches Over a year ago

In that case @tMJ solution seems reasonable

Tian He · Accepted Answer · 2013-11-26 17:37:46Z

I've found that the pandas module serves my purpose well. By using it I can save all controlling variables and measurement results in one table, which later on can be easily accessible for all kinds of slicing, grouping and plotting.

Collectives™ on Stack Overflow

Better ways to organize data

2 Answers 2

5 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Related