Timeline for How to separate a solver from computational experiments in a correct way?
Current License: CC BY-SA 3.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 21, 2016 at 18:53 | vote | accept | Dmitry Kabanov | ||
| Sep 28, 2016 at 10:51 | history | edited | Dmitry Kabanov | CC BY-SA 3.0 | added 1551 characters in body |
| Sep 24, 2016 at 1:13 | answer | added | Geoff Oxberry | timeline score: 3 | |
| Sep 23, 2016 at 17:15 | answer | added | nicoguaro♦ | timeline score: 0 | |
| Sep 23, 2016 at 14:22 | answer | added | Dmitry Kabanov | timeline score: 1 | |
| Sep 16, 2016 at 14:48 | comment | added | Charles | Actually, I would recommend not keeping the executable, that is what git is for (which I consider irreplaceable and an absolute must). You can commit your project at cornerstones when you benchmark particular problems. If you see that your code cannot reproduce a previous experiment, you can compare your code between commits. Again, I'm not an expert in this area though. | |
| Sep 16, 2016 at 14:12 | comment | added | Philip | Are you running a built executable or scripting source code? I think @Charlie was suggesting, if you are building your code and the executable is small, you could build and keep executables per each experiment. This has the added benefit (especially when combined with git) of ensuring highly reproducible research. Modifying this idea for scripting gets you to Kirill's solution, i.e. just make the "build" step "copy all the code into another permanent location." You can keep your build script in version control too, to further ensure reproducibility. | |
| Sep 12, 2016 at 6:43 | history | tweeted | twitter.com/StackSciComp/status/775223217434820608 | ||
| Sep 10, 2016 at 20:50 | comment | added | Dmitry Kabanov | @choward, yes, input is handled through configuration file, not through the code modification. In my question I'm talking about the genuine code modification due to a change of functionality. | |
| Sep 10, 2016 at 19:29 | comment | added | spektr | Does each experiment actually need code changes or could you describe a new experiment via a new configuration file? Then you could pass the config file in as an input without having to make code changes. | |
| Sep 10, 2016 at 17:32 | comment | added | Kirill | @DmitryKabanov I meant something like this: make a separate copy of the current code repository, run the experiment in the new copy, insert the revision number into the outputs. New changes in code would go in the main repository and never affect other copies. Old experiments can be repeated by checking out specific revisions—this also helps with regression testing. Code changes that cannot be inserted into the main solver might be maintained on separate branches. Perhaps the various git workflow and reproducible research with git tutorials might help with ideas too. | |
| Sep 10, 2016 at 16:40 | comment | added | Charles | I'm not an expert in this area, but I would pipeline the 1) build and run process and 2) experiment test runs. This way, after you make changes you can build, run and check if the change affects your previously benchmarked experiments. | |
| Sep 10, 2016 at 10:54 | comment | added | Dmitry Kabanov | @Charlie, this is exactly what I'm trying to do. The question is how to handle code modifications for several experiments running simulatenously such that the earlier started experiment doesn't break during execution. | |
| Sep 10, 2016 at 10:52 | comment | added | Dmitry Kabanov | @Kirill, yes, I decouple the components of my code using strategy pattern. How to tie an experiment to a particular VC version (I use git) in an optimal way? I'd like the scheme to be as simple and transparent as it can be. | |
| Sep 10, 2016 at 5:57 | comment | added | Charles | How similar are your experiments? Will they be tied together in the end? I would suggest thinking about how much code, if any, you can reuse from each of these experiments, if the answer is: a lot, then I would suggest maintaining one project, and keeping this source code separate from the experiments (your second diagram). | |
| Sep 10, 2016 at 4:14 | comment | added | Kirill | Do you mean to say the individual copies of solvers weren't version-controlled? It's probably worth looking at how tightly coupled the different components of the solver are, and whether they can be decoupled better, and this could just indicate a flaw in the program's design; see also strategy pattern, which may or may not be helpful to you. The issue with modifying running code is unusual, and could be solved with proper use of VC (tie each experiment to specific VC revision). | |
| Sep 9, 2016 at 23:02 | history | asked | Dmitry Kabanov | CC BY-SA 3.0 |