Timeline for Why don't research papers that mention custom software release the source code?
Current License: CC BY-SA 3.0
27 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 27, 2022 at 0:17 | comment | added | gnasher729 | If you have the source code and re-run it and get the same results, there is no guarantee at all that the results are correct. However, it tells you that the author hasn’t been blatantly faking their results which would be the pure minimum. | |
| Oct 17, 2013 at 16:51 | history | edited | cbeleites | CC BY-SA 3.0 | added 760 characters in body |
| Dec 19, 2012 at 15:35 | comment | added | Giorgio | @Andreas F.: 'I classify code as part of the "here are my methods" section.': Your classification is wrong: a paper should describe an algorithm in such way that anyone can implement it by reading the paper only (without looking at its reference implementation). | |
| Dec 19, 2012 at 15:08 | history | edited | math | CC BY-SA 3.0 | Another reason may be, that the author may want to publish follow up work. |
| Oct 30, 2012 at 18:23 | history | made wiki | Post Made Community Wiki by Ryan Schmidt | ||
| Oct 27, 2012 at 1:27 | comment | added | DeveloperDon | Another issue for students is that of plagiarism. If you post an original project from a class, if someone copies it there may a mess as people question whose work is it? I hope most cases like this are proven in favor of the non-cheater, but expect sometimes even blameless people are punished or at best, experience serious stress. | |
| Oct 27, 2012 at 1:24 | comment | added | DeveloperDon | I like the comments about student code perhaps not being the best representation to start a career, but there are several counter examples. If we look to other fields, student projects can be well appreciated (for example symphony #1 "The Classical" by Prokofiev might be his most popular among the general public). | |
| Oct 26, 2012 at 18:27 | comment | added | Andres F. | @Tacroy I disagree; I think it is a fundamental misunderstanding of the role of code in scientific research. Lack of code availability is like lack of availability of a mathematical proof: sure, it's awesome if you can derive your own proof. But you must be able to review the existing proof as well. Anyway, I suggest you read the Science Code Manifesto :) | |
| Oct 26, 2012 at 18:19 | comment | added | Tacroy | @AndresF. At that point you should be asking for researchers in other fields to supply videos taken by lab security cameras. The thing is, you don't want to know their method in the detail that the source code provides, because when you go to replicate their findings you need to be coming at it from more of a clean slate perspective. | |
| Oct 26, 2012 at 18:18 | comment | added | Andres F. | @Tacroy I also do not expect the actual program text to be included in the paper; just a link to it. But if it IS used in your research in any meaningful way, it must be available for review. | |
| Oct 26, 2012 at 18:15 | comment | added | Andres F. | @Tacroy I classify code as part of the "here are my methods" section. The purpose of having source code available is to review said methods, not to blindly run it again (which except for the case of blatant fraud, will surely produce the same results). Code IS specification, i.e. the actual method. It is NOT an afterthought; otherwise you could skip it entirely without harm to your research. Note that having a detailed step-by-step specification in pseudocode is almost the same as making the actual code available, but it seems less effort to just make the source available. | |
| Oct 26, 2012 at 17:24 | comment | added | Tacroy | @AndresF. Code is absolutely the least important thing in a paper. A paper is "Here is what I did; here are my methods; here are my results". The code is a codification of the method, and it should produce the exact same results. If you want to reproduce the paper's results but use the paper's code, you haven't reproduced anything; what you're supposed to do is read their methods section, come up with your own implementation, and then write a paper about it when you can't reproduce their results. | |
| Oct 26, 2012 at 14:30 | comment | added | Michael Shopsin | An important exception is that code by government scientists is inherently open source, go to the NASA-GISS website if you want to download some | |
| Oct 26, 2012 at 9:15 | vote | accept | alecail | ||
| Oct 26, 2012 at 2:07 | comment | added | Paul | @Andres F. While I agree broken code could invalidate results, in practice I suspect few professors read code produced by their assistants and even fewer write code or have a clue about what is good or bad about code or think of code as anything other than a tedious one way human to computer medium. The peer review process doesn't typically examine code and generally couldn't if they wanted to, between the problems of code literacy and time constraints. | |
| Oct 25, 2012 at 20:53 | comment | added | Andres F. | @Paul But code is important. Even if the research is not code-related, if code was used in any meaningful capacity to compute or validate the research, then it is a fundamental part of said research. Otherwise it's like publishing a new mathematical result and omitting the proof. Code IS the proof. If, on the contrary, code played no important role, then why mention it at all? Best to omit it altogether, or at least relegate it to a minor footnote. | |
| Oct 25, 2012 at 20:22 | comment | added | Paul | I would suggest adding: (1) code is student-written (raises a variety of issues, including: do you want code you hastily wrote in your college years made public without warning?) ; (2) code isn't important, only results are important; and (3) research competitors should go write their own code. I've heard these concerns, though I do not necessarily agree. | |
| Oct 25, 2012 at 20:07 | comment | added | Alex Gordon | your second to the last point is the strongest | |
| Oct 25, 2012 at 19:29 | comment | added | JustinC | One reason why it may not always be released, which I see as acceptable, is the tooling available in stock form may not permit just anyone to build it at a whim. It may require a custom compiler, licsensed 3rd party libraries, or even a custom OS or custom hardware; which without, makes the code to anyone else nothing more than pseudo-code. | |
| Oct 25, 2012 at 18:52 | comment | added | Etienne Racine | Point 8 (Naked code can be shocking) makes me think about the CRAPL idea | |
| Oct 25, 2012 at 18:47 | comment | added | robi | As a reviewer I tend to favour papers that link to the source or at least give some reason for not publishing. Some of the mentioned reasons still hold today but it is also takes some time for the research community to catch up | |
| Oct 25, 2012 at 17:56 | comment | added | Jimbo Jonny | GREAT post. I'd also add that sometimes separate scientists recreating the software on their own is PART of the repeatability of the experiment. If it only works the way 1 person coded it, but not the way others code it...then the results can be called into question and errors can be identified. | |
| Oct 25, 2012 at 16:28 | comment | added | Joe | In complete agreement with Kaz -- if to peer review the research you need something that's not available (data, code, etc), it shouldn't be accepted by a peer-reviewed journal. Almost all of the arguments that DeveloperDon mentions has pointed out holds true for data release, too ... yet there's now a rather big movement in recent years towards it. | |
| Oct 25, 2012 at 15:44 | comment | added | Kaz | If the paper depends on the source code, then it ought not be published. If you can't publish the code, you can't publish the paper. A paper which says "our program does these wonderful things", and you cannot evaluate the paper without running that program, then the paper borders on being an advertizing brochure for some software. | |
| Oct 25, 2012 at 15:34 | comment | added | James | +1 also, the code may not be theirs to publish, it probably belongs to the institution that pays them to research things | |
| Oct 25, 2012 at 11:20 | history | edited | DeveloperDon | CC BY-SA 3.0 | added 27 characters in body |
| Oct 25, 2012 at 10:48 | history | answered | DeveloperDon | CC BY-SA 3.0 |