1

I generate a set of data files. As the files are supposed to be readable, they text files (opposed to binary files).

To output information to my files, I used very comfortable std::ofstream object.

In the beginning, when the data to be exported was smaller, the time needed to write to the files was not noticeable. However, as the information to be exported has accumulated, it takes now around 5 minutes to generate them.

As I started being bothered by waiting, my question is obvious: Is there any faster alternative to std::ofstream, please? In case there is faster alternative, will it be worth of rewritting my application? In other words, could the time saved be +50%? Thank you.

Update:

I was asked to show you my code that generates the above files, so here you are - the most time consuming loop:

ofstream fout; fout.open(strngCollectiveSourceFileName,ios::out); fout << "#include \"StdAfx.h\"" << endl; fout << "#include \"Debug.h\"" << endl; fout << "#include \"glm.hpp\"" << endl; fout << "#include \"" << strngCollectiveHeaderFileName.substr( strngCollectiveHeaderFileName.rfind(TEXT("\\")) + 1) << "\"" << endl << endl; fout << "using namespace glm;" << endl << endl << endl; for (unsigned int nSprite = 0; nSprite < vpTilesetSprites.size(); nSprite++ ) { for(unsigned int nFrameSet = 0; nFrameSet < vpTilesetSprites[nSprite]->vpFrameSets.size(); nFrameSet++) { // display index definition fout << "// Index Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n"; string strngIndexSignature = strngIndexDefinitionSignature; strngIndexSignature.replace(strngIndexSignature.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("IndexData") ); strngIndexSignature.replace(strngIndexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName ); fout << strngIndexSignature << "[4] = {0, 1, 2, 3};\t\t" << "// " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << ": Index Definition\n\n"; // display vertex definition fout << "// Vertex Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n"; string strngVertexSignature = strngVertexDefinitionSignature; strngVertexSignature.replace(strngVertexSignature.find(TEXT("#aVertexArrayName#")), strlen(TEXT("#aVertexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("VertexData") ); strngVertexSignature.replace(strngVertexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName ); fout << strngVertexSignature << "[" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFramesCount() << "] =\n"; fout << "{\n"; for (int nFrameNo = 0; nFrameNo < vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFramesCount(); nFrameNo++) { fout << "\t" << "{{ vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[0].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 1: vec4(x, y, z, w), vec2(u, v) \n"; fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[1].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 2: vec4(x, y, z, w), vec2(u, v) \n"; fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[2].vTextureUV.fv << "f) }, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 3: vec4(x, y, z, w), vec2(u, v) \n"; fout << "\t" << " { vec4(" << fixed << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fx << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fy << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fz << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vPosition.fw << "f), vec2(" << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vTextureUV.fu << "f, " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->vpFrames[nFrameNo]->aVertices[3].vTextureUV.fv << "f) }}, // " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetShortDescription() << " vertex 4: vec4(x, y, z, w), vec2(u, v) \n\n"; } fout << "};\n\n\n\n"; } } fout.close();
Improve this question
10
  • 5
    In my experience, the problem is rarely with the specific IO interface used, but much more frequently, the problem is with building strings via concatenation: s1 += "foo"; s1 += "bar"; s1 += "baz"; print(s1);. Show us your code, and don't jump to the conclusion that the output is the problem. At least not without good profiling data to back it up... :) Commented Jun 8, 2011 at 2:38
  • @sarnold: +1, hi and thank you for your comment. Despite I use the above mentioned concatenations frequently, at this time, i do not think it is my case. I did as you recommended, and posted above the code of my most consuming loop generating the largest file. Commented Jun 8, 2011 at 2:52
  • On a side note; that is some of the hardest to read code I have seen lately. Line breaks and variables are your friend. Commented Jun 8, 2011 at 2:52
  • "at this time, i do not think it is my case." - how do you know? I don't see any profiling data in your question. Commented Jun 8, 2011 at 2:53
  • 1
    what are these lines doing then? TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("VertexData"). You also have a lot of calls to functions like strlen which may be problematic in a tight loop. Commented Jun 8, 2011 at 3:00

5 Answers 5

Reset to default
4

If you don't want to use C file I/O then you can give a try to; FastFormat. Look at the comparison for more info.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

+1, hi and thanks for your great tip. I went through the home page of FastFormat, and one of the key imporovements is increased speed by 100-900% over std::iostream. However, as others have pointed out, the reason to slow speed in my case can be elsewhere, not in the usage of std::ofstream.
3

How are vpTilesetSprites and vpTilesetSprites[nSprite] stored? Are they implemented with lists or arrays? There is a lot of indexed access to them, and if they are list-like structures, you'll spend a lot of extra time following needless pointers. Ed S.'s comment is right: giving the long indexed temporary variables and linebreaks could make it easier to read, and maybe faster, too:

fout << "// Index Definition: " << vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetLongDescription() << "\n"; string strngIndexSignature = strngIndexDefinitionSignature; strngIndexSignature.replace(strngIndexSignature.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")), TEXT("a") + vpTilesetSprites[nSprite]->GetObjectName() + vpTilesetSprites[nSprite]->vpFrameSets[nFrameSet]->GetFrameSetName() + TEXT("IndexData") ); strngIndexSignature.replace(strngIndexSignature.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName );

vs

string idxsig = strngIndexDefinitionSignature; sprite sp = vpTilesetSprites[nSprite]; frameset fs = sp->vpFrameSets[nFrameSet]; fout << "// Index Definition: " << fs->GetLongDescription() << "\n"; idxsig.replace(idxsig.find(TEXT("#aIndexArrayName#")), strlen(TEXT("#aIndexArrayName#")), TEXT("a") + sp->GetObjectName() + fs->getFrameSetName() + TEXT("IndexData")); idxsig.replace(idxsig.find(TEXT("#ClassName#")), strlen(TEXT("#ClassName#")), strngCollectiveArrayClassName);

But, the much bigger problem is how you're using strings as templates; you're looking for a given text string (and computing the length of your needle string every single time you need it!) over and over again.

Consider this: You're performing the find and replace operations nSprite * nFrameSet times. Each time through, this loop:

  • makes a copy of strngIndexDefinitionSignature
  • creates four temporary string objects when concatenating static and dynamic strings
  • compute strlen(TEXT("#ClassName#"))
  • compute strlen(TEXT("#aIndexArrayName#"))
  • find start point of both
  • replace both texts with new texts

And that's just the first four lines of your loop.

Can you replace your strngIndexDefinitionSignature with a format string? I assume it currently looks like this:

"flubber #aIndexArrayName# { blubber } #ClassName# blorp"

If you re-write it like this:

"flubber a %s%sIndexData { blubber } %s blorp"

Then your two find and replace lines can be replaced with:

sprintf(out, index_def_sig, sp->GetObjectName(), fs->getFrameSetName(), strngCollectiveArrayClassName);

This would remove two find() operations, two replace() operations, creating and destroying four temporary string objects, a string duplicate that was promptly over-written with two replace() calls, and two strlen() operations that return the same result every time (but aren't actually needed anyway).

You can then output your string with << as usual. Or, you can change sprintf(3) to fprintf(3), and avoid even the temporary C string.

Improve this answer

7 Comments

@sarnold: +1, hi again sarnold. Abolutely fabulous answer. Thank you for your willingness to analyze my code. This answer is beneficial for me to improve my proramming skills in general. To answer your first question: all the data are stored in std::vector<> containers including vpTileSprites[]. I like using Hungarian notation, and the prefix vp suggests, the object it is a vector of pointers. I will obey your recommendation, and use variables more, to increase code readability, although, to me, my code is perfectly readable, as it is. :-)
@sarnold: I understand what you say. Regarding your second question: a large number fo data I take from .ini files, including strngIndexDefinitionSignature. I get the information with the usage of DWORD WINAPI GetPrivateProfileString(). So all I have is a char array. I am not aware that I could apply your advice into the text within .ini files where it is formatted as follows: Index Definition = const unsigned short #ClassName#::#aIndexArrayName#. If you have any good and practical advice, that would really help.
Alghough all of the advices are very good here, from my viewpoint, this is the most detailed and valuable answer. Threfore, I wish to mark this answer as the Acceptable Answer to my question.
@sarnold: for clarification what I do: I organize a set of custom files to be processed into directories. Every directory contains .ini file with specification, how the content in the particluar directory is supposed to be processed. My application then takes both the .ini files information and the contained files. It processes them, plus, it otuputs a lot of .txt data files. My question was related to those .txt files.
@Bunkai, excellent feedback all around, I'm glad to have you on Stack Overflow. :) cplusplus claims constant time access by index, so that's alright. It might be expensive constant-time, but at least it is constant. If I knew C++ better, I'd have known that. :) Are you in a position to change the format of your ini files? Index Definition = const unsigned short %s::%s. This fails if the order changes.
|
3

Assuming you do it in large enough chunks, calling write() directly might be faster; that said, it's more likely that your biggest bottleneck doesn't have anything directly to do with std::ofstream. The most obvious thing is to make sure you aren't using std::endl (because flushing the stream frequently will kill performance). Beyond that, I would suggest profiling your app to see where it's actually spending the time.

Improve this answer

2 Comments

+1, hi servn, and thanks for your alternative solutions. This might be elementary question from me, but because, I have never done it, how is profiling performed? Is it by applying some king of specialized profiling tools, that will measure bottlenecks, or is it simple observation analysis, and finding out manually, what areas of code might be problematic?
A profiler is a specialized tool; on Mac, Shark or Instruments, on Linux, oprofile (or callgrind). You can also use a debugger as a sort of profiler; just repeatedly break into the debugger manually, and see what the stack traces look like.
1

The performance of ostream is probably not your actual issue; I suggest using a profiler to determine where your real bottlenecks are. If ostream turns out to be your actual problem, you can drop down to <cstdio> and use fprintf(FILE*, const char*, ...) for formatted output to a file handle.

Improve this answer

3 Comments

+1, Hi Jon. Your advice is basically in agreement with some other advices, saying, that my problem with speed might be elsewhere. If I understand correctly, I should use a software profiler, to analyze my application. Could you advice, a good profiler, please?
@Bunkai.Satori: The answer that @sarnold gave covers your current situation, but for your future reference, gprof (the GNU profiler) is a good profiler to use with GCC.
Hi again, and thanks very much. I will take a look at gprof. It is great that this site offers the opportunity to get advices from the best programmers in the world :-)
1

The best answer will depend on what sort of text you are generating, and how you are generating it. C++ streams can be slow, but that mostly is because they can also do a lot more for you, such as locale-dependent formatting, and so on.

You may find speed ups with streams by bypassing some of the formatting (eg. ostream::write), or by writing characters directly to a streambuf instead (streambuf::sputn). Sometimes increasing the buffer size on the relevant streambuf helps (via streambuf::pubsetbuf).

If this isn't good enough, you might want to try C-style stdio files, eg fopen, fprintf, etc. It takes a little while to get used to the way the text is formatted if you're not used to that method but the performance is usually pretty good.

For the absolute top performance you usually have to go to OS-specific routines. Sometimes the direct low-level file routines are significantly better than the C stdio, but sometimes not - for example, I've seen some people say WriteFile on Win32 is the fastest method on Windows, whereas some Google hits report it as being slower than stdio. Another approach might be a memory-mapped file, eg. mmap + msync - this essentially uses your system memory as the disk and writes the actual data to disk in large blocks, which is likely to be near optimal. However you run the risk of losing all the data if you incur a crash half way for some reason, which may or may not be a problem for you.

Improve this answer

2 Comments

+1, hi and thanks for your very good advices. You offered even more solutions, and you advised which should be the most optimal and fastes choice on Windows platform. Originally, I was looking exactly for information like this, until, somebody has suggested, that my problem can be elsewhere: in usage of time consuming operations in my loop, as string replacing, searching, strlen() etc..
Since stdio calls WriteFile, calling WriteFile directly is less overhead and thus faster when writing lots of data. However, stdio is "faster" with many small writes because of the buffering that defers OS calls.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.