I have to read in a huge text file (>200,000 words) and process each word. I read in the entire file into a string and then attach a string stream to it to process easily each word. The approach is I directly input each word from file using << and process it but comparing both the approaches does not give me any advantage in terms of execution time. Isn't it faster to operate on a string in memory than from a file which needs a system call every time I need a word? Please suggest some performance enhancing methods.
4 Answers
For performance and minimal copying, this is hard to beat (as long as you have enough memory!):
void mapped(const char* fname) { using namespace boost::interprocess; //Create a file mapping file_mapping m_file(fname, read_only); //Map the whole file with read permissions mapped_region region(m_file, read_only); //Get the address of the mapped region void * addr = region.get_address(); std::size_t size = region.get_size(); // Now you have the underlying data... char *data = static_cast<char*>(addr); std::stringstream localStream; localStream.rdbuf()->pubsetbuf(data, size); // now you can do your stuff with the stream // alternatively } 
Comments
If you're going to put the data into a stringstream anyway, it's probably a bit faster and easier to copy directly from the input stream to the string stream:
std::ifstream infile("yourfile.txt"); std::stringstream buffer; buffer << infile.rdbuf(); The ifstream will use a buffer, however, so while that's probably faster than reading into a string, then creating a stringstream, it may not be any faster than working directly from the input stream.
6 Comments
Steve Townsend
If there is other I/O going on in parallel with this processing, then one-shot load of data will be preferable as disk head does not have the same opportunity to move between buffer refreshes.
Steve Townsend
I think this combined with Martin's answer to previous q will perform the best. I gave you +1 for best local info
Steve Townsend
@ruslik - where is there mention of async I/O? How is memory-mapped file different from loading entire file contents using STL, apart from complexity and non-portability of the former?
Jerry Coffin
@Steve: memory mapping can be considerably different if you don't have (close to) enough physical memory to hold the entire file. In such a case, loading the entire file basically ends up copying the data into memory, then copying it back out to the paging file, then reading it back in from the paging file when used. Memory mapping avoids all but the last of those. On Windows it typically helps only a little, but on Linux it can be extremely effective.
Steve Townsend
@Jerry - right, but 200K words is the problem as stated. 1-2MB tops?
|
There is caching involved, so it does not necessarily do a system call each time you extract. Having said that, you may get marginally better performance at parse time by parsing a single contiguous buffer. On the other hand, you are serializing the workload (read entire file, then parse), which can potentially be parallelized (read and parse in parallel).
answered Nov 4, 2010 at 17:50
Michael Goldshteyn
74.9k25 gold badges136 silver badges183 bronze badges
c? Anyway ... try opening the file in "binary" mode (fopen("hugefile", "rb")in C) to make the runtime library NOT do end-of-line, and possibly other, translations.