Buffer Size in C++

Question

I am observing the following behavior with the C++ Std library method std::ostream::write().

For buffering the data I am making use of the following C++ API

std::ofstream::rdbuf()->pubsetbuf(char* s, streamsize n)

This works fine ( verified using the strace utility ) as long as the size of data (datasize) we are writing on the file stream using

std::ofstream::write (const char* s, datasize n)

Is less than 1023 bytes ( below this value the writes are accumulated till the buffer is not full), but when the size of data to write exceeds 1023, the buffer is not taken into account and the data is flushed to the file.

e.g. If I set the buffer size to 10KB and write around 512bytes a time, strace will show that multiple writes have been combined into a single write

writev(3, [{"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 9728}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 512}], 2) = 10240 ( 10 KB ) writev(3, [{"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 9728}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 512}], 2) = 10240 ...

but when I write 1024 bytes a time ( keeping the buffer fixed to 10 KB), now strace shows me that it is not using the buffer and each ofstream::write call is being translated to write system call.

writev(3, [{NULL, 0}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 1024}], 2) = 1024 ( 1KB ) writev(3, [{NULL, 0}, {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"..., 1024}], 2) = 1024 ...

Is there any C++ API Call or Linux Tuning Parameter which I am missing?

There's no such thing. There's the std::flush manipulator, but it forces a flush, which is the opposite of what the OP wants. — Sebastian Redl
– Sebastian Redl, Commented Mar 18, 2014 at 11:23
@SebastianRedl To be honest OP didn't even state what he wants. — Bartek Banachewicz
– Bartek Banachewicz, Commented Mar 18, 2014 at 11:26
Writing 1024 bytes at a time is efficient. In a modern architecture there is very little to gain by making larger writes. — Klas Lindbäck
– Klas Lindbäck, Commented Mar 18, 2014 at 11:32
@KlasLindbäck I disagree: With 10 GB/s memory bus bandwidth, copying 1 kiB takes roughly 100 ns. A syscall generally takes several microseconds. I'd say, it takes roughly 100 kiB for the syscall overhead to become neglegible. — cmaster - reinstate monica
– cmaster - reinstate monica, Commented Mar 18, 2014 at 11:38

Sebastian Redl · Accepted Answer · 2014-03-18 11:35:38Z

3

This is an implementation detail of libstdc++, implemented around line 650 of bits/fstream.tcc. Basically, if the write is larger than 2^10, it will skip the buffer.

If you want the rationale behind this decision, I suggest you send a mail to the libstdc++ development list.

http://gcc.gnu.org/ml/libstdc++/

answered Mar 18, 2014 at 11:35

Sebastian Redl

73.9k8 gold badges137 silver badges177 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Abhishek Over a year ago

Posted a mail to libstdc++ mailed, and yes its kind of hard coded behavior. To add further, I executed same test case on Sun OS ( Sun Studio CC Compiler ), the behavior is different there, it respects the buffer given ( no boundary on 1024 )

iMineLink Over a year ago

This seems to be still valid today (found the hardcoded constant here github.com/gcc-mirror/gcc/blob/master/libstdc++-v3/include/bits/…).

cmaster - reinstate monica · Accepted Answer · 2014-03-18 11:33:59Z

1

Looks like someone writing the stdlib implementation made an "optimization" without giving enough thought to it. So, the only workaround for you would be to avoid the C++ API and use the standard C library.

This is not the only suboptimality in the GNU/Linux implementation of the standard C++ library: on my machine, malloc() is 100 cycles faster than the standard void* operator new (size_t size)...

answered Mar 18, 2014 at 11:33

cmaster - reinstate monica

41.1k9 gold badges69 silver badges110 bronze badges

2 Comments

scai Over a year ago

new() calls the constructor whereas malloc() doesn't. And it does a few more nice things. Comparing the speed of both is just nonsense.

cmaster - reinstate monica Over a year ago

@scai Sorry to contradict, but that is precisely the reason why I gave the entire signature: I'm not talking about the new keyword, and I'm not even talking about class specific new operators, I'm talking about the one global function that does the allocation itself, and nothing else. This is the function, that you can easily implement by just calling through to malloc(), which is the basis for my measurement.

Collectives™ on Stack Overflow

Buffer Size in C++

2 Answers 2

2 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Related