Why am I writing data to a file in java so slow

Question

I am using java.nio.channels.FileChannel to write data to a file. And here is my code :

RandomAccessFile raf = new RandomAccessFile(file, "rws"); FileChannel fileChannel = raf.getChannel(); ByteBuffer buffer = ByteBuffer.allocate(4096); long offset = 0; while (offset < 1024 * 1024 * 1024) { buffer.clear(); fileChannel.write(buffer, offset); buffer.clear(); offset += buffer.remaining(); logger.warn("fake write {} {}", offset, buffer.remaining()); }

and my out put is like :

WARN [11:01:58,498][writer]BackUp(151):fake write 3047424 4096 WARN [11:01:58,507][writer]BackUp(151):fake write 3051520 4096 WARN [11:01:58,515][writer]BackUp(151):fake write 3055616 4096 WARN [11:01:58,523][writer]BackUp(151):fake write 3059712 4096 WARN [11:01:58,532][writer]BackUp(151):fake write 3063808 4096 WARN [11:01:58,540][writer]BackUp(151):fake write 3067904 4096 WARN [11:01:58,549][writer]BackUp(151):fake write 3072000 4096 WARN [11:01:58,557][writer]BackUp(151):fake write 3076096 4096 WARN [11:01:58,565][writer]BackUp(151):fake write 3080192 4096 WARN [11:01:58,574][writer]BackUp(151):fake write 3084288 4096 WARN [11:01:58,582][writer]BackUp(151):fake write 3088384 4096 WARN [11:01:58,590][writer]BackUp(151):fake write 3092480 4096 WARN [11:01:58,599][writer]BackUp(151):fake write 3096576 4096 WARN [11:01:58,607][writer]BackUp(151):fake write 3100672 4096 WARN [11:01:58,615][writer]BackUp(151):fake write 3104768 4096 WARN [11:01:58,624][writer]BackUp(151):fake write 3108864 4096 WARN [11:01:58,632][writer]BackUp(151):fake write 3112960 4096

you can see that I am writing in sequence but every 4K cost 8ms, which is 500 KB/s. It's very slow and I am really confused about that.

If this is a real (spinning) disk, with mode s you are forcing almost every write to wait a disk revolution, and consumer disks are rarely faster than 7200rpm = 8.333ms per revolution. — dave_thompson_085
– dave_thompson_085, Commented Feb 21, 2017 at 5:25

user207421 · Accepted Answer · 2017-02-21 04:01:29Z

I would use write(buffer) and forget about the offset parameter, as it may cause a seek, which would double the number of system calls. Last time I measured it (quite some time ago) seeking was surprisingly inefficient, even seeking to where you already are, as in this case.
If you even doubled the size of the buffer, you would probably nearly halve the total time. 4096 is about the minimum buffer size I would use for anything. BufferedOutputStream uses 8192, for example, and for a backup program I would use something much larger.
But if you're really just trying to write a megabyte of zeros, just extend the RandomAccessFile, with setLength(). No writes necessary.

This is just a test demo, my real program is something like a file system who will deal with a lot of sequence or no-sequence IO.
And I have tried with write(buffer), the result is all the same. But if I make the buffer size 1M, the speed is much faster.

Collectives™ on Stack Overflow

Why am I writing data to a file in java so slow

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related