4

Background

I'm running Musicbrainz Picard to update a ~11M ogg file on a 500GB NTFS disk (Transcend StoreJet) connected via USB and mounted using autofs. The connection is through docking station. I can't be sure I always unmounted it properly...

I'm concerned about is that the operation is taking extremely long; I expected the whole folder be processed under minute but it's taking maybe few hours already. When I fire iotop(1), it reports ~ 25K/s disk write, with ~99% for the picard process. (Picard is not totally hung, GUI does refresh/respond once in few minutes).

In hope to see some progress, I keep checking lsof; the whole output looks like:

$ lsof /mnt/greeno-ntfs COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME picard 2885 amahdal mem REG 8,17 11121609 44661 /mnt/greeno-ntfs/path/to.ogg picard 2885 amahdal 14u REG 8,17 11121609 44661 /mnt/greeno-ntfs/path/to.ogg picard 2885 amahdal 16u REG 8,17 11121609 44661 /mnt/greeno-ntfs/path/to.ogg

but I can't really make sense out of all observations -- I can only make some assumptions. So I figured I'd ask here.

Questions

  • Is it normal that there are 3 FDs for the file? One 'mem' and two "regular" ones?

    I tried to create trivial script that opens and updates a file (sleeping in between to take long), and no, there was just one regular FD (5u), so apparently normal open won't behave like this.

    Assumption: It's a result of (maybe generic) technique to deal with potentially long file I/O that Picard (or its lib) deliberately explains. If so, can somebody shed some light on it? (Eg. why 2+1?)

  • As I have noticed, the SIZE/OFFSET column is actually shrinking over time.

    Assumption: This corresponds to Picard actually seeking inside file and updating, right?

Random assumptions - at this point, does any of them make sense as possible cause?

  • Picard is buggy (extremely inefficient in updating/shrinking),

  • disk is failing (it is 5+ years old...),

  • the filesystem is is badly mounted (who knows how to mount ntfs optimally),

  • the filesystem is is broken from un/docking (can't check as I don't have chkdsk)...

So what next? What can I see next to learn about what's happening?

Improve this question
1
  • mem is lsof output means the file is used as memory-mapped by the process. You mmap file into memory and work with it as with data in memory, not as file. See man mmap. Commented Jun 15, 2016 at 8:19

1 Answer 1

Reset to default
0

Your guesses are easy to test other than picard being buggy. If the file is really only 11 MB large, just copy it to /dev/shm (RAM) or a native filesystem to test it.

And I would certainly expect NTFS to be much slower and use far more CPU, but would not change minutes to hours normally. Avoiding ntfs should be the first thing to test.

And lsof output can be very weird... sometimes even "lsof -p $pid" vs "lsof /path" have different output even if the 2nd lists only that one pid. So I wouldn't try to figure out if that's "normal" or not.

If you want to know about seeking and writing of the file, you should use strace, not lsof.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.