Leonid has shown how to get a big improvement in speed with Java and has mentioned that by using a library function written e.g. in C you could probably get further improvement as you can then avoid additional copies/data transfer.
I just wanted to add an all Mathematica solution as it probably indicates how similar problems can be tackled where for whatever reason something like the Java approach isn't possible or necessary. I've used a similar approach to e.g. read only small parts of huge XML-files. This isn't faster as the memory intensive read-all-at-once but at least isn't slower and should be much more memory efficient (I'd expect it to use about twice the memory of the final result, as we need one copy). It basically reads the file in chunks of adjustable size, extracts the part of interest of each chunk and collects everything. Here is the code:
AbsoluteTiming[ str = OpenRead[filename, BinaryFormat -> True]; chunksize = 500; data = {}; res = {1}; While[Or[data == {}, And[res[[-1, -1]] =!= EndOfFile, Length[res] > 0]], data = {data, res = BinaryReadList[ str, {"Real32", "Real32", "Real32", "Real32", "Real32", "Real32", "Real32", "Real32", "Real32", "Integer32", "Integer32"}, chunksize, ByteOrdering -> +1][[All, {4, 11}]] }; ]; Close[str]; data = Partition[Flatten[data], 2]; ]