Importing large amount of data into MATLAB?

Question

I have a text file that is ~80MB. It has 2 cols and around 6e6 rows. I would like to import the data into MATLAB, but it is too much data to do with the load function. I have been playing around with the fopen function but cant get anything to work.

Ideally I would like to take the first col of data and import and eventually have it in one large array in MATLAB. If that isn't possible, I would like to split it into arrays of 34,013 in length. I would also like to do the same for the 2nd col of data.

Did you try [a,b] = textread(filename, '%f %f')? (assuming your data is numeric). — dpwe
– dpwe, Commented Jul 24, 2014 at 21:16

lakshmen · Accepted Answer · 2014-07-25 22:03:04Z

2

fileID = fopen('yourfilename.txt'); formatSpec = '%f %f'; while ~feof(fileID) C = textscan(fileID,formatSpec,34013); end

Hope this helps..

Edit:

The reason you are getting error is because C has two columns. So you need to take the columns individually and handle them.

For example:

column1data = reshape(C(:,1),301,113); column2data = reshape(C(:,2),301,113);

edited Jul 25, 2014 at 22:03

answered Jul 24, 2014 at 23:16

lakshmen

29.2k68 gold badges183 silver badges286 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Prefoninsane Over a year ago

This works. How do I take the bundles of 34013 and create a matrix of 301x113 each time with the data, with both col 1 and 2? The reshape function keeps giving me error :Product of known dimensions, 34013, not divisible into total number of elements, 2.

Prefoninsane Over a year ago

hmmmm gives me error: To RESHAPE the number of elements must not change.

anandr · Accepted Answer · 2014-07-25 04:37:41Z

You may also consider to convert your file to binary format if your data file is not changing each time you want to load it. Then you'll load it way much faster. Or you may do "transparent binary conversion" like in the function below. Only first time you load the data will be slow. All subsequent will be fast.

function Data = ReadTextFile(FileName,NColumns) MatFileName = sprintf('%s.mat',FileName); % binary file name if exist(MatFileName,'file')==2 % if it exists S = load(MatFileName,'Data'); % load it instead of Data = S.Data; % the original text file return; end fh = fopen(FileName); % if binary file does not exist load data ftom the original text file fh_closer = onCleanup( @() fclose(fh) ); % the file will be closed properly even in case of error Data = fscanf(fh, repmat('%f ',1,NColumns), [NColumns,inf]); Data = Data'; save(MatFileName,'Data'); % and make binary "chache" of the original data for faster subsequent reading end

Do not forget to remove the MAT file when the original data file is changed.

Collectives™ on Stack Overflow

Importing large amount of data into MATLAB?

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related