Vfrdtyky

Following my previous question in this post:

I'm trying to build a 128 row matrix with N column (up to many millions). The sole purpose of this matrix is to calculate the mean or median of all 128 rows column wise and save it as a vector (same number of columns as my data).

The size of my data files varies a lot and with smaller ones I've been able to perform just this without any issues using the code from the previous post (see above). But of course, if I'm dealing with bigger datasets I run out of memory. Keep in mind that the error is in concatenating the 128 rows into a new matrix.

EDIT the code used for concatenating the data in the files is the following:

for k = TTs;   %TTs to plot

    cd (strcat('TT',num2str(k)));   %TT folder



        for w = 1:4;

            load(strcat('TT',num2str(k),'ch',num2str(w),'.mat'));

            allChs1(4*(k-1)+w,:) = data(1,:);   %concatenate into one matrix

        end



    cd ..

end

I've considered averaging 128 rows (on a column by column basis) and consecutively save that value, but have been utterly unsuccessful in doing so...

Any idea on how I could implement this? And, might there be a better way of getting the average of 128 rows on column by column basis?

Cheers,
Oiko

You can incrementally compute an average, so that you only have one dataset and the average value in memory:

mean[n] = value[n]/n + mean[n-1](n-1)/n*

avg_vector=0; % It will be changed to a vector at first iteration

for k = TTs;   %TTs to plot

    folder=['TT',num2str(k)];   %TT folder



        for w = 1:4;

            file = ['TT' num2str(k) 'ch' num2str(w) '.mat'];

            count = 4*(k-1)+w;

            load(fullfile(folder,file));

            avg_vector = (1/count) * data(1,:) + ((count-1)/count) * avg_vector;

        end

end

Provided that the number of columns is not very large (in which case some precision may be lost to round-off errors) this will give the average. The only large vectors in memory are avg_vector and data

For a median, this more complicated as there is no incremental formula. You may have to add another loop over some subset of 1:N and do a selection.

filename=@(k,w) fullfile(['TT',num2str(k)],['TT' num2str(k) 'ch' num2str(w) '.mat']);

load(filename(1,1));

N=size(data,2);

median_all = zeros(1,N);



stride = 1e6;



for nn=1:stride:N

    rng = nn:min(N,nn+stride-1);

    MAT=zeros(128,length(rng));

    for k=TTs

        for w=1:4

            load(filename(k,w));

            MAT(4*(k-1)+w,:)=data(1,rng);

        end

    end

    median_all(1,rng) = median(MAT,1);

    clear MAT

end

There will be 128 million values at most in matrix MAT, so about 1GB if data is a 64-bit type (e.g. double). The downside is, file will have to be read several times. The balance is shifted from memory consumption to file I/O.