Question: Problems with getting consistent execution timings using CodeTools::Usage

I have been using CodeTools::Usage with Maple 17 to compare timings of a couple of matrix inversion routines against each other and against the Maple library routines LinearAlgebra::MatrixInverse and LinearAlgebra::Determinant.

The small section of code that I use to collect the cpu times for each method is:

----------------------------------------

udata := Usage(invTest(x,y,z,delta,mu), output=['cputime', 'bytesused'],
              iterations=nRepeats, quiet);
dat[1]:=udata[1]; dat[2]:=udata[2];

----------------------------------------
nRepeats has been set to 25.

In order to get the timings as accurate as possible, I only open a couple of terminal windows on my laptop so as to run the tests with as low a loaded machine as I can get. (I'm running Ubuntu 14.04 on an 8 core Intel (R) Core i7-3840QM, 2.8GHz CPU with 16Gb of memory.)

A test run consists of a sequence of test matrices each of which is run for an increasing sequence of matrix orders. I either time an implementation on its own or I time both an implementation and the Maple library routines. In the later case for each test matrix and order of matrix, I use Usage to first time the Maple routines and then the times for one of the implementations:

----------------------------------------

if compMaple then
udata := Usage(MatrixInverse(A), output=['cputime', 'bytesused'],
              iterations=nRepeats, quiet);
mdat[1]:=udata[1]; mdat[2]:=udata[2];

udata := Usage(Determinant(A), output=[cputime, bytesused],
              iterations=nRepeats, quiet);
mdat[1]:=mdat[1]+udata[1];
mdat[2]:=mdat[2]+udata[2];
else
  mdat[1..2] := 0:
end if:

udata := Usage(invTest(x,y,z,delta,mu), output=['cputime', 'bytesused'],
              iterations=nRepeats, quiet);
dat[1]:=udata[1]; dat[2]:=udata[2];

----------------------------------------

I have noticed the following:

a) If I time both Maple and an implementation, and the same implementation on its own I can get timings that vary up to a factor of 2 or more,

b) Changes in timings for the same test run and as close to the same environment as I can get (i.e., just the two user terminal windows open) can generate timing differences of up to 50%.

c) Sometimes `chaotic' timings are generated, for example,

    n = 150, t = 3 (secs)
    n = 200, t = 24 (secs)
    n = 250, t = 18 (secs)

which cannot be reflecting the times required to perform the calculations. This type of behaviour always seems to occur when the reported memory bytecount is increasing from ~0.25* 10^9 to over 10^9.

Is there anything I can be doing here to get more consistent timings?

I want to use the timings from these tests in a journal article so it would be good to have the same test run under very similar conditions to return timings within a few percent difference of one another. I certainly get this sort of tolerances (<10%) if I use the Fortran cpu_time intrinsic to time Fortran code.

Any advice would be gratefully received.

Please Wait...