MaplePrimes - comments on Post, Fast writing to and reading from a disc

<rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" version="2.0">
  <channel>
    <title>MaplePrimes - comments on Post, Fast writing to and reading from a disc</title>
    <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc</link>
    <language>en-us</language>
    <copyright>2026 Maplesoft, A Division of Waterloo Maple Inc.</copyright>
    <generator>Maplesoft Document System</generator>
    <lastBuildDate>Thu, 11 Jun 2026 17:27:40 GMT</lastBuildDate>
    <pubDate>Thu, 11 Jun 2026 17:27:40 GMT</pubDate>
    <itunes:subtitle />
    <itunes:summary />
    <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
    <image>
      <url>http://www.mapleprimes.com/images/mapleprimeswhite.jpg</url>
      <title>MaplePrimes - comments on Post, Fast writing to and reading from a disc</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc</link>
    </image>
    <item>
      <title>buffering?</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98178</link>
      <itunes:summary>&lt;p&gt;I wonder whether this is due to some inefficiency related to (lack of decent) buffering. I guess that I mean: it doesn't sound so good to read from disk a byte at a time (even if not importing integer[1] and even if endianness is a concern). I wonder what the performance would be from just using something like Linux fread (buffering according to cache size, say)?&lt;/p&gt;
&lt;!--break--&gt;
&lt;p&gt;acer&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98178</guid>
      <pubDate>Wed, 27 Oct 2010 01:56:57 Z</pubDate>
      <itunes:author>acer</itunes:author>
      <author>acer</author>
    </item>
    <item>
      <title>Buffering</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98181</link>
      <itunes:summary>&lt;p&gt;I also think so (buffering). In Windows both fread and read are in the msvcrt dll(s) which have cdecl calling convention, so one can't call them that simple from Maple. It's not that hard to write a wrapper and compile it in a dll with correct calling convention - I think I have an example with mktime somewhere on this site, but it still takes some time. Buffering in zlib is pretty good. The standard distribution, zlib1.dll, also uses cdecl convention, that's one of the reasons I used that one. Plus, this one has some optimized assembly code.&lt;/p&gt;
&lt;p&gt;It would be interesting to compare speeds with fread. It uses just a FILE structure (in addition to simple hardware types) so it could be called in Linux using WRAPPER option in define_external, I think.&lt;/p&gt;
&lt;p&gt;By the way, Maple 14 includes a version of zlib (called mzlib.dll in Windows), but it has only 2 functions exported there - used in StringTools:-Compress and Uncompress.&lt;/p&gt;
&lt;!--break--&gt;
&lt;p&gt;_______________&lt;br&gt; Alec Mihailovs, PhD&lt;br&gt; Maplesoft Member&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98181</guid>
      <pubDate>Wed, 27 Oct 2010 03:09:29 Z</pubDate>
      <itunes:author>Alec Mihailovs</itunes:author>
      <author>Alec Mihailovs</author>
    </item>
    <item>
      <title>In 64bit Linux</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98195</link>
      <itunes:summary>&lt;p&gt;Just tried it in the 64bit Linux with Maple 12 and not the newest zlib version 1.2.3.3. and gzread is still much faster,&lt;/p&gt;
&lt;pre&gt;    |\^/|     Maple 12 (X86 64 LINUX)
._|\|   |/|_. Copyright (c) Maplesoft, a division of Waterloo Maple Inc. 2008
 \  MAPLE  /  All rights reserved. Maple is a trademark of
 &amp;lt;____ ____&amp;gt;  Waterloo Maple Inc.
      |       Type ? for help.
&amp;gt; A:=Array(1..2^26,99,datatype=integer[1]):
memory used=64.9MB, alloc=64.8MB, time=0.19
&amp;gt; time(writebytes("A",A));close("A");
                                     3.170

&amp;gt; B:=Array(1..2^26,1,datatype=integer[1]):
memory used=128.9MB, alloc=128.9MB, time=3.50
&amp;gt; time(readbytes("A",B));close("A");
                                     2.000

&amp;gt; B[1],B[-1];
                                    99, 99

&amp;gt; myreadbytes:=proc(f)
&amp;gt; local gzopen, gzread, gzclose, n, p, A;
&amp;gt; gzopen:=define_external('gzopen',
&amp;gt;     'path'::string,
&amp;gt;     'mode'::string,
&amp;gt;     'RETURN'::integer[8],
&amp;gt;     'LIB'="libz.so.1");
&amp;gt; gzread:=define_external('gzread',
&amp;gt;     'file'::integer[8],
&amp;gt;     'buf'::REF(ARRAY(datatype=integer[1])),    
&amp;gt;     'len'::integer[8],
&amp;gt;     'RETURN'::integer[4],
&amp;gt;     'LIB'="libz.so.1");
&amp;gt; gzclose:=define_external('gzclose',
&amp;gt;     'file'::integer[8],
&amp;gt;     'RETURN'::integer[4],
&amp;gt;     'LIB'="libz.so.1");
&amp;gt; n:=FileTools:-Size(f);
&amp;gt; A:=Array(1..n,datatype=integer[1]);
&amp;gt; try p:=gzopen(f,"rb");
&amp;gt; if gzread(p,A,n)=n
&amp;gt; then return A end if
&amp;gt; finally gzclose(p)
&amp;gt; end try
&amp;gt; end proc:
&amp;gt; time(assign(C=myreadbytes("A")));
memory used=193.0MB, alloc=193.0MB, time=5.57
                                     0.070

&amp;gt; C[1],C[-1];
                                    99, 99

&amp;gt; 'time(myreadbytes("A"))'$5;
memory used=257.7MB, alloc=257.4MB, time=5.63
memory used=321.7MB, alloc=321.5MB, time=5.67
memory used=385.7MB, alloc=385.6MB, time=5.70
memory used=449.7MB, alloc=449.7MB, time=5.73
memory used=513.8MB, alloc=513.9MB, time=5.78
                       0.059, 0.040, 0.030, 0.030, 0.049

&amp;gt; E:=Array(1..2^26,2,datatype=integer[1]):
memory used=577.8MB, alloc=578.0MB, time=5.91
&amp;gt; time(ArrayTools:-Copy(A,E));
                                     0.089

&lt;/pre&gt;
&lt;!--break--&gt;
&lt;p&gt;_______________&lt;br&gt; Alec Mihailovs, PhD&lt;br&gt; Maplesoft Member&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98195</guid>
      <pubDate>Wed, 27 Oct 2010 12:48:51 Z</pubDate>
      <itunes:author>Alec Mihailovs</itunes:author>
      <author>Alec Mihailovs</author>
    </item>
    <item>
      <title>fread</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98213</link>
      <itunes:summary>&lt;p&gt;As Acer suggested, I tried fread both in 64bit Linux and in 32bit Windows - the speed is comparable with myreadbytes, and much faster than Maple operates without it. &lt;br&gt;&lt;br&gt;First, in Linux,&lt;/p&gt;
&lt;pre&gt;fopen1:=define_external('fopen',
    'filename'::string,
    'mode'::string,
    'RETURN'::integer[8],
    'LIB'="libc.so.6"):
fread1:=define_external('fread',
    'ptr'::REF(ARRAY(datatype=integer[1])),
    'size'::integer[8],
    'count'::integer[8],
    'stream'::integer[8],
    'RETURN'::integer[8],
    'LIB'="libc.so.6"):
fwrite1:=define_external('fwrite',
    'ptr'::REF(ARRAY(datatype=integer[1])),
    'size'::integer[8],
    'count'::integer[8],
    'stream'::integer[8],
    'RETURN'::integer[8],
    'LIB'="libc.so.6"):
fclose1:=define_external('fclose',
    'stream'::integer[8],
    'RETURN'::integer[4],
    'LIB'="libc.so.6"):
A:=Array(1..2^26,97,datatype=integer[1]):
p:=fopen1("aaa","wb"):
time[real](fwrite1(A,1,2^26,p));
                                    0.324
fclose1(p):
p:=fopen1("aaa","rb"):
B:=Array(1..2^26,98,datatype=integer[1]):
time[real](fread1(B,1,2^26,p));
                                    0.135
fclose1(p):
B[1],B[-1];
                                   97, 97
E:=Array(1..2^26,datatype=integer[1]):
time[real](ArrayTools:-Copy(A,E));
                                    0.203
&lt;/pre&gt;
&lt;p&gt;In Windows, first I compiled a dll, I called it fread.dll, using the following code,&lt;/p&gt;
&lt;pre&gt;#include &amp;lt;stdio.h&amp;gt;

__declspec(dllexport) FILE * __stdcall fo(
    const char * filename, const char * mode ){
	
	return fopen(filename, mode); 
}

__declspec(dllexport) size_t __stdcall fr(
    void * ptr, size_t size, size_t count, FILE * stream ){
	
	return fread(ptr, size, count, stream); 
}

__declspec(dllexport) size_t __stdcall fw(
    const void * ptr, size_t size, size_t count, FILE * stream ){
	
	return fwrite(ptr, size, count, stream); 
}

__declspec(dllexport) int __stdcall fc(
    FILE * stream ){
	
	return fclose(stream); 
}
&lt;/pre&gt;
&lt;p&gt;and then used it in Maple as follows&lt;/p&gt;
&lt;pre&gt;fopen1:=define_external('fo',
    'filename'::string,
    'mode'::string,
    'RETURN'::integer[4],
    'LIB'="fread.dll"):
fread1:=define_external('fr',
    'ptr'::REF(ARRAY(datatype=integer[1])),
    'size'::integer[4],
    'count'::integer[4],
    'stream'::integer[4],
    'RETURN'::integer[4],
    'LIB'="fread.dll"):
fwrite1:=define_external('fw',
    'ptr'::REF(ARRAY(datatype=integer[1])),
    'size'::integer[4],
    'count'::integer[4],
    'stream'::integer[4],
    'RETURN'::integer[4],
    'LIB'="fread.dll"):
fclose1:=define_external('fc',
    'stream'::integer[4],
    'RETURN'::integer[4],
    'LIB'="fread.dll"):
A:=Array(1..2^26,97,datatype=integer[1]):
p:=fopen1("aaa","wb"):
time[real](fwrite1(A,1,2^26,p));
                                    0.881
fclose1(p):
p:=fopen1("aaa","rb"):
B:=Array(1..2^26,98,datatype=integer[1]):
time[real](fread1(B,1,2^26,p));
                                    0.072
fclose1(p):
B[1],B[-1];
                                   97, 97
E:=Array(1..2^26,datatype=integer[1]):
time[real](ArrayTools:-Copy(A,E));
                                    0.092
&lt;/pre&gt;
&lt;!--break--&gt;
&lt;p&gt;_______________&lt;br&gt; Alec Mihailovs, PhD&lt;br&gt; Maplesoft Member&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98213</guid>
      <pubDate>Wed, 27 Oct 2010 22:20:36 Z</pubDate>
      <itunes:author>Alec Mihailovs</itunes:author>
      <author>Alec Mihailovs</author>
    </item>
    <item>
      <title>excellent</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98243</link>
      <itunes:summary>&lt;p&gt;&lt;a href="http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc#comment98213"&gt;@Alec Mihailovs&lt;/a&gt; This is very interesting. I did a quick check on 32bit Linux, with comparable results. I did all tests on local hardisk, under /tmp.&lt;/p&gt;
&lt;p&gt;I also tested fread'ind into a 2^26 length Array in the case that the file only had 5 bytes previously written to it in total. The `fread1` command did the right thing, and did not introduce garbage into the entries past the 5th position.&lt;/p&gt;
&lt;p&gt;At length 2^26 the timings differed between fread1 and readbytes was about a factor of one hundred. Very impressive. It's not even clear that the result isn't even better than that, since the fread1 timing varies a bit and is so small that it maybe mostly noise.&lt;/p&gt;
&lt;p&gt;It was mostly in the range of sizes 2^22 through 2^24 that readbytes started to slow down. It performed fine at, say 2^18.&lt;/p&gt;
&lt;p&gt;One thing that I noticed (Linux 32bit) was that at size 2^26 the readbytes command took this amount of time in various Maple releases:&lt;/p&gt;
&lt;p&gt;Maple 14:&amp;nbsp; 5.240 sec&lt;/p&gt;
&lt;p&gt;Maple 13.01: 5.644 sec&lt;/p&gt;
&lt;p&gt;Maple 12.02: 5.112 sec&lt;/p&gt;
&lt;p&gt;Maple 11.02: 1.832 sec&lt;/p&gt;
&lt;p&gt;Maple 10.06: 1.552 sec&lt;/p&gt;
&lt;p&gt;But 0.05 sec is still much better than 1.552 sec.&lt;/p&gt;
&lt;p&gt;I used this code below, in a file. The file "aaa" must first be written out to currentdir(), and contain the data, naturally. Eg, writebytes("aaa",[3,4,5,6,7]) for the "short test".&lt;/p&gt;
&lt;p&gt;### cut here ### snip 8&amp;lt; ### check.mpl ###&lt;/p&gt;
&lt;pre&gt;restart:&lt;br&gt;kernelopts(printbytes=false):&lt;br&gt;&lt;br&gt;str,st,ba,bu:=&lt;br&gt;time[real](),time(),kernelopts(bytesalloc),kernelopts(bytesused):&lt;br&gt;   B:=Array(1..2^26,98,datatype=integer[1]):&lt;br&gt;time[real]()-str,time()-st,&lt;br&gt;kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu;&lt;br&gt;&lt;br&gt;str,st,ba,bu:=&lt;br&gt;time[real](),time(),kernelopts(bytesalloc),kernelopts(bytesused):&lt;br&gt;   readbytes("aaa",B):&lt;br&gt;time[real]()-str,time()-st,&lt;br&gt;kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu;&lt;br&gt;&lt;br&gt;Vector[row](B[1..10]),Vector[row](B[-10..-1]);&lt;br&gt;&lt;br&gt;restart:&lt;br&gt;kernelopts(printbytes=false):&lt;br&gt;&lt;br&gt;fopen1:=define_external('fopen',&lt;br&gt;    'filename'::string,&lt;br&gt;    'mode'::string,&lt;br&gt;    'RETURN'::integer[4],&lt;br&gt;    'LIB'="libc.so.6"):&lt;br&gt;fread1:=define_external('fread',&lt;br&gt;    'ptr'::REF(ARRAY(datatype=integer[1])),&lt;br&gt;    'size'::integer[4],&lt;br&gt;    'count'::integer[4],&lt;br&gt;    'stream'::integer[4],&lt;br&gt;    'RETURN'::integer[4],&lt;br&gt;    'LIB'="libc.so.6"):&lt;br&gt;fclose1:=define_external('fclose',&lt;br&gt;    'stream'::integer[4],&lt;br&gt;    'RETURN'::integer[4],&lt;br&gt;    'LIB'="libc.so.6"):&lt;br&gt;&lt;br&gt;B:=Array(1..2^26,98,datatype=integer[1]):&lt;br&gt;&lt;br&gt;str,st,ba,bu:=&lt;br&gt;time[real](),time(),kernelopts(bytesalloc),kernelopts(bytesused):&lt;br&gt;   p:=fopen1("aaa","rb"):&lt;br&gt;   fread1(B,1,2^26,p):&lt;br&gt;   fclose1(p):&lt;br&gt;time[real]()-str,time()-st,&lt;br&gt;kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu;&lt;br&gt;&lt;br&gt;Vector[row](B[1..10]),Vector[row](B[-10..-1]);&lt;br&gt;&lt;/pre&gt;
&lt;p&gt;### cut here ### snip 8&amp;lt; ### end of file ###&lt;/p&gt;
&lt;p&gt;ps. It can be handy to use integer[kernelopts('wordsize')/8] although Windows 64 sometimes makes that problematic.&lt;/p&gt;
&lt;p&gt;It's probably worth mentioning that calling readbytes("aaa",B,N) for N the size didn't seem to speed it up.&lt;/p&gt;
&lt;p&gt;So, now I wonder how fast endianness can be accomodated (in a C wrapper, obviously) for wider formats. And I wonder about ImageTools:-Read and ImportMatrix:-ReadBinaryFile. It's hard to tell, since as I am sure you know iolib(4,..) is a kernel built-in.&lt;/p&gt;
&lt;p&gt;acer&lt;/p&gt;
&lt;p&gt;reality := realities[reality];&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98243</guid>
      <pubDate>Thu, 28 Oct 2010 22:31:16 Z</pubDate>
      <itunes:author>acer</itunes:author>
      <author>acer</author>
    </item>
    <item>
      <title>Reading float[8] arrays</title>
      <link>http://www.mapleprimes.com/posts/98083-Fast-Writing-To-And-Reading-From-A-Disc?ref=Feed:MaplePrimes:Fast writing to and reading from a disc:Comments#comment98285</link>
      <itunes:summary>&lt;p&gt;float[8] arrays could be written and read as simple as binary arrays, just changing integer[1] in fread1 and fwrite1 to float[8] in fread8 and fwrite8, example for 32bit Windows,&lt;/p&gt;
&lt;pre&gt;fread8:=define_external('fr',
    'ptr'::REF(ARRAY(datatype=float[8])),
    'size'::integer[4],
    'count'::integer[4],
    'stream'::integer[4],
    'RETURN'::integer[4],
    'LIB'="fread.dll"):
fwrite8:=define_external('fw',
    'ptr'::REF(ARRAY(datatype=float[8])),
    'size'::integer[4],
    'count'::integer[4],
    'stream'::integer[4],
    'RETURN'::integer[4],
    'LIB'="fread.dll"):

A:=Array(1..3,[1.25,234.7,0.0018],datatype=float[8]);

    A := [1.25000000000000, 234.700000000000, 0.00180000000000000]

p:=fopen1("fla","wb"):
fwrite8(A,8,3,p):
fclose1(p):
B:=Array(1..4,datatype=float[8]):
p:=fopen1("fla","rb"):
fread8(B,8,3,p):
fclose1(p):
B;

    [1.25000000000000, 234.700000000000, 0.00180000000000000, 0.]

A:=Matrix(2,[1.25,234.7,0.0018, Pi],datatype=float[8]);

                 [ 1.25000000000000      234.700000000000]
            A := [                                       ]
                 [0.00180000000000000    3.14159265358979]

p:=fopen1("mat","wb"):
fwrite8(A,8,4,p):
fclose1(p):
B:=Matrix(2,datatype=float[8]):
p:=fopen1("mat","rb"):
fread8(B,8,4,p):
fclose1(p):
B;

              [ 1.25000000000000      234.700000000000]
              [                                       ]
              [0.00180000000000000    3.14159265358979]
&lt;/pre&gt;
&lt;p&gt;The only problem with that is that the user should take care to avoid the overflowing of B - if more data is written in B than it's size, the Maple kernel vector might get damaged and crash Maple kernel, in the way that the worksheet can't be saved. It is safer to use it not directly, but inside a procedure similar to myreadbytes in the original post, taking care of preventing possible overflowing.&lt;/p&gt;
&lt;!--break--&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;_______________&lt;br&gt; Alec Mihailovs, PhD&lt;br&gt; Maplesoft Member&lt;/p&gt;</itunes:summary>
      <description>The latest comments added to the Post, Fast writing to and reading from a disc</description>
      <guid>98285</guid>
      <pubDate>Fri, 29 Oct 2010 13:48:41 Z</pubDate>
      <itunes:author>Alec Mihailovs</itunes:author>
      <author>Alec Mihailovs</author>
    </item>
  </channel>
</rss>