DJ, I use ArrayTools:-Alias quite often, but I might never have though of what you did in IR4. That is very neat and very, very efficient. Since ArrayTools:-Alias produces a new rtable, without actually copying the data, it costs very little. It's not even clear that it'd be much benefit to "scatter-point index" directly on M (even if one could thus access the antidiagonal all at once, without a loop). It's possible to offload most of the indexing to the time of element access, using an indexing function. Of course, this induces a huge penalty at access time (which I don't measure because it depends on the final usage, which includes printing!). Really, I just show this for fun.

`index/antident`:=proc(idx,M,val)
if nargs>2 then return NULL; end if;
if idx[1]=op([1,2],M)-idx[2]+1 then -1 else 0 end if;
end proc:

IR5 := proc(n,{compact::truefalse:=true},$)
   local M;
   M:=Matrix(n,n,shape='antident',storage=empty);
   if not compact then
      Matrix(M);
   else
      M;
   end if;
end proc:

st,ba,bu:=time(),kernelopts(bytesalloc),kernelopts(bytesused):
IR5(100):
#IR5(100,compact=false): # slow
#IR4(100):
time()-st,kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu;

Something that puzzled me. If I debug the indexing function, and call IR5(1) with a colon to suppress printing, then why is the indexing function called twice!? acer

Does it speed it up, to have the inner assignment done by another pair of explicit loops, instead of creating many new scaled copies of Matrix b?

KroneckerProduct2 := proc(a::Matrix,b::Matrix)
        local i,j,aRow,aCol,bRow,bCol,p,k,l;
        aRow,aCol := LinearAlgebra:-Dimension(a);
        bRow,bCol := LinearAlgebra:-Dimension(b);
        p := Matrix(aRow * bRow,aCol * bCol);
        for i from 1 to aRow do
        for j from 1 to aCol do
                for k from 1 to bRow do
                for l from 1 to bCol do
                    p[(i-1)*bRow+k,(j-1)*bCol+l]:=a[i,j]*b[k,l];
                end do;
                end do;
        end do;
        end do;
        p
end proc:

Hopefully I did it correctly. All those a[i,j]*b Matrices are collectible garbage. With enough of them produced, Maple can slow right down. acer

Some person's been doing that, now and then, for months. The admins clear off the accounts pretty quickly. It's always some random name. I suspect that it's done by creating blog entries and then deleting them, in such a way that the points are retained. I haven't figured out how, exactly.. When I reload and notice that it's active, sometimes I try to get a glimpse of a blog entry, so that I can reply and so "fix" it as uneditable. I've never seen an entry though, so I guess that they get removed quite quickly. I've never tried subscribing to the RSS feed, though. I wonder why the admins haven't fixed the loophole. acer

I find the documentation for CodeTools[Profiling] to be obscure. But nprofile, that is easier (and it is easier still, if one also reads the help-page for exprofile). Maybe I rely too much on the debugger. But the profiling did show some interesting things to look for inside the debugger. Here are some things which don't look so right. The procedure returned by, RandomTools:-Generate('distribution(Normal(0, 1))', makeproc = true)) will itself call Statistics:-Sample(distribution). But, as discussed in other threads recently on mapleprimes, that call to Sample() will in turn create and return a new distinct procedure! Surely that is the wrong way to use Sample(), in this circumstance. My initial surmise turns out to be true after all -- this usage is resulting in a new randon number generating procedure being created for each entry in the final Vector. Let's look at the procedure returned by RandomTools:-Generate. Instead of the way it does, answer := Statistics:-Sample(distribution); answer(1)[1] it would be better to do, Statistics:-Sample(distribution,1)[1]; Even that is not perfect, as it creates a new 1-Vector for each call. Why can't RandomTools:-Generate call Statistics:-Sample(distribution) just once, and stuff the resulting procedure into the proc that RandomTools:-Generate returns? I can't see how it isn't seriously wrong for RandomTools:-Generate(...,makeproc=true) to create a procedure which calls Statistics:-Sample(distribution) each time it's invoked. Another unfortunate thing is this. Every time Sample(distribution) is done, it calls ProcessDistribution(). And each time ProcessDistribution() is called, that in turn calls Statistics:-Distributions:-Inventory:-ListKnownDistributions. And ListKnownDistributions() does a lexorder sort on all the distributions in DISTRIBUTION_TAB. 33% of the memory usage in the original submitter's example is undergone inside ListKnownDistributions(). Isn't that a problem in ProcessDistribution() more generally? Couldn't the DISTRIBUTION_TAB be ordered up front. If anything ever added to it, then it could be re-ordered only then. acer

These are just ideas for amending Sample to act inplace. They don't actually work. Joe, the ability to handle ranges nicely, which you recently showed for the copy case, could be trickier. with(Statistics): X := RandomVariable(Normal(1,0)): # This next could operate inplace on a Vector argument, # and fill its first N entries with values. S:=Sample(X,N,V); # This next could return a procedure that expects # two arguments of types posint,Vector. S:=Sample(X,inplace=true); S(N,V); # This next could be the same as the current Sample(X). S:=Sample(X,inplace=false): S(N); An optional offset= parameter is also possible, as opposed to a range. It would look slightly awkward, to allow an offset for dealing with the inplace Vector when that's not currently supported in the copy (current, non-inplace) implementation. Also, if were to take an offset option, then why not a stride option too? Thinking about it a bit harder, maybe neither offset nor stride might be implemented since one might be able to modify the programmatic access to the entries to get the same effects? acer

Very nice, as proof of concept, and in performance. > (st,ba,bu):=time(),kernelopts(bytesalloc),kernelopts(bytesused): > test4(30000,100): > time()-st,kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu; 0.322, 3276200, 3816512 Even the popular C rng's re-use arrays like this. Your code shows the benefit so directly. Both the speed and memory usage are lower. With normal use of current Statistics, it's only possible to get one of the two so low while the other remains several powers of 2 greater. An interesting question then becomes: should Statistics:-Sample(X) return a proc that allowed an optional Vector argument, or would processing for that option add too much overhead? Would it better to have a new routine like SampleRtable(X) which returned a proc which expected exactly two arguments? What's the tradeoff, between the confusion of more user-level routines and the ability to get very low overhead generated procedures? I mention the idea of two arguments to the procedure returned by SampleRtable(X), since one might only want to generate less random numbers than fit in the rtable. acer

That's quite neat, Joe. Looking at the help-page, ?Statistics,Sample , I don't see where even the simpler invocation Sample(X) is shown. When it is called as Sample(X), for a random variable X, then omitting the second parameter N will result a procedure being returned. That procedure can itself be polled, to return separate Vector samples. This usage appears in several of the examples higher up in this thread. The first person to show it may have been user "alex". The ?Statistics,Sample help-page could be filled with more details on these otherwise hidden bits of functionality. I wonder what other eggs may be found in Statistics. acer

You are quite right, it does create just one generator procedure, sorry. When using the RandomTools generator, more Maple function calls are done to fill in each vector element, and that is also overhead. But garbage collection does indeed happen here, as evinced by the big difference between bytesused and bytesalloc and the regular memory usage "updates". I notice that you omitted the datatype=float[8] in your testing loop. If you add it in, then that might give some assurance that the memory use is not to store many software float Vectors. It doesn't clear it all up, by far, but adding it back might help lay the cause elsewhere. Why bytesalloc keeps growing, through your testing loop, when garbage collections are happening, that I don't know. It might need expert knowledge of inner workings of the garbage collection mechanism to explain it properly. I think that Statistics rather than RandomTools is the better tool for this sort of computation with many generated random numbers. I don't see how to use RandomTools:-Generate to produce a hardware datatype Vector without incurring the overhead of production of many intermediate software float objects or of multiple Maple function calls per element. acer

You have the RandomTools:-Generate makeproc call inside the Vector constructor call, as the initializer. Doesn't that mean that for every element it will try to create a distinct randon number generator? If so then that's a lot of (garbage) procedures being made. So instead, you could pull the makeproc of the random number generator outside of the Vector() constructor call? Ie, > restart: > kernelopts(printbytes=false): > st,ba,bu:=time(),kernelopts(bytesalloc),kernelopts(bytesused): > g := RandomTools:-Generate('distribution(Normal(0, 1))', makeproc = true): > Vector(10000,i->g()): > time()-st,kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu; 9.339, 11139080, 349536216 Notice how much bytesused grew by. Even better, you could use Statistics to do it very leanly. The random number generator produced above by RandomTools might itself supply "software" float numbers. But you intend to put them into a datatype=float[8] Vector. In a sense those intermediate software float objects all end up as collectible garbage. That's more garbage, on top of any unwanted distinct generator procedures. The Statistics package can be seen by inspection to call external code, the speed (and apparent lack of garbage production) of which indicates that it populates a hardware datatype Vector with no intermediate "software" float garbage. It generates hardware float random numbers directly, and places them into the hardware datatype Vector right away (externally). > restart: > kernelopts(printbytes=false): > st,ba,bu:=time(),kernelopts(bytesalloc),kernelopts(bytesused): > X := Statistics:-RandomVariable(Normal(0,1)): > S := Statistics:-Sample(X): > result:=S(10000): > rtable_options(result,subtype),rtable_options(result,datatype); Vector[row], float[8] > time()-st,kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu; 0.019, 1048384, 1384072 See this post for more discussion of using Statistics effiectively. acer

As an earlier post sugegsted, verify() may be used here, as a way to apply testfloat for the individual floating-point number comparisons. Digits:=50: x:=evalf(Pi): x2:=x+Float(6.0,-49): S1:={{[x,evalf(sqrt(2))],[x2,evalf(sqrt(2))]}} union {{x} union {[1.0,0]}}: S2:={{[x2,evalf(sqrt(2))],[x2,evalf(sqrt(2))+Float(7.0,-30)]}} union {{x} union {[1.0+Float(7.0,-29),0+Float(1.0,-30)]}}: verify(S1,S2,'set'('set'({'float'(5,digits=11,test=2),'list'('float'(1,digits=10,test=2))}))); But you also wanted it to work for scalar floats too, it seems. And maybe sets of scalar floats. F:='float'(5,digits=11,test=2): verify(x,x2,{F,'set'({F,'set'({F,'list'(F)})})}); verify(S1,S2,{F,'set'({F,'set'({F,'list'(F)})})}); It should be possible to programmatically construct the set of checks that you want, as long as you know the possible nestings in advance. I'm not really a big fan of the evalindets method posted, or other schemes which essentially do some sort of conversion to exact values. It's tricky to get that right, especially if the accuracy bound and precision and test model need ever need to vary. Unless the size of the data to be compared, or the level of nesting is large, I'd go with something like this using verify(). Of course, if those are big, then the number of unnecessary checks would be prohibitive. The number of combinations (scalars, lists, sets) would grow too big and be too expensive to apply. But even then, I would try to construct some other scheme that used testfloat() at its heart. acer

As an earlier post sugegsted, verify() may be used here, as a way to apply testfloat for the individual floating-point number comparisons. Digits:=50: x:=evalf(Pi): x2:=x+Float(6.0,-49): S1:={{[x,evalf(sqrt(2))],[x2,evalf(sqrt(2))]}} union {{x} union {[1.0,0]}}: S2:={{[x2,evalf(sqrt(2))],[x2,evalf(sqrt(2))+Float(7.0,-30)]}} union {{x} union {[1.0+Float(7.0,-29),0+Float(1.0,-30)]}}: verify(S1,S2,'set'('set'({'float'(5,digits=11,test=2),'list'('float'(1,digits=10,test=2))}))); But you also wanted it to work for scalar floats too, it seems. And maybe sets of scalar floats. F:='float'(5,digits=11,test=2): verify(x,x2,{F,'set'({F,'set'({F,'list'(F)})})}); verify(S1,S2,{F,'set'({F,'set'({F,'list'(F)})})}); It should be possible to programmatically construct the set of checks that you want, as long as you know the possible nestings in advance. I'm not really a big fan of the evalindets method posted, or other schemes which essentially do some sort of conversion to exact values. It's tricky to get that right, especially if the accuracy bound and precision and test model need ever need to vary. Unless the size of the data to be compared, or the level of nesting is large, I'd go with something like this using verify(). Of course, if those are big, then the number of unnecessary checks would be prohibitive. The number of combinations (scalars, lists, sets) would grow too big and be too expensive to apply. But even then, I would try to construct some other scheme that used testfloat() at its heart. acer

The routine testfloat is available as a flexible but powerful means to compare floating-point numbers. Why reinvent this functionality? For example, Digits:=50: epsilon:=Float(1.0,-30): for x in [0,0.0,1,sqrt(2),Pi] do x,testfloat(evalf(x),evalf(x+epsilon),5,digits=11,test=2); od; acer

This is the kind of post that makes me want bookmarking capability within Mapleprimes. acer

If you mean csc(x)^3 , then yes, it does seem to endlessly hint and use the Parts rule. You can do a start it off better by doing a sequence of change of variables (Change rule), starting with say u=csc(x)^2. Or you can roll some of those Changes together and apply the change u=sqrt(1-1/csc(x)^2) immediately. acer

Hi Alex, Only by mentioning such issues can they be resolved. The more get pointed out here, the more kinks can be ironed out. For example, the mod help-page which mentions &^ could have a new subsection specifically for 2D Math entry. This could work well. A user tries the obvious, encounters a problem, looks at the most obvious help-page, and sees a solution. Moving down to the very next example after &^, on that mod help-page, shows another item. How does one bring this next one about, as 2D Math input, in a Document? I just get an error. `mod` := mods I'm not sure how I'd want to define this next one, but why can't I enter it as 2D Math input? It gives me an error in 11.00, "invalid neutral operator". But as 1D Math input it seems fine. x &? w Why does entering this next one (and a few others I've seen) give an error message about an "invalid minus"? Where's the minus? x &\? w While I'm mentioning these things, why is there a `sum` item in the Expression palette, but no `Sum` item? They are not the same, even when the former returns unevaluated (hence the colour difference as 2D output). How does one use the summation symbol (sigma) in the large operators palette? All I ever got from it was an error message "invalid sum (need variable)" but with no apparent way to specify the variable of summation. And it just seems to be `&sum`. Where is inert Sum in the palette? Why is the fraction bar in 2D Math Input a/(b/c) (without the brackets) so much wider than that in a/b/c ? To enter those, type them in as shown below. Why should it matter to horizontal width considerations like fraction bars whether another fraction appears in the numerator vs the denominator? a/b/c a/b/c acer