Thanks, itsme!

That answers my question here. I had almost come to that workaround yesterday, but was having trouble getting it to execute properly, so you've completely answered that question. I think that's such a useful thing to know how to do.

In following up on pagan's response to that question, I just uploaded my Maple code, which could really benefit from optimisation. Of course I'd appreciate any help with that that you might be able to give.

Very much appreciated! Thanks so much.

Daryl

Thanks, itsme!

That answers my question here. I had almost come to that workaround yesterday, but was having trouble getting it to execute properly, so you've completely answered that question. I think that's such a useful thing to know how to do.

In following up on pagan's response to that question, I just uploaded my Maple code, which could really benefit from optimisation. Of course I'd appreciate any help with that that you might be able to give.

Very much appreciated! Thanks so much.

Daryl

 

Hi Pagan,

Thanks for offering to help with the memory issues in my code. It does perform a numeric calculation. Basically, it starts by reading a file with heights and soil depths on a grid, profiles it to rank the heights of four nearest neighbours to each point and work out the probability that the soil will become saturated, based on soil depth. Then it just goes through and iteratively selects points as saturated (turns 0s to 1s) based on the PDF, and if already saturated it routes through to the lowest unsaturated point along the perimeter of locally connected 1s. It then needs to start over a given number of times and calculate an average pattern of saturation. When that number of Monte Carlo runs is done, I want to calculate a few things, save them, and then change the values of two parameters and start again. As you can see, it's all done in serial, so if it can be optimised to work that way, that would be fantastic.

It's already come a really long way through your suggestions, as I've added datatype=float[8] to a few Arrays and changed evalf to evalhf in a few instances where I was able to do so without causing errors; but the allocated memory is still noticably increasing with each run (albeit by about 5M rather than 100M), and the whole thing could probably go a lot more quickly. One more thing: those few calculations that are performed at the end of a set of Monte Carlo runs with a given set of parameters are sucking up huge amounts of memory, and I can't see why. This wasn't my primary concern, as I need to first be able to run through some number of Monte Carlo runs over the grid of parameters without having to be here to execute that every 20 mins; but it is another looming issue, so if you do see a quick way of making the very last for loop not suck up memory, that would be fantastic.

I've attached my .mw file, along with the .xlsx file that it reads from. I've moved everything into one execution block, but there are really about four parts: the initial physical profiling that ends with a garbage collection that works great; a procedure that gets called in order to determine the extent of the locally connected patch; the Monte Carlo for loop, with a given set of parameters; and then the calcullations that are based on the results with those parameters. Ultimately, I'd write two for loops around these last two steps in order to vary the parameters and get my results in serial. 

I hope the code doesn't look too much like I went off and invented a square-shaped wheel. As I said before, I haven't got a computing background.

Thanks again for responding (over here)!

Best,

Daryl

Hillslope_perc_Panol.mw PanolaBedrockDEM_ang.xlsx

 

 

Hi Pagan,

I'm having some memory allocation issues that I want to work around by going to command line and calling maple sessions in a loop, in order to restart it prior to doing independent calculations over a grid of parameter values---and I think I've basically been trying to do what you've suggested here as an alternative in the second paragraph. I'm just having a problem sorting out how to call maple in a for loop, define a few parameters according to the loop variable(s), and then execute a .mpl file with those param values, before quitting maple.

For example, if I've got HelloWorld.mpl on my desktop, which just contains the line print("Hello World!");, I can open Terminal on my Mac and execute

export MAPLE="/Library/Frameworks/Maple.framework/Versions/16"
for i in 1 2 3 4 5
do
$MAPLE/bin/maple /Users/Daryl/Desktop/HelloWorld.mpl;
done

This opens maple, prints "Hello World!", and then quits, five times over. But what I want to be able to do is print i each time. So I want to open maple inside the for loop like this, define something like a:=$i, and then execute a .mpl file with the line print(a); and then quit.

Ultimately, it would be wonderful to be able to process this in parallel, so if you have any tips on how to do that I'd really appreciate it, but even solving this more immediate problem would be huge for me, since I've only got 8 processors and can just execute that many for loops individually.

Thanks very much for any help you're able to offer!

Daryl

Hi Pagan,

I'm having some memory allocation issues that I want to work around by going to command line and calling maple sessions in a loop, in order to restart it prior to doing independent calculations over a grid of parameter values---and I think I've basically been trying to do what you've suggested here as an alternative in the second paragraph. I'm just having a problem sorting out how to call maple in a for loop, define a few parameters according to the loop variable(s), and then execute a .mpl file with those param values, before quitting maple.

For example, if I've got HelloWorld.mpl on my desktop, which just contains the line print("Hello World!");, I can open Terminal on my Mac and execute

export MAPLE="/Library/Frameworks/Maple.framework/Versions/16"
for i in 1 2 3 4 5
do
$MAPLE/bin/maple /Users/Daryl/Desktop/HelloWorld.mpl;
done

This opens maple, prints "Hello World!", and then quits, five times over. But what I want to be able to do is print i each time. So I want to open maple inside the for loop like this, define something like a:=$i, and then execute a .mpl file with the line print(a); and then quit.

Ultimately, it would be wonderful to be able to process this in parallel, so if you have any tips on how to do that I'd really appreciate it, but even solving this more immediate problem would be huge for me, since I've only got 8 processors and can just execute that many for loops individually.

Thanks very much for any help you're able to offer!

Daryl

I've temporarily solved this problem for myself by altering both of the sections of code I pasted in the previous quote. To the lists of values ulimit can be set as in lines 307 and 415 of the maple script, I've added `ulimit -d unlimited 1>/dev/null 2>&1 || ' and `ulimit -d unlimited 2>/dev/null || ', respectively (after changing the file to read & write). After executing the file, I find that my datalimit is infinity when I open Maple. In general, one can set this value to an arbitrary number, like 2048000, and that will be the external limit placed on the values listed in mserver_ulimit. For example, in the current distribution of the mserver_ulimit script, the relevant line is ulimit -d unlimited > /dev/null 2>&1 || ulimit -d 524288 > /dev/null 2>&1 || ulimit -d 409600 > /dev/null 2>&1 || ulimit -d 307200 > /dev/null 2>&1 ulimit -s 65536 > /dev/null 2>&1 || ulimit -s 32768 > /dev/null 2>&1 || ulimit -s 16384 > /dev/null 2>&1 So, if the external limit in the maple script is set to 2048000, the datalimit within Maple will not be able to be set above 524288, which is the next highest value in `mserver_ulimit', after the one that has been reset in `maple'. This would be the `hard limit' known to the application. Of course, the value `524288' could be set to any number up to 2048000, and that would become the hard limit. Alternatively, all values other than the first (unlimited) could be deleted, and the maple script would in turn set the limit to 2048000. For some reason, the developers in the past had set up this two tiered procedure for limiting the amount of memory Maple can allocate when run in OS X. Is this script just really outdated? Why can't the datalimit be allowed to be infinity on a Mac, with the simple user option of defining kernelopts(datalimit=whateverarbitraryvalueyouwant); within the opened application? Can a developer please answer this question, as well as update the current distribution of Maple so that the outdated limit of 512 MB is not set by default for Mac users?