I now got this response from the support team. I haven't tested it yet. Here is what the wrote:

Dear Florian,

on a Macintosh you have to change this limit by editing the following
file:

/Library/Frameworks/Maple.frameworks/Version/12/bin.APPLE_UNIVERSAL_OSX/
mserver_ulimit and the mserver_ulimit32 files in a text editor and
editing the line:

ulimit -d 524288 || ulimit -d 409600 || ulimit -d 307200

to something like:

ulimit -d 1000000 || ulimit -d 409600 || ulimit -d 307200

The first number sets the maximum amount of memory allowed for the
kernel, in KB.

It may be necessary to change the file permissions on the mserver_ulimit
file so you can save your changes. You can do this by using the context
menu (right-click) for the file and selecting "Get Info". The
permissions for the file should be set to read & write.

Some users have found that it may also be necessary to run Maple from a
terminal session in order for this to take effect. If that is the case
you can do the following:

1. Open a terminal window, navigate to the Maple bin directory
/Library/Frameworks/Maple.framework/versions/12/bin
2. Issue the command "ulimit -d #"
3. Launch Maple using "./xmaple &"

Where # is the amount of memory you want to have Maple be able to use.
You can either use the number you want or you can have unlimited where
it will use up to the system maximum.

Incidentally you can check in a Maple session how much memory Maple will
use before the kernel generates an allocation error by using the Maple
command

"kernelopts(datalimit);"

I hope this helps. Please let us know if you have any questions.

Regards,

Ian
Maplesoft Technical Support
www.maplesoft.com/support/