Being a Mackie I cannot comment on Windows specifica, but I have seen similar things happening on OS X; esp on 32-bit versions on PPC.
32 bit systems in general don't allow more than 2 GB per process (and likely this includes some system overhead). If your numbers are given by Maple directly (e.g. by the ticker at thebottom) the process likely allocates or uses more than what is shown (certainly that is the case in OS X). Then there is the GUI process which in itself can run up to GB size esp. when doing large plots, and its memory is definitely not included in the ticker readout. However, the GUI memory consumption does not count against the limit on the mapleserver process as they are separate (and I assume here Maple uses the same client-server model in Windows as it does in OS X). If you can, check what the GUI process is doing (on OSX Maple18 would be the GUI whereas mserver is the Maple kernel, the process that crunches the numbers).
So it seems like you are running into memory limits. Just adding RAM sticks may help, but I would doubt it. I think you want a 64 bit setup AND more physical RAM. I certainly see less of such issues on my OS X 10.6 64bit setup than on my 32 bit setup.
Having said that, I do find the GUI process flaky in general and some worksheets just stress it for no obvious reason. My usual defense is to avoid plotting (which helps) or to fine tune the number of points used in a plot (helps also).
I have had thoughts about checking and increasing the heap space for the Java process that runs the GUI. But I haven't figured out yet where I can insert the relevant cmd-line switches.