If you're using 32-bit versions of Maple and Windows, the operating system may limit Maple to 2GB of RAM.  Do you have 64-bit Windows?

The 34th Fermat number will be 2^34/8/1024^3 = 2GB in size.  You can create it, but I doubt you can do very much.  Every operation in Maple will allocate new memory for its result.  You can't tell it to explicitly reuse some memory.  So most operations are probably going to allocate one or two gigabytes, and unless you have 100 GB of RAM, your computer will probably grind to a halt.

You are getting killed by rational arithmetic. In Maple 13, the fastest way to do this is probably the following:

F := [x^32, x^32+(a^15+a^11+a^6)*x^31+(a^13+a^10+a^3)*x^23+
(a^11+a^7+a^2+1)*x^11+x^7+a^13+a^10+a^3+a^1+1]:
r := 1+a^31+a^23+a^11+a^7+a^2+a^32:
with(Algebraic:-RecursiveDensePolynomials):
f1 := rpoly(F[1],[x,a],r);
f2 := rpoly(F[2],[x,a],r);
TIMER := time[real]():
g := gcdexrpoly(f1,f2):
TIMER := time[real]()-TIMER;
g,s,t := op(map(rpoly, g)):

In Maple 14 gcdex will be faster. A better approach might be to do this computation mod p and reconstruct the solutions for s and t by lifting and rational reconstruction.

You want coefficients with respect to x, right? In your loop above you're computing them over and over. Start with this:

f := a00*x^123+a45*x^233+a02*x^123+a67*x^156+a47*x^67;
C := [coeffs(f,x,'M')]:  # coefficients
M := [M]:   # monomials

Powmod(x, 12345678987654321, x^32+x^26+x^19+x^15+x^13+x^11+x^9+x^8+x^4+x+1, x) mod 2;

Make a set of the equations you want to solve and use the solve command.

S := {2*x+y-1, x-y+z, z^2-1};
solve(S);

You could also just write solve({2*x+y-1, x-y+z, z^2-1});

A Groebner basis can produce all algebraic consequences of a set of polynomials by ordinary division. So in this example, you want to compute a Groebner basis G of {p1,p2} and divide q by G. To obtain the coefficients which express q directly in terms of {p1,p2}, you would normally express G in terms of {p1,p2} and also q in terms of G. Here is how to do it in Maple in graded reverse lexicographical order.

q := x1^3 + x2^3;
P := [x1 + x2, x1^2 + x2^2];
tord := tdeg(x1,x2);
G,M := Groebner[Basis](P,tord,output=extended);
r := Groebner[NormalForm](q,G,tord,'S');
C := inner(S,M);  # coefficients
q = expand(inner(C,P));

The output=extended option to Groebner[Basis] forces the use of the Buchberger algorithm and it keeps track of the quotients. It produces a matrix M with G = M*P which is a matrix vector product. The 4th argument to NormalForm also keeps track of the quotients. It produces a vector S with q-r = S*G. In this case the remainder r is zero. Then S*M is the coefficients you want, and q-r = S*M*P. Note that this method does not simplify the coefficients at all. If you do a large example they will become very big with many redundant terms. To find the simplest coefficients you can use Groebner bases for modules, which is essentially the same thing as Groebner basis for polynomials but with vectors of polynomials and it reduces terms across the whole vector. Maple can do that too but it's probably more than you want to hear at this point. Good luck!

Most expressions in Maple are not sorted, and the order of the terms can affect numerical results.

There is no way to do this that I know of.

Isn't there a Ubuntu package for this, called libcompat or compat-libstdc or something? Try installing that.

Wow, these equations really are unknown. Can you try reposting them in text format. Use the lprint command in Maple to print the equations as text.

Manually compute a remainder mod 2.

p := x^3+x+1:
f := x^2+x+1:
g := x^2+1:
Rem(f*g, p, x) mod 2;

The upper case Rem works mod p. Use lowercase rem to compute remainders over the rationals.

Niestety, nie odpowiadamy na pytania dotyczące pracy szkoły.

Be sure to update Maple to the newest version from the menu. Windows 7 is a supported platform for Maple 13. I haven't tried earlier versions, but I believe the problem you mention appeared on Vista so there may be updates for them as well.

Not really. Maple sorts everything according to its own order to make basic operations efficient.