g:=(u,v)->alpha(u,v)*beta(u,v);
h:=(u,v)->alpha(u,v)+beta(u,v);
f:=(u,v)->D[1](g)(u,v)*h(u,v);
f(u,v);
                          

Or at once:

f:=unapply(D[1](g)(u,v)*h(u,v), u,v);      

Your system is an overdetermined one, because it has 2 unknown functions, but 3 equations. I just removed the last equation and now everything is all right. You can solve the last equation separately, because it depends only on g(eta).

sol1 := dsolve([diff(diff(diff(f(eta), eta), eta), eta)+f(eta)*(diff(diff(f(eta), eta), eta))-(diff(f(eta), eta))^2 = 0, diff(diff(diff(g(eta), eta), eta), eta)+f(eta)*(diff(diff(g(eta), eta), eta))+2*g(eta)*(diff(diff(f(eta), eta), eta))-3*(diff(g(eta), eta))*(diff(f(eta), eta)) = 0, f(0) = 1, (D(f))(0) = 1, (D(f))(5) = 0, g(0) = 1, (D(g))(0) = -1, (D(g))(5) = 0], numeric, method = bvp);
plots[odeplot](sol1, [[eta, f(eta)], [eta, g(eta)]], eta = 0 .. 5, color = [red, blue], scaling = constrained);

To see the solution steps, you can use this command:

Student:-Basics:-LinearSolveSteps( "-(2*x+10)+8=4*(-x+1)", x);

Use  solve  instead of  fsolve :

eqn1:=-x*y^2+4*x=5;
eqn2:=(1/3)*x^3+y^2=1;

sol:=solve({eqn1, eqn2, x>=-10, x<=10, y>=-10, y<=10}, {x,y});
evalf(%);

For plotting use  plots:-implicitplot  command.

f := L->max(abs~(L));

Example of use:

f([1+I, 2-I, -1+3*I]);
                                           10^(1/2)

evalf~([solve(z^5-5*z^4-z^2-2)]);
[argument, abs]~(%);

But it is a correct answer:

limit(n^(3/2)*sum(2*k/(2*k+3), k=1..n), n=infinity);

                                    infinity

In Maple  e  is just a symbol, not  2.71828...

Should be:

int(exp(-x)*sin(x^2)/(2+x), x=1..infinity);
evalf(%);

Also, do not use square brackets to group expressions. They serve to form lists.

If you already know the result, you can just do :

is(abs(u+v)^2 + abs(u-v)^2 = 2*abs(u)^2 + 2*abs(v)^2);
                                          true

Your matrix equation can easily be reduced to a linear system:

gc();
restart:
with(LinearAlgebra):
A := <<a__11|a__12>,<a__21|a__22>>;
P := <<p__11|p__12>,<p__12|p__22>>;
Id := <<1|0>,<0|1>>;
eqn := Transpose(A).P+P.A =~ -Id;
solve({seq(seq(eqn[i,j], j=1..2), i=1..2)}, {p__11,p__12,p__12,p__22});
 

Although your system is linear with respect to the unknowns, it has 10 parameters. Even if Maple wrote formal expressions for the roots through the determinants, we would get huge expressions useless for use. I remind you that the matrix determinant 15 by 15 has in general 15! = 1307674368000 terms. Therefore, it is reasonable to solve the system by specifying the values of the parameters. See the example in the file.

System_new.mw

In 2d math mode that you use, the easiest way to calculate the determinant is to use two vertical bars, which you can enter from the keyboard:

restart;
u:=u0(x,y)+z*u1(x,y):
v:=v0(x,y)+z*v1(x,y):
A:=<a,b; c,d>;
B:=<diff(u,x), diff(v,y)>;
C:=A.B;

General approach in applying to your example:

restart; 
F := ((1/2)*a[2]-a[1]+(1/2)*a[0])*n^2+(-(1/2)*a[2]+2*a[1]-(3/2)*a[0])*n+a[0]; 
N := 3; 
var := [seq(a[n-1], n = 1 .. N)]; 
factor~([seq(op(k, foldl(collect, F, var[]))/var[N-k+1], k = 1 .. N)]);

This is a simple arithmetic:

480*1/(5+1+1/4);
                                      384/5

So the answer is $76.8