@Susana30  You've already been shown one way. Here's another way, where we replace this region with a polygon for coloring:

restart;
f:=y->y^2-3: g:=y->y-y^2:
L1:=seq([g(y),y], y=-1..3/2, 0.1):
L2:=seq([f(y),y], y=3/2..-1, -0.1):
plots:-display(plot([L1], color=red, thickness=3),plot([L2], color=blue, thickness=3), plottools:-polygon([L1,L2], color="LightGreen"), scaling=constrained);

@minhthien2016  Why do you think this angle is 3*Pi/4?

Here is another solution to your problem if S(0,0,sqrt(2)/2). This solution uses the symmetry of this polyhedron about the ASC plane:

restart;
local D, O:
with(Student:-MultivariateCalculus):
A, B, C, D, S :=  [0,0,0], [1,0,0], [1,1,0], [0,1,0], [0,0,sqrt(2)/2]:
L:=Line(C,S);
O:=Projection(B, L);
v1:=convert(B-O,Vector);
v2:=convert(D-O,Vector);
Angle(v1,v2); # The answer

@Carl Love  Through the points M, M1, M2 (from my answer) we draw a plane intersecting the line of intersection of our half-planes at point O. Consider the quadrilateral OM1MM2. Obviously, the sum of the angles M1MM2 and M1OM2 is equal to Pi (we are looking for the angle M1OM2).

@JAMET  Insert  the line  s1:=polygon([A,M,N,P], color = "Pink", transparency = 0.7);  and add  s1  to the arguments of the  display  command.

Maple 2018.2 instantly returns a solution in terms of RootOf:

Download ode.mw

@JAMET  Your problem can be solved much more simply if you notice (it is not difficult to prove) that angles CHK and CDK are right angles. Therefore, the circle with diameter CK passes through points C and K. Therefore, O is the midpoint of segment СК.

@JAMET  Obviously ABCD is a rectangle and therefore the center of the circle described around it coincides with the center of this rectangle, that is  O(a/2, b/2, 0) . Maple is not needed here at all.

I looked at your code, but I didn't understand the meaning of what you are doing. In the title you write about the circumscribed circle of some triangle. What triangle do you mean?

Here is the solution to your example in Maple 2018.2:
 

Download no_warning_messages.mw

@JAMET  See code below. x, y, z are the barycentric coordinates of point M:

restart;
with(LinearAlgebra):
A:=<1,-3,0>: B:=<-2,1,1>: C:=<3,1,2>: M:=x*A+y*B+z*C:
{DotProduct(B-A,C-M)=0, DotProduct(C-A,B-M)=0, x+y+z = 1}:
solve({DotProduct(B-A,C-M)=0, DotProduct(C-A,B-M)=0, x+y+z = 1}); # barycentric coordinates of M
'M'=eval(M, %); # cartesian coordinates of M

@vv But  evala  works in other similar situations too. For example

restart;
M:=<a, -b; a/b, a*b>;
a:=2: b:=3:
evala(M);

@Susana30  For this example use the  surd  function:

A:=int((x^2)^(1/3), x);
surd(A^3, 3);

@Scot Gould  If we have already guessed that we can use the well-known difference of squares formula, then the process the factoring can be automated in Maple. The sum of any two terms can be expanded like this (Maple considers   a - b  as the sum  of operands  a  and  -b ):

restart;
expr:=x^10/36 - 4/25*y^24*z^8;
subsindets(expr, `+`, t->(sqrt(op(1,t))-sqrt(-op(2,t)))*(sqrt(op(1,t))+sqrt(-op(2,t)))) assuming positive;

It is worth noting that if you place an arrow over a symbol, then Maple does not consider this object as a vector, but considers it to be just a symbol:

                                     
                                      symbol

@SHIVAS 

subs({-5*sqrt(Sc*(S*Sc*S+4*Kr))=m1+5*S*Sc, ((-S*Sc+sqrt(Sc*(S^2*Sc+4*Kr))))=2*m2},Phi0_exact);