@toandhsp  I was looking for it of right triangles with legs   x+3  and  y+3  and the hypotenuse  x+y . Radius of the inscribed circle is  3  for any such triangle.

isolve((x+3)^2+(y+3)^2=(x+y)^2):
op(select(s->rhs(s[1])>0 and rhs(s[2])>0,[%]));

{x = 4, y = 21}, {x = 5, y = 12}, {x = 6, y = 9}, {x = 9, y = 6}, {x = 12, y = 5}, {x = 21, y = 4}

@toandhsp   [[4,-2], [4,7], [-8,-2]]

see  http://www.mapleprimes.com/posts/200122-The-Largest-Incircle-And-The-Smallest-Covering-Circle-

@Markiyan Hirnyk  Nice and simple solution. Now I am preparing two procedures in which  both problems will be solved exactly (symbolically). For example, the exact answer to this example is

[[57/22, 43/22], 1/242*1625^(1/2)*242^(1/2)];
evalf[15](%);

@sainimu78  Syntax error! Under the symbol  ∑  the command  add  should not be used.

@isbah It is impossible to obtain solution explicitly in the form of formulas. The reason is that the solutions are the roots of the polynomial of high degree  RootOf(...) . It is well known that if the degree of a polynomial more than 4, then the roots are not expressed as a function of its coefficients. But the procedure

f:=a->fsolve({0.5704+0.5211*exp(-0.6569*x)-0.03849*y=a, 0.561+0.5284*exp(-0.6479*x)-0.03929*y=a}):

can be used also successfully as  explicit formulas, for examples, to find the values ​​at certain points, build plots, etc.

@Carl Love  In initial task, we don't need use these commands, because they slow down the program and we can do the work without them.

Example:

A:=[1,4,8]: B:=[8,9,12]:
type(sqrt(A[1]^2+A[2]^2+A[3]^2), posint);
type(sqrt(B[1]^2+B[2]^2+B[3]^2), posint);
type(sqrt((A[1]-B[1])^2+(A[2]-B[2])^2+(A[3]-B[3])^2), posint);

                                               true

                                               true

                                               false

The main disadvantages of the source code - it is slow and duplicates occurs.

It is too difficult and it is not well suited for this forum. Look at the links found by Google

https://www.google.ru/#newwindow=1&q=Maximum+area+rectangle+inscribed+in+a+convex+polygon

@Markiyan Hirnyk  Of course my reasoning is incorrect. Thank you for this error message.

@Markiyan Hirnyk  I meant that, for example, both numbers  F1(x0, y0, z0))  and  F1(x5, y5, z5)  are positive or both ones are negative numbers. In both cases should be  F1(x0, y0, z0))*F1(x5, y5, z5)>0 . Function  sign  we have not use.

Unfortunately, Maple does not solve even the simplest equations containing the function  floor :

solve(floor(x) = 1, x);

                RootOf(floor(_Z)-1)

@Markiyan Hirnyk  If you remove the empty line   [``,``,``,``,``]  then there are no problems.

@Markiyan Hirnyk  In the attached file there is an ordinary Fourier series.

@Markiyan Hirnyk  See the timing in M 12:

restart;
a := rsolve({X(0) = 0, X(1) = 1, X(2) = 2, X(n) = n*(X(n-3)+X(n-2)+X(n-1))}, X, makeproc):

t:=time(): ceil(log[10](a(10000))); time()-t;

                                                                 35664

                                                                 11.578

restart;
NumberOfDigits:=proc(N)
local X, n;
X[0]:=0; X[1]:=1; X[2]:=2;
   for n from 3 to N do
   X[n]:=n*(X[n-3]+X[n-2]+X[n-1]);
   od;
length(%);
end proc:

t:=time(): NumberOfDigits(10000); time()-t;

                                                           35664

                                                           0.297

restart;
X:= proc(n)
option remember;
n*(X(n-3)+X(n-2)+X(n-1))
end proc:

t:=time():  X(0),X(1),X(2):= 0,1,2:
length(X(10000));  time()-t;

                                                            35664

                                                            0.312

I think the reason is that  length  command a lot more effective than ceil@log[10] .

@Carl Love  Unfortunately, in M 12 your code works incorrectly. Easy adjustment solves the problem:

X:= proc(n)

option remember;

     n*(X(n-3)+X(n-2)+X(n-1))

end proc:

X(0),X(1),X(2):= 0,1,2:

length(X(2013));

              5782