@tdavid It's difficult to diagnose a problem without seeing the worksheet.  Upload it here.

To do that, in the window where you reply to this post click on the big fat green arrow.

It's not clear to me what you are asking.

Let's look at the first term.  The exponent of the cosine is 2*beta. The coefficient is a*lambda^2/2.  If we equate these we get

2*beta = a*lambda^2/2.

Is that really what you are asking?  If not, then say clearly what you want.

@vv You are quite right.  I didn't look closely enough.  Thanks for pointing this out.

@verdin The original eq[2] was undefined at (0,0,0).  The modified one is well-defined at (0,0,0), however it is not differentiable there, and therefore the equation cannot be linearized.  Here are the details.

Download mw.mw

In your worksheet you have eq[1], eq[3], eq[3].  I assume that the middle one is meant to be eq[2].

Let's evaluate your eq[2] at (x1,x2,x3)=(0,0,0):
eval(eq[2], [x1(t)=0, x2(t)=0, x3(t)=0]);

We see that Maple complains with:
Error, numeric exception: division by zero

This says that your eq[2] is undefined at (x1,x2,x3)=(0,0,0), therefore linearizing about (x1,x2,x3)=(0,0,0) is not meaningful. You need to think that over.

@Carl Love As the parameter y changes, the graph of the function makes a transition from having two extrema (the blue curve below), to having a flat plateau (red curve) to monotone decreasing (green curve).  At the plateau both the first and second derivatives are zero.

@jan123 The labels (x and y in this case) are taken from the names of the variables in the equations in sys, but you may override them by specifying labels explicitly.  For instance,

There is no direct way of specifying label positions; they are placed automatically by Maple. For full control, you may disable the labels altogether, as in:
p1 := DEtools:-DEplot(sys, [x(t),y(t)], t=-2..2, [ic], x=-1..1, y=-1..1,
    linecolor=blue, color=red, thickness=1, tickmarks=[0,0],
    labels=["",""]);

and then design your own labels with the help of plots:-textplot(), as in

p2 := plots:-textplot([[0.9,-0.1,e__1], [0.1,0.9,e__2]]);

and then display p1 and p2 together:

plots:-display([p1,p2]);

@Carl Love I also see the behavior described by the OP.  Perhaps that's specific to Linux.  Here's what I see:

Download mw.mw

@Alger

Download orthogonality.mw

@Kitonum That's a nice animation—the red segments stand out and nicely demonstrate their equal lengths and orthogonality.

It is interesting that when one of the sides of the quadrilateral degenerates to zero length, that is, when the quadrilateral turns into a triangle (even a right triangle!), the theorem's assertion still remains valid and nontrivial.  I don't recall having seen that special case stated anywhere.  One would think that Euclid would have noticed that.

@Markiyan Hirnyk I invite you to go back and read (carefully now!) the reply under the heading "Yes, I have the response" and don't pull things out of your imagination, or somewhere else.

@Markiyan Hirnyk I am not going to stop you from laughing, but consider reading carefully what is written rather than what you imagine is written.  And an apology will not hurt either.

@Markiyan Hirnyk The introduction of biorthogonality in this discussion is a red herring.  For the purpose applying the Fourier transform, what is needed is the orthogonality of the basis functions which in the context this problem consists of the family
{ C(n,x), S(m,x) }_{n = 0...∞, m = 1...∞},
and where the orthogonality is relative to the given weighted L² inner product.  I pointed out that family of functions given in the original post was not orthogonal, that's why an attempt to apply the Fourier transform was leading to an incorrect result.   The family of functions in the amended post is orthogonal, therefore applying the Fourier transform is legitimate in that case.  That's all.

@Alger OK, then.  After that change, the issue raised in your original post should go away.

@vv Nice application of complex algebra!