@David1 An Array has 4 operands: indexing function, dimension ranges, contents, options, which you can see by

op(sols)  or individually by op(1,sols) etc,

though there is no indexing function in this case.

@David1 If it fails to find the global minimum, it has found the lowest local minimum that it can.

The solutions are in an Array, so ArrayDim(sols) will return 1..31, 1..3. To get the 31 use rhs(ArrayDim(sols)[1])

@David1 Your worksheet runs for me. DirectSearch is not a build-in package, so has to be downloaded from the cloud (for Maple 2017) or  installed from the applications centre, then I think it will run. For a general optimization problem, you could add a very sharp negative spike to the surface that was a global minimum and might be missed (see the DirectSearch:-GlobalOptima examples), but in real life you may know that your surface has some nice properties or is sufficiently smooth or is convex.

fsolve uses many different algorithms, and is likely using a more efficient method that only works for polynomials when that is the case.

@David1 I think using only 6 variables (3 Euler angles and the three components of the translation vector) makes the optimization problem tractable, with no constraints except the ranges of the angles. I corrected the range of theta in worksheet PtsAndLines.mw above, and now there seems to be a unique minimum. DirectSearch:-GlobalSearch looks for all minima, and the examples in the documentation are very convincing that it succeeds in this, though of course it might miss some for extreme problems (which yours isn't, I think). Once you trust it, DirectSearch:-GlobalOptima gives only the minimum, but not any faster. Hope this solution works for you.

@David1 I think for numerical work it doesn't matter about the trig functions - they are still evaluated efficiently and the complexity of the problem depends more on the number of variables. Here is my attempt for a 5 point problem. Optimize:-Minimize minimizes in 0.063 s and DirectSearch:-GlobalOptima takes 117 s. GlobalOptima works hard at finding a global mimimum and could be helped further, but cannot give a guarantee. In this case the same sum of squares is obtained with the same translation vector, so I'm thinking that there is still some redundancy in how I've bounded the Euler angles but I didn't think too hard about that.

PtsAndLines.mw

[Edit - range of theta changed]

@David1 It seems the problem is indeed too big for Groebner. For Optimization, your rotation matrix has redundancy - I would formulate it in terms of the three Euler angles:

with(Student[LinearAlgebra]):
Rx:=RotationMatrix(phi,<1,0,0>);
Ry:=RotationMatrix(theta,<0,1,0>);
Rz:=RotationMatrix(psi,<0,0,1>);
R:=Rz.Ry.Rx;

So then there are 6 variables for your minimization, three angles and three translation components.

@David1 Then you should use Maple's Optimization package, e.g., Optimization:-Minimize. DirectSearch also is a possiblility. You always have problems knowing whether or not you have found the global mimimum, but there are lots of algorithms that fairly reliably avoid getting stuck in local minima. 

@David1 Use convert(.223,rational) to convert floating point numbers to rationals. It should work also on an expression or equation. Once you have the final 18 equations with rational coefficients, if you upload a worksheet containing them, it will be clearer what any remaining issues are - use the large green up arrow to upload a worksheet.

[Edit: I agree with the directsearch approach if you don't need all solutions - you do say "I get the solution I expected" as if you only want one, and you know when you have it, so can specify the constraints to directsearch]

I played around with different methods, initmesh, and using b as a continuation parameter. Everything is fine for b=0..0.45, but can't get a solution for b>= 0.4600, so probably the equations have some sort of singularity there. Perhaps someone else has a suggestion. 

@Carl Love I was just (carelessly) following the OP when I titled the post; I do understand the difference between parameters and arguments. The $ is a nice catch; I didn't expect a "parameter sequence" to contain anything but a parameter. But the name ` $` can be a parameter, so your procedure npar has a problem with this case:

f:=proc(` $`);` $`^2;end proc;

npar(f); gives zero.

Interestingly, 

f:=` $`->` $`^2;
gives the error message:

Error, end-of-parameters marker cannot be used in functional operators

Edit: I should have checked that it worked:
f:=proc(` $`);` $`^2;end proc;

f(2) gives:

Error, invalid input: too many and/or wrong type of arguments passed to f; first unused argument is 2
So it looks syntactically like a parameter, but is actually an end of parameter marker.

You win here!

@DSkoog I downloaded the package from the application centre, opened the file "The Lattice Package Users Guide.mw" in Maple 2017.1, clicked on the workbook tab and when I attempted to save it as a workbook I got an error message: "A problem was encountered when saving the workbook. node 'Section' cannot have child 'Pagebreak' " which suggested I delete the pagebreaks. After I did that I could reproduce the process and save the workbook. I will send you the files offline.

@vv Good point.

@tomleslie If there is a bug as @james1482  asserted, it may be selective in some way, and not necessarily lead to failure for all graphs with loops, and as you show the trace 3 ones at least don't raise an error. It may be that IsIsomorphic is not working properly in these cases, though.

I was hoping the package might be fixed up to handle graphs with loops, but it sounds like it will just be fixed to raise error messages in the presence of loops. 

@tomleslie I agree - when I run your worksheet without change (Maple 2015.2, Windows 7) I get 25 errors and then the same 25, but not 3.

I can get various algorithms to work for the 2x2 case, but all fail for the 3x3 case, with the same error.

@mehdibaghaee If you use F[3,1] etc in your integrand, then you can't use the variable F to assign the result of the dchange - just change it to FF or something else