@David1 I think for numerical work it doesn't matter about the trig functions - they are still evaluated efficiently and the complexity of the problem depends more on the number of variables. Here is my attempt for a 5 point problem. Optimize:-Minimize minimizes in 0.063 s and DirectSearch:-GlobalOptima takes 117 s. GlobalOptima works hard at finding a global mimimum and could be helped further, but cannot give a guarantee. In this case the same sum of squares is obtained with the same translation vector, so I'm thinking that there is still some redundancy in how I've bounded the Euler angles but I didn't think too hard about that.

PtsAndLines.mw

[Edit - range of theta changed]

@David1 It seems the problem is indeed too big for Groebner. For Optimization, your rotation matrix has redundancy - I would formulate it in terms of the three Euler angles:

with(Student[LinearAlgebra]):
Rx:=RotationMatrix(phi,<1,0,0>);
Ry:=RotationMatrix(theta,<0,1,0>);
Rz:=RotationMatrix(psi,<0,0,1>);
R:=Rz.Ry.Rx;

So then there are 6 variables for your minimization, three angles and three translation components.

@David1 Then you should use Maple's Optimization package, e.g., Optimization:-Minimize. DirectSearch also is a possiblility. You always have problems knowing whether or not you have found the global mimimum, but there are lots of algorithms that fairly reliably avoid getting stuck in local minima. 

@David1 Use convert(.223,rational) to convert floating point numbers to rationals. It should work also on an expression or equation. Once you have the final 18 equations with rational coefficients, if you upload a worksheet containing them, it will be clearer what any remaining issues are - use the large green up arrow to upload a worksheet.

[Edit: I agree with the directsearch approach if you don't need all solutions - you do say "I get the solution I expected" as if you only want one, and you know when you have it, so can specify the constraints to directsearch]

I played around with different methods, initmesh, and using b as a continuation parameter. Everything is fine for b=0..0.45, but can't get a solution for b>= 0.4600, so probably the equations have some sort of singularity there. Perhaps someone else has a suggestion. 

@Carl Love I was just (carelessly) following the OP when I titled the post; I do understand the difference between parameters and arguments. The $ is a nice catch; I didn't expect a "parameter sequence" to contain anything but a parameter. But the name ` $` can be a parameter, so your procedure npar has a problem with this case:

f:=proc(` $`);` $`^2;end proc;

npar(f); gives zero.

Interestingly, 

f:=` $`->` $`^2;
gives the error message:

Error, end-of-parameters marker cannot be used in functional operators

Edit: I should have checked that it worked:
f:=proc(` $`);` $`^2;end proc;

f(2) gives:

Error, invalid input: too many and/or wrong type of arguments passed to f; first unused argument is 2
So it looks syntactically like a parameter, but is actually an end of parameter marker.

You win here!

@DSkoog I downloaded the package from the application centre, opened the file "The Lattice Package Users Guide.mw" in Maple 2017.1, clicked on the workbook tab and when I attempted to save it as a workbook I got an error message: "A problem was encountered when saving the workbook. node 'Section' cannot have child 'Pagebreak' " which suggested I delete the pagebreaks. After I did that I could reproduce the process and save the workbook. I will send you the files offline.

@vv Good point.

@tomleslie If there is a bug as @james1482  asserted, it may be selective in some way, and not necessarily lead to failure for all graphs with loops, and as you show the trace 3 ones at least don't raise an error. It may be that IsIsomorphic is not working properly in these cases, though.

I was hoping the package might be fixed up to handle graphs with loops, but it sounds like it will just be fixed to raise error messages in the presence of loops. 

@tomleslie I agree - when I run your worksheet without change (Maple 2015.2, Windows 7) I get 25 errors and then the same 25, but not 3.

I can get various algorithms to work for the 2x2 case, but all fail for the 3x3 case, with the same error.

@mehdibaghaee If you use F[3,1] etc in your integrand, then you can't use the variable F to assign the result of the dchange - just change it to FF or something else

@nk2016  You will get the right result if you paste into 1-D Maple (try control-M on the line before you paste), but if you paste into 2-D, the != characters change into "not equal to" and you get a quite different result.

@mehdibaghaee 

The idea is that if you are solving, say, dy(x)/dx = -k*y(x) with y(0)=1 (y(x) is "unknown") the solution is y(x)=exp(-k*x), and if you change variables x=2*X, then the solution is y(X)=exp(-2*k*X) in which y(X) is a different function than y(x). But if you have a known function like sin(x) then you want sin(x) to change to sin(2*X)
 

Download inttt3.mw

@mehdibaghaee  add known=u0 to get this.

@mehdibaghaee Looks like a bug

Using dchange gives the expected double integral
 

Download inttt2.mw