MaplePrimes - answers and comments on Question, NLPSolve Matrix Form

NLP or QP

acer — Tue, 10 Jan 2012 14:52:43 Z

Is this just a QP minimization problem in disguise, rather than an NLP minimization problem? By which I mean, a quadratic objective and linear constraints?

Sure, your con3 constraint is nonlinear, involving a 2-norm. But it is the only place that variable `r` appears in constraints, and your objective is simply `r`. So can't you just make the norm part of con3 as a new objective, and then get rid of con3 as a constraint?

Note that the setup of this problem gets expensive quickly, as `nstock` grows. And there is a lot of room for improvements to the first parts which generate the randomized data. So, unless all that is just a stand-in for some other means of data acquisition then you can get a bit of speedup just by refining all that code. I mean, everything up to the Cholesky decomposition of `Cov`. On the machine used for the timings below, at nstock=200 the construction of `Cov`, `R` and the constraints takes about 70 seconds. What comes before constructing `Cov` can likely be reduced to under 4-5 seconds.

I set nstock=200 and compared these two Optimization calls below. I'll leave out all the set up here, except for the constraint assignments,

con1 := add(W[i], i = 1 .. nstock) <= 1:
con2 := EV.W-pr >= 0:

NRW2 := Norm(R.W, 2, conjugate = false):
con3 := NRW2 - r <= 0:

# As you solved it with NLPSolve...

sol1 := CodeTools:-Usage(NLPSolve(r,{con1,con2,con3},seq(w[i]=0..1,i=1..nstock)));

memory used=1.74GiB, alloc change=55.49MiB, cpu time=60.68s, real time=60.70s

sort(select(t -> rhs(t)>0 and lhs(t)<>r, sol1[2]));

        [w[21] = 0.0270237593173089, w[25] = 0.0322056751304918, 

          w[35] = 0.000847852100225448, w[67] = 0.0492087037211113, 

          w[78] = 0.0438242963135371, w[86] = 0.0416518751919245, 

          w[88] = 0.0417036500369194, w[89] = 0.0717631323305688, 

          w[100] = 0.00837739752003451, w[107] = 0.137179654344253, 

          w[109] = 0.107669160523004, w[125] = 0.0289124908386128, 

          w[131] = 0.160793105594057, w[153] = 0.0236924836513717, 

          w[157] = 0.0341339386566807, w[180] = 0.0119978162730577, 

          w[185] = 0.0405857945037750, w[190] = 0.129027000241401, 

          w[192] = 0.00940221371168578]

r = eval(r, sol1[2]);

                            r = 2.03629907765324

# And now...

sol2 := CodeTools:-Usage(QPSolve(NRW2^2,{con1,con2},seq(w[i]=0..1,i=1..nstock)));

memory used=448.67MiB, alloc change=0 bytes, cpu time=7.71s, real time=7.72s

sort(t -> rhs(t)>0 and lhs(t) <> r, sol2[2]));

        [w[21] = 0.0270237589337289, w[25] = 0.0322056750319796, 

          w[35] = 0.000847852090717340, w[67] = 0.0492087042316293, 

          w[78] = 0.0438242958978819, w[86] = 0.0416518740369907, 

          w[88] = 0.0417036488838085, w[89] = 0.0717631314373649, 

          w[100] = 0.00837739703763107, w[107] = 0.137179652525067, 

          w[109] = 0.107669163079282, w[125] = 0.0289124902420816, 

          w[131] = 0.160793108650386, w[153] = 0.0236924831211481, 

          w[157] = 0.0341339380256316, w[180] = 0.0119978167550648, 

          w[185] = 0.0405857946727649, w[190] = 0.129027001732921, 

          w[192] = 0.00940221361388309]

r = sqrt(eval(NRW2^2, sol2[2])); # `r` in the original sense of `con3`

                            r = 2.03629907765382

Unless I did something quite wrong, those two attempts produce essentially the same solution, but the QPSolve is 7 times faster.

And the QPSolve use can likely also be improved using Matrix form for its input.

acer

acer, you are absolutly correct. It is a

alex_01 — Tue, 10 Jan 2012 15:32:32 Z

acer, you are absolutly correct. It is a QP minimization problem in disguise!

S=R'.R   where S is the covariance matrix and R is the upper triangular
factor of the Cholesky Decomposition matrix

w'.R'=(R.w)'    sqrt(x'x)=norm(x)

w'.S.w < r^2     w'.R'.R.w< r^2    (R.w)'.(R.w)< r^2    sqrt((R.w)'.(R.w))< sqrt(r^2)    norm(R.w)< r

Unfortunately, I already know that the QP and the NLP optimization produces
the same allocations (even though the objective functions returns different values since the
objective is different).

And yes, the first part can be made much more efficient.

# later comment-1
And yes you are right the two problems can be solved equivalent as:

problem := r, {con1, con4, con5}, seq(w[i] = 0 .. 1, i = 1 .. nstock):
NLPSolve(Norm(R.W, 2, conjugate = false), {con1, con2, con3, con4});
NLPSolve(problem) ;

My question remains though how do I convert it to NLP[Matrix form] using
Optimization:-Convert:-AlgebraicForm:-NLPToMatrix(problem) ?????????

# later comment-2
or alternativly how do you solve Min(Norm(R.W)) with LP?

restart:
randomize():
with(ListTools):
with(LinearAlgebra):
with(ArrayTools):
with(Statistics):
with(plots):
with(Optimization):

n := 40:
nstock := 10:
A := RandomMatrix(n, nstock, generator = -15 .. 15, outputoptions = [datatype = float[8]]):
Cov := CovarianceMatrix(A):
EV := Vector([seq(ExpectedValue(Column(A, i)), i = 1 .. nstock)]):
R := LUDecomposition(Cov, 'method' = 'Cholesky', output = 'U'):
W := Vector([seq(w[i], i = 1 .. nstock)]):

pr := .6*max([seq(ExpectedValue(Column(A, i)), i = 1 .. nstock)]):

con1 := add(W[i], i = 1 .. nstock) <= 1:
con2 := seq(W[i] <= 1, i = 1 .. nstock):
con3 := seq(W[i] >= 0, i = 1 .. nstock):
con4 := EV.W-pr >= 0:
con5 := Norm(R.W, 2, conjugate = false):

problem := r, {con1, con4, con5}, seq(w[i] = 0 .. 1, i = 1 .. nstock):
problem_matrix_form := Optimization:-Convert:-AlgebraicForm:-NLPToMatrix(problem):

QPSolve(Transpose(W).Cov.W, {con1, con2, con3, con4});
NLPSolve(problem);
NLPSolve(problem_matrix_form[[1, 2, 8, 9]])

con1..con2(con3)?

herclau — Wed, 11 Jan 2012 02:38:20 Z

@alex_01

con1 := add(W[i], i = 1 .. nstock) con2 := seq(W[i] con3 := seq(W[i] >= 0, i = 1 .. nstock):

does not work. I do not understand the syntax

@herclau sorry about that. I have to copy

alex_01 — Wed, 11 Jan 2012 03:41:00 Z

@herclau sorry about that. I have to copy from maple and then past into a
wordpad and then rearange everything to be able to post the code here.

I have uppdated the constraints...but it still dont work as it should.

let's see now...

acer — Wed, 11 Jan 2012 07:46:51 Z

@alex_01

You wrote, "later comment-2... how do you solve Min(Norm(R.W)) with LP?", and incidentally you were using the 2-norm.

Well, that is not a linear objective so it isn't a Linear Programming problem, so `LPSolve` won't solve that formulation. I showed that your posted problem can be taken as a Quadratic Programming problem with a quadratic objective and linear constraints, and you agreed. What now are your grounds for thinking that it can also be reformulated as an equivalent LP problem!?

You also wrote, "Unfortunately, I already know that the QP and the NLP optimization produces the same allocations..." which makes it sound as if you are unhappy with the computed solution, and are hoping for a better one. But this (according to evidence from you) is a convex QP problem and the objective is bounded from below in the feasibility region, so the computed solution should be globally optimal. How then can you hope for a better solution!?

The only way that I can currently envision getting a more desirable solution is if you have additional qualifying restrictions or constraints in mind (but as yet not stated in this thread). For example, perhaps you want an integer-valued solution (well, binary valued, for these ranges), which is something that I just make up as an example. Do you have additional constraints?

So, you want to construct the Matrix form for the NLP formulation of this QP class problem. Why is that? Is it so that you can form procedures for objective, constraints, objective gradient, and constraint jacobian which all accept purely numeric vector arguments, compute results and fill some arguments with the results inplace, and that is evalhf'able or ideally even compilable? And you want them highly optimized and tuned to the quadratic & linear natures of the relevant formulas? It can be done (though keeping down overhead from callbacks through main Maple to get from external Optimization wrappers to Compiled procs is awkward).

But you know who's already done that, without any bypass overhead? The Numerical Algorithms Group. They wrote all the specialized QP objective, linear constraint evaluation, and gradient/jacobian/hessian stuff and compiled it, together. And Maple's Optimization:-QPSolve passes the float[8] data of the QP Matrix form to NAG's e04nfa via Optimization:-External:-E04NFA.

Are your problems so large and sparse that e04nfa is not good enough? How much specialized optimization to the procs and code do you think you can muster, to make up for what appears to be a speed difference factor of about 7 between QPSolve and NLPSolve used plainly.

Look, I don't want to be rude, but it's a little frustrating that you still seem so very disinclined to believe my advice. You wanted to do QP, so I set up the Matrix form and explained it. You wanted to do LP, so I set up the Matrix form and explained it. Why not believe what I suggest about this route into NLP?

I'm also still confused about how much it seems to matter to you whether a 7 second (nstock=200) code section is 100% doubleplus-interior-point-semidefinite-cone-fantastic when the preamble before it is running 80 seconds instead of 30 or so. But then, you have to date left unanswered queries as to how large your typical problems will be.

@acer You seem to misunderstand my intentions

alex_01 — Wed, 11 Jan 2012 14:26:45 Z

@acer You seem to misunderstand my intentions a bit. All I want is to be able to express all
my numerous different problems in all different kinds of forms ie algebraic or matrix form.
Now the notation for matrix form is more complex hence it is more difficult to set up such
a problems not to mention the notation for NLP[mtrix form] which is even more complex.

That was the reasons why I liked the Optimization:-Convert:-AlgebraicForm:-
LPToMatrix(problem) solution because it gave you something to hold one to.
Instead of just trying stuff randomly and gettting error messages slaped in your face
when you are trying to convert you problem to matrix form you could actually see what
the final result of your specific problem should look like in matrix form.
Which made things a little bit easier.

However, I recently discovered that such a solution is not necessarily the "holy grail" because
even though it gives you the final results it does not explain how it got there. For simple
problems that might be easy to figure out but for complex problems that is a whole different
ball game. As you pointed out you cant just do the Optimization:-Convert:-AlgebraicForm:-
LPToMatrix(problem) and hope for a speed up but you need to rewrite the output in float
matricies, hence you need to understand how those matricies where generated in
order to be able to set up more general problems like if you change the number of data
columns ie the number of stocks or the number of observations etc etc

Its not like I dont believe your advice. I do believe that for this particular problem
QPSolve[Matrix Form] might be the fastest and the most appropriate to use however that
does not mean that in the future I will not be faced with the task of converting another NLP to
matrix form. Hence, it would be nice at this stage to have such knowledge so the conversion
can be as painless as possible. This specific problem represent just one of many
different problems I have in my computer.

Also I am glad that you pointed out that Min(Norm(R.W)) is not linear hence LP cant be used!
Then I dont have to bang my head against the wall trying to figure it out (I always have the
insecure feeling that LP is better than QP since some people use LP to solve problems with 10's
of thousands of variables and constrainst (not in maple though) but this might be wrong. I also
have the insecure feeling that Cone programming is the most efficient of them all but this might
also be wrong)

For example if you look in the begining of Chapter 10 in this referens (Boyd etc, which
apperently is an expert in optimization )

http://www.ee.ucla.edu/~vandenbe/103/reader.pdf

They state that the solution to the least-norm problem ie norm(x) Ax=b is unique and given by
x = A' (AA') ^(-1) .b which means (I quote) "that we can solve the least-norm problem by solving
(AA').z=b (10.2) and then calculating ˆx = A'.z. The equations (10.2) are a set of m linear equations
in m variables, and are called the normal equations associated with the least-norm problem.

My specific problem is a little bit different (even though they are both min norm problems) hence it
was difficult to determind if my problem was linear or not.