After substitution of (10) into (4), how to collect the terms of like powers of eta (i.e., eta^-3, eta^-2,eta^-1, eta^0, eta^1,eta^2 ), and equate the coefficients to
zero, get a system of algebraic equations for A[m]?

PA.mw 

How to find values of w for which determinant of A is zero?

determinant.mw

Can we convert expression into determinant of 3 rows and 3 columns?

convert.mw

I am trying to solce eq (2) by integration. But maple integrate only 1st term in eq. Why not other two terms? 

Solve_integral.mw

Why maple give empty solutions? See ref. (12) in the attached wokrsheet. Is there anything missing when deriving Sols?

sols_tw.mw