MaplePrimes - answers and comments on Question, Can I filter a signal with Maple 14

filter (pass through)

pagan — Tue, 21 Sep 2010 05:17:14 Z

It wasn't quite clear to me whether you wanted to filter out a range of frequencies (possibly all above some value) or whether you wanted to allow only a range or band of frequences to get passed through. If you want to remove all frequences above a given value then either of those apporaches should suffice.

The examples below attempt to show a band getting passed through.

restart:

freqpass:=proc(M::Vector,rng::range(nonnegint))
local Mt, Xlow, Xhi;
  Xlow,Xhi := op(rng);
  Mt := DiscreteTransforms:-FourierTransform(M):
    if Xlow>1 then ArrayTools:-Fill(Xlow-1,0.0,Mt,0,1); end if;
    if Xhi<op(1,Mt) then ArrayTools:-Fill(op(1,Mt)-Xhi-1,0.0,Mt,Xhi+1,1); end if;
    Mt[1]:=Mt[1]/2;
    2*map(Re,DiscreteTransforms:-InverseFourierTransform(Mt));
end proc:

>	Make a sample signal M

>	f := x -> ((x-1/3)^5) + 50sin(68x):

>	a,b,n := -Pi,Pi,1000: M := Vector(n,(i)->(evalhf@f)(a+i*(b-a)/n),datatype=float[8]):

>	plots:-pointplot([seq([k,M[k]],k=1..op(1,M))],style=line, view=[0..1000,-600..300]);

>	filtered := freqpass(M,1..67):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line, view=[0..1000,-600..300]);

>	filtered := freqpass(M,68..71):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line, view=[0..1000,-600..300]);

>	filtered := freqpass(M,72..100):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line, view=[0..1000,-600..300]);

>	f := x -> sin(5x) + sin(118x) - sin(130x) + sin(123x) - sin(127x) - 1/3sin(113x) + 1/5sin(300*x):

>	a,b,n := -Pi,Pi,1000: M := Vector(n,(i)->(evalhf@f)(a+i*(b-a)/n),datatype=float[8]):

>	plots:-pointplot([seq([k,M[k]],k=1..op(1,M))],style=line,view=[0..1000,-7..7]);

>	filtered := freqpass(M,1..120):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line,view=[0..1000,-7..7]);

>	filtered := freqpass(M,1..100):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line,view=[0..1000,-7..7]);

Download fftfun00.mw

All this is doing is zeroing the result of the FFT except for the range you specify, and then doing the inverse-FFT. With a few tricks to try and stay efficient. Remember, the result of the FFT is (real part) frequency and (imaginary part) shift.

I'm not doing anything to allow you to express things in terms of time, hertz, sample duration. That's just bookkeeping and nomenclature. Bells and whistles. But let me know if this aspect has you puzzled. These samples below are just a number of data points, so how you interpret them according to some sampling speed, etc, is up to you. With this below, there are simply a number of discrete frequencies (depending on the number of data points and whatever you say the elapsed duration is) and you can pass them through or filter them out.

Notice the magnitude of the (supposedly non-noisy) lower frequency filtered result in the latter example. In this case the higher frequencies (now filtered out) contained a measurable portion of the signal strength.

If you do want to use such terminology then you have to say what you want done in the case that either frequency bound you specify doesn't match the granularity of your data. For given data and a given duration in units of time there will be a minimal attainable frequency determined by the number of nodes of data. If you specify a frequency cut-off that doesn't fall on a boundary then you have to agree how to handle it: eg. apportion it between the pair of neighbouring attainable discrete frequencies, or apportion it all to the single nearest attainable frequency.

window function correction, with padding

acer — Thu, 23 Sep 2010 05:16:59 Z

@brian abraham One of the common goals of using a Window Function is to correct the oscillation phenomenon near the end-points. Since there is (by defn) no original data beyond what is supplied then the FFT will usually not be able to accomodate such a discontinuity. The FFT process relies on "knowing" that the data extends in some (overlay of) a periodic way. A hard nonzero boundary for the data thus induces an effect similar to the Gibbs phenonmenon.

A key purpose of using a Window Function to scale the data so that it tends strongly to zero near the original data point boundary is to allow the data to be be safely padded further on with zero values. Since the scaled data is converging to zero, then padding with zero (or near zero) values will make the FFT process see a natural continuation. This makes the FFT and Inverse FFT behave much better at the "old" boundary, since they now act on a wider data set. And it's not just that the "old" boundaries are well inside the extended data set. The fact that the data is all nice is also key. The augmented data can easily be made to appear to extend the old curve that is tending to zero near the old boundary. The augmented data doesn't even have to oscillate in order to effectively do its job. There is no significant distinction from the augmented data and the (now tiny) oscillation in the scaled old data near the old boundary. The augmented data is effectively periodic (though literally it is not) because it so close to zero and its would-be oscillation is miniscule.

I will try to demonstrate using the previously posted example. I've added yet another, third, data set. It is based on the Windowed data set, but padded by ten percent more points. It's not padded with zero. It's padded with a continuation of the scaled curve. I've added two more plots, to show close-up where the padded data meets the scaled original data.

restart:

>	freqpass:=proc(M::Vector,rng::range(nonnegint))

>	local Mt, Xlow, Xhi;

>	Xlow,Xhi := op(rng);

>	Mt := DiscreteTransforms:-FourierTransform(M):

>	if Xlow>1 then ArrayTools:-Fill(Xlow-1,0.0,Mt,0,1); end if;

>	if Xhi<op(1,Mt) then ArrayTools:-Fill(op(1,Mt)-Xhi-1,0.0,Mt,Xhi+1,1); end if;

>	Mt[1]:=Mt[1]/2;

>	2*map(Re,DiscreteTransforms:-InverseFourierTransform(Mt));

end proc:

>	f := x -> 5x + sin(128x) - sin(130x) + sin(133x) - sin(137*x);

(1)

>	a,b,n := -Pi,Pi,1000:

>	M := Vector(n,(i)->(evalhf@f)(a+i*(b-a)/n),datatype=float[8]):

Mg := Vector(n,(i)->evalhf(f(a+i*(b-a)/n)*exp(-1/2*(a+i*(b-a)/n)^2)),datatype=float[8]):
numpad := trunc(n*0.10):
Mgpadded := Vector(n+2*numpad,datatype=float[8]):
Mgpadded[numpad+1..numpad+1+n]:=Mg:
Mgpadded[1..numpad] := Vector(numpad,(i)->M[1]
*evalhf(exp(-1/2*(a+(-numpad+i)*(b-a)/n)^2)),datatype=float[8]):
Mgpadded[n+numpad+1..n+2*numpad] := Vector(numpad,(i)->M[n]
*evalhf(exp(-1/2*(a+(n+i)*(b-a)/n)^2)),datatype=float[8]):

>	plots:-pointplot([seq([k,M[k]],k=1..op(1,M))],style=line,view=[0..1000,-20..20]);

>	plots:-pointplot([seq([k,Mg[k]],k=1..op(1,Mg))],style=line); plots:-pointplot([seq([k,Mgpadded[k]],k=1..op(1,Mgpadded))],style=line); plots:-pointplot([seq([k,Mgpadded[k]],k=1..numpad+100)],style=line); plots:-pointplot([seq([k,Mgpadded[k]],k=n+numpad+1-100..n+2*numpad)],style=line);

>	filtered := freqpass(M,1..30):

>	filteredg := freqpass(Mg,1..30): filteredgpadded := freqpass(Mgpadded,1..30)[numpad+1..numpad+1+n]:

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line);

>	plots:-pointplot([seq([k,exp(1/2(a+k(b-a)/n)^2)filteredg[k]], k=1..op(1,filteredg))],style=line); plots:-pointplot([seq([k,exp(1/2(a+k(b-a)/n)^2)filteredgpadded[k]], k=1..op(1,filteredgpadded))],style=line);

Download window_correction.mw

start and end of filtered signal deviates from unfiltered signal: is this related to the unit impulse

brian abraham — Tue, 21 Sep 2010 15:15:48 Z

Thank you very much.

I can understand why ‘zeroing the result of the FFT’ produces the result: I think that is the first time in signal-processing that the how-to seems so straight forward.

I have your code up and running with my data (collected at 50 Hz) and it works with range 0..5 and so on.

If I might take advantage of your expertise with a question:

The lower I set the second frequency (high pass?) the more the filtered signal deviates from the unfiltered signal at the signal start and end.

Am I correct in thinking that this is related to the fact that the signal I pass to the DFT is actually the combination of a unit impulse for the duration of the signal and the signal itself?

If I am correct why does the filtered signal start and end deviation from the unfiltered signal decrease as higher frequencies are introduced?

Many thanks once again.

good questions

pagan — Tue, 21 Sep 2010 18:29:30 Z

@brian abraham These are good questions.

I suggest this link as reading material. Since I take all the data as given (ie, not a stream of data) then it can be considered as one big window in that context.

Given the earlier code, compare the following results. I've added a multiplicative term to the signal, so that it approaches zero at the end points. And then its multiplicative inverse is used to scale the result back again, following the inverse FF Transformation. Hopefully you can see some corrective effect this attempt has. (There is still some end-point effect, but remember that some amount of the actual signal is lost due to completely zeroing the high frequencies alongside spectral leakage. The topic of filtering noise while retaining signal gets advanced, as the link should explain. Also, I use a weighting only sketchily similar to that of a Gaussian window -- and I didn't give it a lot of thought. For your signal may wish to experiment with some other Window Function, eg. exp(-abs(a+i*(b-a)/n)) or something better entirely.)

restart:

>	freqpass:=proc(M::Vector,rng::range(nonnegint))

>	local Mt, Xlow, Xhi;

>	Xlow,Xhi := op(rng);

>	Mt := DiscreteTransforms:-FourierTransform(M):

>	if Xlow>1 then ArrayTools:-Fill(Xlow-1,0.0,Mt,0,1); end if;

>	if Xhi<op(1,Mt) then ArrayTools:-Fill(op(1,Mt)-Xhi-1,0.0,Mt,Xhi+1,1); end if;

>	Mt[1]:=Mt[1]/2;

>	2*map(Re,DiscreteTransforms:-InverseFourierTransform(Mt));

end proc:

>	f := x -> 5x + sin(128x) - sin(130x) + sin(133x) - sin(137*x);

(1)

>	a,b,n := -Pi,Pi,1000:

>	M := Vector(n,(i)->(evalhf@f)(a+i*(b-a)/n),datatype=float[8]):

>	Mg := Vector(n,(i)->evalhf(f(a+i(b-a)/n)exp(-1/2(a+i(b-a)/n)^2)),datatype=float[8]):

>	plots:-pointplot([seq([k,M[k]],k=1..op(1,M))],style=line,view=[0..1000,-20..20]);

>	plots:-pointplot([seq([k,Mg[k]],k=1..op(1,Mg))],style=line);

>	filtered := freqpass(M,1..30):

>	filteredg := freqpass(Mg,1..30):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line);

>	plots:-pointplot([seq([k,exp(1/2(a+k(b-a)/n)^2)*filteredg[k]],k=1..op(1,filteredg))],

>	style=line);

>	filtered := freqpass(M,1..90):

>	filteredg := freqpass(Mg,1..90):

>	plots:-pointplot([seq([k,filtered[k]],k=1..op(1,filtered))],style=line);

>	plots:-pointplot([seq([k,exp(1/2(a+k(b-a)/n)^2)*filteredg[k]],k=1..op(1,filteredg))],

>	style=line);

Download fftfun03.mw

[I made a revision to this comment. It had an earlier, inferior code variant.]

Can a windowing worsen spectral leakage

brian abraham — Wed, 22 Sep 2010 13:43:17 Z

Thank you, I have read and found the link helpful.

I have plotted the data multiplied by the window function you provided and it progresses from zero and back to zero as expected.

However, when I pass the data multiplied by the window function through the filter the results at the start and end data show much greater deviation from the signal than occurs without using the window function. Is this possible or have I made an error in implementing your code?

On the other hand when I choose my window start and end so that the signal has a similar magnitude at the start and end, the filtered signal follows the unfiltered signal much more closely as the article suggest.

various

pagan — Wed, 22 Sep 2010 15:57:59 Z

There are various kinds of window function, and yes they can have a variety of effects. (Pls see a few comments above on your posible need to choose.)