Learning Vector Quantization

# Learning Vector Quantization

Learning Vector Quantization LVQ, different from Vector quantization VQ and Kohonen Self-Organizing Maps KSOM

, basically is a competitive network which uses supervised learning. We may define it as a process of classifying the patterns where each output unit represents a class. As it uses supervised learning, the network will be given a set of training patterns with known classification along with an initial distribution of the output class. After completing the training process, LVQ will classify an input vector by assigning it to the same class as that of the output unit.

## Architecture

Following figure shows the architecture of LVQ which is quite similar to the architecture of KSOM. As we can see, there are “n” number of input units and “m” number of output units. The layers are fully interconnected with having weights on them.

## Parameters Used

Following are the parameters used in LVQ training process as well as in the flowchart

• x = training vector (x1,…,xi,…,xn)
• T = class for training vector x
• wj = weight vector for jth output unit
• Cj = class associated with the jth output unit

## Training Algorithm

Step 1 − Initialize reference vectors, which can be done as follows −

• Step 1a

− From the given set of training vectors, take the first “mnumberofclusters

• training vectors and use them as weight vectors. The remaining vectors can be used for training.
• Step 1b
• − Assign the initial weight and classification randomly.
• Step 1c
• − Apply K-means clustering method.

Step 2 − Initialize reference vector α

Step 3 − Continue with steps 4-9, if the condition for stopping this algorithm is not met.

Step 4 − Follow steps 5-6 for every training input vector x.

Step 5 − Calculate Square of Euclidean Distance for j = 1 to m and i = 1 to n

D(j)=i=1nj=1m(xiwij)2

Step 6 − Obtain the winning unit J where Dj

is minimum.

Step 7 − Calculate the new weight of the winning unit by the following relation −

if T = Cj then wj(new)=wj(old)+α[xwj(old)]

if T ≠ Cj then wj(new)=wj(old)α[xwj(old)]

Step 8 − Reduce the learning rate α

.

Step 9 − Test for the stopping condition. It may be as follows −

• Maximum number of epochs reached.
• Learning rate reduced to a negligible value.

## Variants

Three other variants namely LVQ2, LVQ2.1 and LVQ3 have been developed by Kohonen. Complexity in all these three variants, due to the concept that the winner as well as the runner-up unit will learn, is more than in LVQ.

## LVQ2

As discussed, the concept of other variants of LVQ above, the condition of LVQ2 is formed by window. This window will be based on the following parameters −

• x − the current input vector
• yc − the reference vector closest to x
• yr − the other reference vector, which is next closest to x
• dc − the distance from x to yc
• dr − the distance from x to yr

The input vector x falls in the window, if

dcdr>1θanddrdc>1+θ

Here, θ

is the number of training samples.

Updating can be done with the following formula −

yc(t+1)=yc(t)+α(t)[x(t)yc(t)]

belongstodifferentclass

yr(t+1)=yr(t)+α(t)[x(t)yr(t)]

belongstosameclass

Here α

is the learning rate.

## LVQ2.1

In LVQ2.1, we will take the two closest vectors namely yc1 and yc2 and the condition for window is as follows −

Min[dc1dc2,dc2dc1]>(1θ)

Max[dc1dc2,dc2dc1]<(1+θ)

Updating can be done with the following formula −

yc1(t+1)=yc1(t)+α(t)[x(t)yc1(t)]

belongstodifferentclass

yc2(t+1)=yc2(t)+α(t)[x(t)yc2(t)]

belongstosameclass

Here, α

is the learning rate.

## LVQ3

In LVQ3, we will take the two closest vectors namely yc1 and yc2 and the condition for window is as follows −

Min[dc1dc2,dc2dc1]>(1θ)(1+θ)

Here θ0.2

Updating can be done with the following formula −

yc1(t+1)=yc1(t)+β(t)[x(t)yc1(t)]

belongstodifferentclass

yc2(t+1)=yc2(t)+β(t)[x(t)yc2(t)]

belongstosameclass

Here β

is the multiple of the learning rate α and β=mα(t) for every 0.1 < m < 0.5