FAM: Fuzzy Associative Memory

Introduction

In this chapter, you we will learn about fuzzy sets and their elements, both for input and output of an associative neural network. Every element of a fuzzy set has a degree of membership in the set. Unless this degree of membership is 1, an element does not belong to the set (in the sense of elements of an ordinary set belonging to the set). In a neural network of a fuzzy system the inputs, the outputs, and the connection weights all belong fuzzily to the spaces that define them. The weight matrix will be a fuzzy matrix, and the activations of the neurons in such a network have to be determined by rules of fuzzy logic and fuzzy set operations.

An expert system uses what are called crisp rules and applies them sequentially. The advantage in casting the same problem in a fuzzy system is that the rules you work with do not have to be crisp, and the processing is done in parallel. What the fuzzy systems can determine is a fuzzy association. These associations can be modified, and the underlying phenomena better understood, as experience is gained. That is one of the reasons for their growing popularity in applications. When we try to relate two things through a process of trial and error, we will be implicitly and intuitively establishing an association that is gradually modified and perhaps bettered in some sense. Several fuzzy variables may be present in such an exercise. That we did not have full knowledge at the beginning is not a hindrance; there is some difference in using probabilities and using fuzzy logic as well. The degree of membership assigned for an element of a set does not have to be as firm as the assignment of a probability.

The degree of membership is, like a probability, a real number between 0 and 1. The closer it is to 1, the less ambiguous is the membership of the element in the set concerned. Suppose you have a set that may or may not contain three elements, say, a, b, and c. Then the fuzzy set representation of it would be by the ordered triple (ma, mb, mc), which is called the fit vector, and its components are called fit values. For example, the triple (0.5, 0.5, 0.5) shows that each of a, b, and c, have a membership equal to only one-half. This triple itself will describe the fuzzy set. It can also be thought of as a point in the three-dimensional space. None of such points will be outside the unit cube. When the number of elements is higher, the corresponding points will be on or inside the unit hypercube.

It is interesting to note that this fuzzy set, given by the triple (0.5, 0.5, 0.5), is its own complement, something that does not happen with regular sets. The complement is the set that shows the degrees of nonmembership.

The height of a fuzzy set is the maximum of its fit values, and the fuzzy set is said to be normal, if its height is 1. The fuzzy set with fit vector (0.3, 0.7, 0.4) has height 0.7, and it is not a normal fuzzy set. However, by introducing an additional dummy component with fit value 1, we can extend it into a normal fuzzy set. The desirability of normalcy of a fuzzy set will become apparent when we talk about recall in fuzzy associative memories.

The subset relationship is also different for fuzzy sets from the way it is defined for regular sets. For example, if you have a fuzzy set given by the triple (0.3, 0.7, 0.4), then any fuzzy set with a triple (a, b, c) such that a ≤ 0.3, b ≤ 0.7, and c ≤ 0.4, is its fuzzy subset. For example, the fuzzy set given by the triple (0.1, 0.7, 0) is a subset of the fuzzy set (0.3, 0.7, 0.4).

Association

Consider two fuzzy sets, one perhaps referring to the popularity of (or interest in) an exhibition and the other to the price of admission. The popularity could be very high, high, fair, low, or very low. Accordingly, the fit vector will have five components. The price of admission could be high, modest, or low. Its fit vector has three components. A fuzzy associative memory system then has the association (popularity, price), and the fuzzy set pair encodes this association.

We describe the encoding process and the recall in the following sections. Once encoding is completed, associations between subsets of the first fuzzy set and subsets of the second fuzzy set are also established by the same fuzzy associative memory system.

FAM Neural Network

The neural network for a fuzzy associative memory has an input and an output layer, with full connections in both directions, just as in a BAM neural network. Figure Fuzzy associative memory neural network shows the layout. To avoid cluttering, the figure shows the network with three neurons in field A and two neurons in field B, and only some of the connection weights. The general case is analogous.

Fuzzy associative memory neural network.

Encoding

Encoding for fuzzy associative memory systems is similar in concept to the encoding process for bidirectional associative memories, with some differences in the operations involved. In the BAM encoding, bipolar versions of binary vectors are used just for encoding. Matrices of the type XiT Yi are added together to get the connection weight matrix. There are two basic operations with the elements of these vectors. They are multiplication and addition of products. There is no conversion of the fit values before the encoding by the fuzzy sets. The multiplication of elements is replaced by the operation of taking the minimum, and addition is replaced by the operation of taking the maximum.

There are two methods for encoding. The method just described is what is called max–min composition. It is used to get the connection weight matrix and also to get the outputs of neurons in the fuzzy associative memory neural network. The second method is called correlation–product encoding. It is obtained the same way as a BAM connection weight matrix is obtained. Max–min composition is what is frequently used in practice, and we will confine our attention to this method.

Example of Encoding

Suppose the two fuzzy sets we use to encode have the fit vectors ( 0.3, 0.7, 0.4, 0.2) and (0.4, 0.3, 0.9). Then the matrix W is obtained by using max–min composition as follows.

   0.3 [0.4 0.3 0.9] min(0.3,0.4) min(0.3,0.3) min(0.3,0.9) 0.3 0.3 0.3
 W=0.7              =min(0.7,0.4) min(0.7,0.3) min(0.7,0.9)=0.4 0.3 0.7
   0.4               min(0.4,0.4) min(0.4,0.3) min(0.4,0.9) 0.4 0.3 0.4
   0.2               min(0.2,0.4) min(0.2,0.3) min(0.2,0.9) 0.2 0.2 0.2

Recall for the Example

If we input the fit vector (0.3, 0.7, 0.4, 0.2), the output (b1, b2, b3) is determined as follows, using bj = max( min(a1, w1j), …, min(am, wmj), where m is the dimension of the ‘a’ fit vector, and wij is the ith row, jth column element of the matrix W.

     b1 = max(min(0.3, 0.3), min(0.7, 0.4), min(0.4, 0.4),
          min(0.2, 0.2)) =  max(0.3, 0.4, 0.4, 0.2) = 0.4
     b2 = max(min(0.3, 0.3), min(0.7, 0.3), min(0.4, 0.3),
          min(0.2, 0.2)) = max( 0.3, 0.3, 0.3, 0.2 ) = 0.3
     b3 = max(min(0.3, 0.3), min(0.7, 0.7), min(0.4, 0.4),
          min(0.2, 0.2)) = max (0.3, 0.7, 0.4, 0.2) = 0.7

The output vector (0.4, 0.3, 0.7) is not the same as the second fit vector used, namely (0.4, 0.3, 0.9), but it is a subset of it, so the recall is not perfect. If you input the vector (0.4, 0.3, 0.7) in the opposite direction, using the transpose of the matrix W, the output is (0.3, 0.7, 0.4, 0.2), showing resonance. If on the other hand you input (0.4, 0.3, 0.9) at that end, the output vector is (0.3, 0.7, 0.4, 0.2), which in turn causes in the other direction an output of (0.4, 0.3, 0.7) at which time there is resonance. Can we foresee these results? The following section explains this further.

Recall

Let us use the operator o to denote max–min composition. Perfect recall occurs when the weight matrix is obtained using the max–min composition of fit vectors U and V as follows:

(i)  U o W = V if and only if height (U) ≥ height (V).

(ii)  V o WT = U if and only if height (V) ≥ height (U).

Also note that if X and Y are arbitrary fit vectors with the same dimensions as U and V, then:

(iii)  X o WV.

(iv)  Y o WTU.


A ∴ B is the notation to say A is a subset of B.


In the previous example, height of (0.3, 0.7, 0.4, 0.2) is 0.7, and height of (0.4, 0.3, 0.9) is 0.9. Therefore (0.4, 0.3, 0.9) as input, produced (0.3, 0.7, 0.4, 0.2) as output, but (0.3, 0.7, 0.4, 0.2) as input, produced only a subset of (0.4, 0.3, 0.9). That both (0.4, 0.3, 0.7) and (0.4, 0.3, 0.9) gave the same output, (0.3, 0.7, 0.4, 0.2) is in accordance with the corollary to the above, which states that if (X, Y) is a fuzzy associated memory, and if X is a subset of X’, then (X’, Y) is also a fuzzy associated memory.

C++ Implementation

We use the classes we created for BAM implementation in C++, except that we call the neuron class fzneuron, and we do not need some of the methods or functions in the network class. The header file, the source file, and the output from an illustrative run of the program are given in the following. The header file is called fuzzyam.hpp, and the source file is called fuzzyam.cpp.

Program details

The program details are analogous to the program details given in Chapter 8. The computations are done with fuzzy logic. Unlike in the nonfuzzy version, a single exemplar fuzzy vector pair is used here. There are no transformations to bipolar versions, since the vectors are fuzzy and not binary and crisp.

A neuron in the first layer is referred to as anrn, and the number of neurons in this layer is referred to as anmbr. bnrn is the name we give to the array of neurons in the second layer and bnmbr denotes the size of that array. The sequence of operations in the program are as follows:

  We ask the user to input the exemplar vector pair.

  We give the network the X vector, in the exemplar pair. We find the activations of the elements of bnrn array and get corresponding output vector as a binary pattern. If this is the Y in the exemplar pair, the network has made a desired association in one direction, and we go on to the next.step. Otherwise we have a potential associated pair, one of which is X and the other is what we just got as the output vector in the opposite layer. We say potential associated pair because we have the next step to confirm the association.

  We run the bnrn array through the transpose of the weight matrix and calculate the outputs of the anrn array elements. If, as a result, we get the vector X as the anrn array, we found an associated pair, (X, Y). Otherwise, we repeat the two steps just described until we find an associated pair.

  We now work with the next pair of exemplar vectors in the same manner as above, to find an associated pair.

  We assign serial numbers, denoted by the variable idn, to the associated pairs so we can print them all together at the end of the program. The pair is called (X, Y) where X produces Y through the weight matrix W, and Y produces X through the weight matrix, which is the transpose of W.

  A flag is used to have value 0 until confirmation of association is obtained, when the value of the flag changes to 1.

  Functions compr1 and compr2 in the network class verify if the potential pair is indeed an associated pair and set the proper value of the flag mentioned above.

  Functions comput1 and comput2 in the network class carry out the calculations to get the activations and then find the output vector, in the respective directions of the fuzzy associative memory network.

A lot of the code from the bidirectional associative memory (BAM) is used for the FAM. Here are the listings, with comments added where there are differences between this code and the code for the BAM of Chapter 8.

Example of Encoding

Suppose the two fuzzy sets we use to encode have the fit vectors ( 0.3, 0.7, 0.4, 0.2) and (0.4, 0.3, 0.9). Then the matrix W is obtained by using max–min composition as follows.

   0.3 [0.4 0.3 0.9] min(0.3,0.4) min(0.3,0.3) min(0.3,0.9) 0.3 0.3 0.3
 W=0.7              =min(0.7,0.4) min(0.7,0.3) min(0.7,0.9)=0.4 0.3 0.7
   0.4               min(0.4,0.4) min(0.4,0.3) min(0.4,0.9) 0.4 0.3 0.4
   0.2               min(0.2,0.4) min(0.2,0.3) min(0.2,0.9) 0.2 0.2 0.2

Recall for the Example

If we input the fit vector (0.3, 0.7, 0.4, 0.2), the output (b1, b2, b3) is determined as follows, using bj = max( min(a1, w1j), …, min(am, wmj), where m is the dimension of the ‘a’ fit vector, and wij is the ith row, jth column element of the matrix W.

     b1 = max(min(0.3, 0.3), min(0.7, 0.4), min(0.4, 0.4),
          min(0.2, 0.2)) =  max(0.3, 0.4, 0.4, 0.2) = 0.4
     b2 = max(min(0.3, 0.3), min(0.7, 0.3), min(0.4, 0.3),
          min(0.2, 0.2)) = max( 0.3, 0.3, 0.3, 0.2 ) = 0.3
     b3 = max(min(0.3, 0.3), min(0.7, 0.7), min(0.4, 0.4),
          min(0.2, 0.2)) = max (0.3, 0.7, 0.4, 0.2) = 0.7

The output vector (0.4, 0.3, 0.7) is not the same as the second fit vector used, namely (0.4, 0.3, 0.9), but it is a subset of it, so the recall is not perfect. If you input the vector (0.4, 0.3, 0.7) in the opposite direction, using the transpose of the matrix W, the output is (0.3, 0.7, 0.4, 0.2), showing resonance. If on the other hand you input (0.4, 0.3, 0.9) at that end, the output vector is (0.3, 0.7, 0.4, 0.2), which in turn causes in the other direction an output of (0.4, 0.3, 0.7) at which time there is resonance. Can we foresee these results? The following section explains this further.

Recall

Let us use the operator o to denote max–min composition. Perfect recall occurs when the weight matrix is obtained using the max–min composition of fit vectors U and V as follows:

(i)  U o W = V if and only if height (U) ≥ height (V).

(ii)  V o WT = U if and only if height (V) ≥ height (U).

Also note that if X and Y are arbitrary fit vectors with the same dimensions as U and V, then:

(iii)  X o WV.

(iv)  Y o WTU.


A ∴ B is the notation to say A is a subset of B.


In the previous example, height of (0.3, 0.7, 0.4, 0.2) is 0.7, and height of (0.4, 0.3, 0.9) is 0.9. Therefore (0.4, 0.3, 0.9) as input, produced (0.3, 0.7, 0.4, 0.2) as output, but (0.3, 0.7, 0.4, 0.2) as input, produced only a subset of (0.4, 0.3, 0.9). That both (0.4, 0.3, 0.7) and (0.4, 0.3, 0.9) gave the same output, (0.3, 0.7, 0.4, 0.2) is in accordance with the corollary to the above, which states that if (X, Y) is a fuzzy associated memory, and if X is a subset of X’, then (X’, Y) is also a fuzzy associated memory.

C++ Implementation

We use the classes we created for BAM implementation in C++, except that we call the neuron class fzneuron, and we do not need some of the methods or functions in the network class. The header file, the source file, and the output from an illustrative run of the program are given in the following. The header file is called fuzzyam.hpp, and the source file is called fuzzyam.cpp.

Program details

The program details are analogous to the program details given in Chapter 8. The computations are done with fuzzy logic. Unlike in the nonfuzzy version, a single exemplar fuzzy vector pair is used here. There are no transformations to bipolar versions, since the vectors are fuzzy and not binary and crisp.

A neuron in the first layer is referred to as anrn, and the number of neurons in this layer is referred to as anmbr. bnrn is the name we give to the array of neurons in the second layer and bnmbr denotes the size of that array. The sequence of operations in the program are as follows:

  We ask the user to input the exemplar vector pair.

  We give the network the X vector, in the exemplar pair. We find the activations of the elements of bnrn array and get corresponding output vector as a binary pattern. If this is the Y in the exemplar pair, the network has made a desired association in one direction, and we go on to the next.step. Otherwise we have a potential associated pair, one of which is X and the other is what we just got as the output vector in the opposite layer. We say potential associated pair because we have the next step to confirm the association.

  We run the bnrn array through the transpose of the weight matrix and calculate the outputs of the anrn array elements. If, as a result, we get the vector X as the anrn array, we found an associated pair, (X, Y). Otherwise, we repeat the two steps just described until we find an associated pair.

  We now work with the next pair of exemplar vectors in the same manner as above, to find an associated pair.

  We assign serial numbers, denoted by the variable idn, to the associated pairs so we can print them all together at the end of the program. The pair is called (X, Y) where X produces Y through the weight matrix W, and Y produces X through the weight matrix, which is the transpose of W.

  A flag is used to have value 0 until confirmation of association is obtained, when the value of the flag changes to 1.

  Functions compr1 and compr2 in the network class verify if the potential pair is indeed an associated pair and set the proper value of the flag mentioned above.

  Functions comput1 and comput2 in the network class carry out the calculations to get the activations and then find the output vector, in the respective directions of the fuzzy associative memory network.

A lot of the code from the bidirectional associative memory (BAM) is used for the FAM. Here are the listings, with comments added where there are differences between this code and the code for the BAM of Chapter BAM: Bidirectional Associative Memory.

Output

The illustrative run of the previous program uses the fuzzy sets with fit vectors (0.1, 0.3, 0.2, 0.0, 0.7, 0.5) and (0.4, 0.2, 0.1, 0.0). As you can expect according to the discussion earlier, recall is not perfect in the reverse direction and the fuzzy associated memory consists of the pairs (0.1, 0.3, 0.2, 0.0, 0.4, 0.4) with (0.4, 0.2, 0.1, 0.0) and (0.1, 0.2, 0.2, 0, 0.2, 0.2) with (0.2, 0.2, 0.1, 0). The computer output is in such detail as to be self-explanatory.

THIS PROGRAM IS FOR A FUZZY ASSOCIATIVE MEMORY NETWORK. THE NETWORK IS
SET UP FOR ILLUSTRATION WITH SIX INPUT NEURONS, AND FOUR OUTPUT NEURONS.
1 exemplars are used to encode
 
X vector you gave is: 0.1  0.3  0.2  0  0.7  0.5
Y vector you gave is: 0.4  0.2  0.1  0
 
  weights--input layer to output layer:
 
0.1  0.1  0.1  0
0.3  0.2  0.1  0
0.2  0.2  0.1  0
0    0    0    0
0.4  0.2  0.1  0
0.4  0.2  0.1  0
 
weights--output layer to input layer:
 
0.1  0.3  0.2  0  0.4  0.4
0.1  0.2  0.2  0  0.2  0.2
0.1  0.1  0.1  0  0.1  0.1
0    0    0    0  0    0
 
Input vector is: 0.1 0.3 0.2 0 0.7 0.5
 
output layer neuron  0 activation is 0.4
output layer neuron  0 output is 0.4
output layer neuron  1 activation is 0.2
output layer neuron  1 output is 0.2
output layer neuron  2 activation is 0.1
output layer neuron  2 output is 0.1
output layer neuron  3 activation is 0
 output layer neuron  3 output is 0
X vector in possible associated pair is: 0.1  0.3  0.2  0  0.7  0.5
Y vector in possible associated pair is: 0.4  0.2  0.1  0
input layer neuron 0 activation is 0.1
input layer neuron  0 output is 0.1
input layer neuron 1 activation is 0.3
input layer neuron  1 output is 0.3
input layer neuron 2 activation is 0.2
input layer neuron  2 output is 0.2
input layer neuron 3 activation is 0
input layer neuron  3 output is 0
input layer neuron 4 activation is 0.4
input layer neuron  4 output is 0.4
input layer neuron 5 activation is 0.4
input layer neuron  5 output is 0.4
output layer neuron  0 activation is 0.4
output layer neuron  0 output is 0.4
output layer neuron  1 activation is 0.2
output layer neuron  1 output is 0.2
output layer neuron  2 activation is 0.1
output layer neuron  2 output is 0.1
output layer neuron  3 activation is 0
output layer neuron  3 output is 0
X vector in possible associated pair is: 0.1  0.3  0.2  0  0.4  0.4
Y vector in possible associated pair is: 0.4  0.2  0.1  0
PATTERNS ASSOCIATED:
X vector in the associated pair no. 1 is: 0.1  0.3  0.2  0  0.4  0.4
Y vector in the associated pair no. 1 is: 0.4  0.2  0.1  0
Input vector is: 0.6 0 0.3 0.4 0.1 0.2
 output layer neuron  0 activation is 0.2
 output layer neuron  0 output is 0.2
 output layer neuron  1 activation is 0.2
 output layer neuron  1 output is 0.2
 output layer neuron  2 activation is 0.1
 output layer neuron  2 output is 0.1
 output layer neuron  3 activation is 0
 output layer neuron  3 output is 0
X vector in possible associated pair is: 0.6  0  0.3  0.4  0.1  0.2
Y vector in possible associated pair is: 0.2  0.2  0.1  0
input layer neuron 0 activation is 0.1
 input layer neuron  0 output is 0.1
input layer neuron 1 activation is 0.2
 input layer neuron  1 output is 0.2
input layer neuron 2 activation is 0.2
 input layer neuron  2 output is 0.2
input layer neuron 3 activation is 0
 input layer neuron  3 output is 0
input layer neuron 4 activation is 0.2
 input layer neuron  4 output is 0.2
input layer neuron 5 activation is 0.2
 input layer neuron  5 output is 0.2
 output layer neuron  0 activation is 0.2
 output layer neuron  0 output is 0.2
 output layer neuron  1 activation is 0.2
 output layer neuron  1 output is 0.2
 output layer neuron  2 activation is 0.1
 output layer neuron  2 output is 0.1
 output layer neuron  3 activation is 0
 output layer neuron  3 output is 0
X vector in possible associated pair is: 0.1  0.2  0.2  0  0.2  0.2
Y vector in possible associated pair is: 0.2  0.2  0.1  0
 output layer neuron  0 activation is 0.2
 output layer neuron  0 output is 0.2
 output layer neuron  1 activation is 0.2
 output layer neuron  1 output is 0.2
 output layer neuron  2 activation is 0.1
 output layer neuron  2 output is 0.1
 output layer neuron  3 activation is 0
 output layer neuron  3 output is 0
 output layer neuron  0 activation is 0.2
 output layer neuron  0 output is 0.2
 output layer neuron  1 activation is 0.2
 output layer neuron  1 output is 0.2
 output layer neuron  2 activation is 0.1
 output layer neuron  2 output is 0.1
 output layer neuron  3 activation is 0
 output layer neuron  3 output is 0
X vector in possible associated pair is: 0.1  0.2  0.2  0  0.2  0.2
Y vector in possible associated pair is: 0.2  0.2  0.1  0
PATTERNS ASSOCIATED:
X vector in the associated pair no. 2 is: 0.1  0.2  0.2  0  0.2  0.2
Y vector in the associated pair no. 2 is: 0.2  0.2  0.1  0
THE FOLLOWING ASSOCIATED PAIRS WERE FOUND BY FUZZY AM
X vector in the associated pair no. 1 is: 0.1  0.3  0.2  0  0.4  0.4
Y vector in the associated pair no. 1 is: 0.4  0.2  0.1  0
X vector in the associated pair no. 2 is: 0.1  0.2  0.2  0  0.2  0.2
Y vector in the associated pair no. 2 is: 0.2  0.2  0.1  0

Summary

In this chapter, bidirectional associative memories are presented for fuzzy subsets. The development of these is largely due to Kosko. They share the feature of resonance between the two layers in the network with Adaptive Resonance theory. Even though there are connections in both directions between neurons in the two layers, only one weight matrix is involved. You use the transpose of this weight matrix for the connections in the opposite direction. When one input at one end leads to some output at the other, which in turn leads to output same as the previous input, resonance is reached and an associated pair is found. In the case of bidirectional fuzzy associative memories, one pair of fuzzy sets determines one fuzzy associative memory system. Fit vectors are used in max–min composition. Perfect recall in both directions is not the case unless the heights of both fit vectors are equal. Fuzzy associative memories can improve the performance of an expert system by allowing fuzzy rules.