Neural Networks Tutorial, Part #1

Posted on November 25, 2007
Filed Under Algorithms, Signal Processing |

This is the first part of a set of postings on neural networks (NNs). NN are constructs that attempt to mimic our brain’s activity to a degree. NNs have become so common that it’s useful to know what people are talking about, so I’ve decided to write a brief tutorial - in several parts - on creating a simple NN. I’ll be providing code in MATLAB, but you should be able to implement what I say in any language.

What is a Neural Network?

The basic building block of a NN is a neuron. Each neuron has some value - a number - attached to it. Some neurons are binary, meaning they can only have a value of 0 or 1. Others can have any real value - those are the neurons we’ll be working with. Each neuron also has several incoming synapses and several outgoing synapses. The neurons are all interconnect, with the outputs of each wired into the inputs of others. When a neuron receives an input it processes the input somehow and “computes” the matching output. The way this computation is set up is completely up to the maker of the net. However, there are several computational models of networks that have proven themselves over the past few decades. We will be studying one such network, the Feed Forward Network.

What NNs Are There?

NNs are roughly divided into two parts: those that learn by themselves and those that are taught. Those that learn by themselves try to find regularity in a given set of data - for example, a NN might be given a large random set of internet sites and attempt to cluster them into categories based on which sites link where. Those that are taught are first trained on several pairs of inputs and outputs and, once the training it over, are supposed to classify new input appropriately - for example, a NN might initially be given the various letters of the alphabet in some person’s handwriting and told that this and that drawing corresponds to this and that letter in the alphabet. It can then be asked to decipher that person’s handwriting by identifying the letters used.

Our project will revolve around a small NN of the second kind that will “learn” the function y=x^2. The net will first be trained with several pairs of numbers of the form (input, output), with the output being the square of the input. For example: (1,1), (2,4), (3,9), (4,16), (5,25). After this initial “training” the net will (hopefully) act as a simple calculator, returning the square of any input we feed it. Admittedly, this isn’t a very fancy network, but it has all of the ingredients of a more complicated network, and it’s simple to study.

Feed-Forward NNs

As mentioned The best and most common way of setting up a NN that learns by example is to use a feed forward neural network, dubbed henceforth FFNN. A FFNN is made of “layers” of neurons. Let’s look at one such layer:

A Feed Forward Layer

A layer has inputs, x_1 and x_2. These initial inputs are then “fanned out”, resulting in values y_1,y_2,y_3. Note that the number of these intermediate values does not necessarily equal the number of inputs.

y_1 = W_{11} x_1 + W_{12} x_2 \\ y_2 = W_{21} x_1 + W_{22} x_2 \\ y_3 = W_{31} x_1 + W_{32} x_2

This can be rewritten using matrix notation:

\left( y_1 \\ y_2 \\ y_3 \right) = \left( \begin{array}{cc} W_{11} & W_{12} \\ W_{21} & W_{22} \\ W_{31} & W_{32} \end{array} \right) \left( x_1 \\ x_2 \right)

This can be succinctly written as

y=Wx

where y and x are vectors and W is a matrix.
Once the input has been “fanned out”, each intermediate value y_j gets a number, b_j, added to it. This is called a bias. Let’s denote:

h_j=y_j + b_j

or, in matrix notation:

\left( h_1 \\ h_2 \\ h_3 \right) = \left( \begin{array}{cc} W_{11} & W_{12} \\ W_{21} & W_{22} \\ W_{31} & W_{32} \end{array} \right) \left( x_1 \\ x_2 \right) + \left( b_1 \\ b_2 \\ b_3 \right)

We can write this succinctly as:

h=Wx + b

Finally, each h_j goes through an input function. In theory, each h_j could be processed by a different function, but in almost all real life situations, each intermediate value h_j is processed by the same function f. The exact form of f varies from network to network, but here are some common ones:

Various Input Functions

Finally, following the input function, the outputs - which we shall call s_j - will be

s_j=f(h_j)

This is often written, using matrix notation, as:

s=f \left( Wx + b \right)

This comprises the basic layer in a FFNN. As you can see, there’s no magic involved, nor are there any deep conceptual pitfalls to avoid. A convenient way of visualizing a layer is as a “black box”, with N inputs and M outputs:

A Layer as a Black Box

Each layer is characterized by a matrix, W, a bias-vector, b, and an input function, f.
A FFNN is nothing more than a series of layers strung together. One must, of course, match the number of outputs of a layer to the number of inputs of the layer following it:

A Feed-Forward Neural Network

What Next?

How does one FFNN differ from another? As all FFNN are simply made of layers, and each layer is characterized by a matrix W, a vector b and a function f, we can only change a FFNN by altering W, b and f for each layer. In practice, the input functions f are rarely, if ever, changed. W and b are altered. It is possible to prove that by changing W and b any continuous function can be approximated. The trick is to find a method for changing the weights and biases for each layer such that it will “adapt” to the training set we present it with. This trick is called backpropagation, and will be covered in the next part of our tutorial. Stay tuned!

Exercise

Until next time, try and implement a feedforward network in your language of choice. I’ll be giving MATLAB code in our next tutorial, but I assume many of you have their own favorite language. FFNNs are natural candidates for object-oriented programming, as they are made out of layers. A layer is characterized by W, b and f - make it possible for the user to initialize the layer with any input function he chooses out of these three: y=tanh(x), y=x and y=1/(1+e^(-x)), also illustrated above. A layer should accept an input vector and output a vector as well (not necessarily of the same length, of course).

Comments

3 Responses to “Neural Networks Tutorial, Part #1”

  1. Rohit Agarwal on March 1st, 2008 6:31 am

    Hi,
    Using NNtool i have trained feedforward network with 2 layers 1st layer with 65 neurons and tansig as transfer function. 2nd layer with 1 neuron and tansig has tansfer function.
    Traning converge to a satisfactory perfomance value.Trainlm was used as training algo.
    Now by using weights i am trying to check in matlab but i am not getting correct results.
    op = layer1 * ip
    op1 = op + b1
    op1=tansig(op1)
    op2 = op1 * layer2
    op3 = op2 + b2
    op3 is giving 174 where in NNsimulaton it gives 151.
    Am i making some error ???
    Please help this is urgent

  2. vivek chaitanya on March 18th, 2008 10:47 am

    hi,
    Is it possible to do training and testing using NNtool box. If, yes can i know the code for backpropagation algorithm with three layers (input layer with five neurons, hidden layer with three neurons and one output element)……..

    If, no is there any code with the above specifications with matlab basic toolbox

  3. admin on March 19th, 2008 4:53 am

    Hi Vivek,

    It is indeed possible. I’m not really familiar with the toolbox, as I tend to write my own code, but chapters 2-5 of the documentation that comes with the toolbox clearly explain how to set the number of layers, etc … the docs are very good, so there shouldn’t be any problem figurning it out quickly.

    Good luck! :)

Leave a Reply