Neural Networks Tutorial, Part #2
Posted on December 7, 2007
Filed Under Algorithms |
A Short Review
In the previous part of our tutorial, I’ve spoken about feed forward neural networks (FFNN), and said that the basic building block of a FFNN is a layer, which can be concisely summarized using the following diagram:
A layer has N inputs and M outputs (schematically represented here by a single line at the input and a single line at the output), and is characterized by three parameters: a weight matrix W, a bias vector b, and a transfer function, also called an input function, f. If the input of such a layer is given by the vector x = (x1, x2, …, xN), then the output, y=(y1,y2,…,yM) will be given by:
y = f(Wx + b)
There are a few input functions that are used repeatedly throughout the literature. Here are the most common 3:
- The linear transfer function:
, where beta is some constant. - The sigmoid hyperbolic tangent:
, where beta is some constant. - The sigmoid function
, where beta is some constant.
Let’s Get Practical
We will be studying a two-layer FFNN. Our goal will be to “teach it” the function 
. We will do this by training the network on a set of inputs and outputs, and then hope it will manage to infer the function at all remaining points. Ok, beating around the bush. It’s time to introduce some notations. Here is a schematic representation of our network:
I’ve introduced a few notations in this diagram. Let’s go over them.
- x stands for our input vector - at the moment we won’t any assumptions on its size, or the number of inputs/outputs of our network, although eventually we’ll choose our network to have a single input and a single output (the function has a single input, x, and a single output, f(x)).
-
. Be aware I’m using matrix notation here. -
. -
. - And finally,
is our output.
Note that
are all distinct variables. Although we will eventually choose f to be a sigmoid and 
to be linear, we won’t need these assumptions to derive our conclusions below.
Training Our Network
As noted earlier,our final goal will be to train our network to mimic the function . This means that we will “feed” our network with various inputs - say, x=1, 2, 3, 4, 5, etc. - and observe the outputs. Ideally, when inputting x, we’d like to see the network output its square - e.g., when inputting 7, we’d like to get 49 as our output. Obviously, this will not happen by default. We will need to observe the actual outputs, and adjust 
appropriately, until the actual outputs approximate the desired outputs closely enough. This is known as training the network.
How do we go about training our network? Fortunately for us, there is a nifty algorithm known as backpropagation which was invented for training FFNN. We will cover backpropagation in our next installment, so stay tuned!
Comments
One Response to “Neural Networks Tutorial, Part #2”
Leave a Reply
[…] our previous tutorial we’ve laid out the basic form of our two layer feed-forward neural network (FFNN). In this […]