Basics for Beginners — Activation Functions — Part 4
So far we have seen Machine Learning, Neural Network, Different types of Learning. Let’s dive into Activation Functions.
So far we have discussed about introduction to machine learning, neural networks, different types of learning algorithms. Now let’s look into the activation functions or transfer functions. I will explain with graph and description what exactly it is and how can we interpret this for machine learning modelling. This blog also will be small and to the point.
Transfer Functions:
Activation or Transfer Functions are used to establish the bounds for output of neurons. There are different activation functions for neural network. The format of the input data depends on the selection of activation function. In this section the reader will be able to gain some knowledge about the activation functions.
Linear Function:
This is the most basic function, even though this has less practical use it can be a starting point for next set of transfer functions. It is the identity mapping. The input output relation can be described as shown in Fig.1
Step Function:
Step function is another simple function. This function can be described mathematically as shown in the Fig.1 If the input is 0.5 or greater than it, the function outputs 1 or else the function outputs 0 for all other values. The graph for the function can be shown as in Fig.1
Sigmoid Transfer Function:
The logistic or sigmoid activation function is very common choice for Neural Networks like feed forward which need to output only positive values. Its value is restricted to 0 and 1. The same can be described as Eq.2 and Fig.2
Hyperbolic Tangent:
This activation function is similar to sigmoid function expect that the range is restricted between -1 and +1 . It has similar shape as sigmoid. Fig.2 shows the behaviour of Hyperbolic Tangent.
Hyperbolic Tangent Sigmoid:
In this student project Hyperbolic Tangent Sigmoid is used as activation function. This is the transfer function provided in matlab neural network toolbox. This is mathematically equivalent to hyperbolic tangent, but this function runs faster than the matlab implementation of tanh. This transfer function is chosen for a reason, since the datasets consists of strong-negative values also, if this is provided to logistic sigmoid it results in zero values. Thus negative inputs are mapped to negative outputs and are squashed between -1 and 1. Eq. 4 describes its behaviour mathematically. Here n is the net input vectors.
