Why is non linear activation function needed?

Non-linear activation functions are essential in neural networks for several reasons:

1. Introduction of Non-linearity: Non-linear activation functions introduce non-linearity into the network, allowing neural networks to model complex relationships and patterns in the data. Without non-linear activation functions, neural networks would only be able to represent linear transformations of the input data, severely limiting their expressive power.

2. Facilitation of Learning Complex Functions: Neural networks with non-linear activation functions can approximate complex functions and decision boundaries. This enables them to learn and represent highly non-linear relationships between input features and output targets, making them suitable for a wide range of tasks, including image recognition, natural language processing, and speech recognition.

3. Avoidance of Vanishing Gradient Problem: Non-linear activation functions help mitigate the vanishing gradient problem, which occurs when gradients become increasingly small as they propagate backward through deep networks during training. By introducing non-linearity, activation functions enable the network to capture and propagate gradients more effectively, facilitating the training of deep architectures.

4. Enhancement of Model Expressiveness: Non-linear activation functions increase the expressiveness of neural networks by allowing them to learn complex mappings between input and output spaces. This flexibility enables neural networks to capture intricate patterns and variations in the data, leading to improved generalization performance and robustness.

Common non-linear activation functions used in neural networks include the sigmoid function, hyperbolic tangent (tanh) function, rectified linear unit (ReLU), leaky ReLU, and variants like exponential linear units (ELU) and parametric ReLU (PReLU). Each activation function has its characteristics and benefits, and the choice of activation function depends on the specific requirements of the task and the architecture of the neural network.