Softmax Activation Function

Definition:

The Softmax function is a function that takes as input a vector of K real numbers and normalized it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. After Softmax, each component will be in interval [0,1] and the components will be added up to 1, so that they can be interpreted as probabilities.

In words: we apply the standard exponential function to each element zi of the input vector z and normalized these value by dividing by the sum of all these exponentials. This normalization ensures that the sum of the exponents of the output vector(z) is 1.

data = [1, 3, 2 ]

# Max
result = max(data)
# -> 3

# ArgMax
from numpy import argmax
result = argmax(data)
# -> 1

# SoftMax
from math import exp
p1 = exp(1) / (exp(p1) + exp(p2) + exp(p3))
p2 = exp(3) / (exp(p1) + exp(p2) + exp(p3))
p3 = exp(2) / (exp(p1) + exp(p2) + exp(p3))

print(p1, p2, p3)
# -> 0.09003057317038046 0.6652409557748219 0.24472847105479767
print(p1 + p2 + p2)
# -> 1.0

# SoftMax input list
from numpy import exp
import numpy as np

def softmax(data):
    e = exp(data)
    return e/e.sum()

result = softmax(data)
print(result) # [0.09003057 0.66524096 0.24472847]
print(sum(result)) # 1.0

# Softmax use scipy import package
from scipy.special import softmax
result = softmax(data)