Deep Dive into Non-linear Activation Functions

Embarking on a deeper exploration of non-linear activation functions, this article is a continuation of our series on machine learning. Here, we dissect the intricacies of functions like Sigmoid, Tanh, ReLU, and others, providing Python code examples and discussing their applications and limitations.

For a foundational understanding, see our introduction piece on machine learning basics and the importance of activation functions. To learn about selecting the right activation function and implementing it in your models, our next article, Choosing and Implementing the Right Activation Function in ML Models, is a must-read, offering best practices and solutions to common challenges.

Non-linear Activation Functions

Sigmoid Function

The sigmoid function, historically one of the most widely used activation functions in neural networks, serves as a bridge between linear and non-linear processing of data within a network. Its popularity stems from its mathematical properties and the specific advantages it offers in certain neural network configurations.

Description and Mathematical Formula

The sigmoid function is a smooth, S-shaped curve defined by the formula \(f(x) = \frac{1}{1 + e^{-x}}\)​. This function takes any real-valued number and maps it to a value between 0 and 1, making it particularly useful for models where the output is interpreted as a probability.

The shape of the sigmoid function is crucial for its application in neural networks. Early in the training process, small changes in the input can result in significant changes in the output, providing a strong gradient for the optimization algorithm to work with. However, as the input moves away from 0, the function flattens out, leading to smaller changes in the output for a given change in the input.

Python Code Example using TensorFlow/Keras

Implementing a neural network layer with a sigmoid activation function in TensorFlow and Keras is straightforward, thanks to the high-level APIs provided by these libraries. Here’s a simple example of how to use the sigmoid function in a neural network layer:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Define a neural network model with a sigmoid activation function in the output layer
model = Sequential([
    Dense(128, activation='relu', input_shape=(input_shape,)),  # Using ReLU for hidden layers
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')  # Sigmoid function used in the output layer
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

This code snippet creates a neural network suitable for a binary classification problem, using the sigmoid function in the output layer to predict a probability that the input belongs to one of two classes.

Use Cases and Limitations

Use Cases

• Binary Classification: The sigmoid function is ideal for binary classification tasks. Since it outputs values between 0 and 1, it can be interpreted as the probability of the input belonging to a particular class.
• Probability Outputs: In any context where the output needs to be interpreted as a probability, the sigmoid function is an appropriate choice due to its bounded output range.

Limitations

• Vanishing Gradients: One of the major drawbacks of the sigmoid function is the problem of vanishing gradients. As the absolute value of the input grows large, the gradient of the function approaches zero. This flattening effect means that during backpropagation, updates to the weights can become very small, significantly slowing down the training process or leading to convergence at suboptimal weights.
• Not Zero-Centered: The output of the sigmoid function is not zero-centered, which can lead to the gradients for weights being all positive or all negative, potentially resulting in inefficient learning during gradient descent.

Despite these limitations, the sigmoid function played a foundational role in the development of neural network architectures. Its use in modern networks has diminished, particularly in hidden layers, due to the rise of other activation functions like ReLU that mitigate some of the sigmoid’s drawbacks. However, it remains a critical tool for binary classification problems and when modeling probabilities. The understanding of when and how to use the sigmoid function effectively is a valuable part of a machine learning practitioner’s toolkit.

Tanh Function

The hyperbolic tangent function, or tanh, is a popular activation function in neural networks that, like the sigmoid function, transitions inputs smoothly but outputs values ranging from -1 to 1, unlike sigmoid’s 0 to 1 range. This feature provides certain advantages, especially in hidden layers of a network.

Overview and Benefits over Sigmoid

The tanh function is mathematically defined as \(tanh(x) = \frac{e^x – e^{-x}}{e^x + e^{-x}}\). It shares a similar S-shaped curve with the sigmoid function but is zero-centered, meaning it outputs negative values for negative inputs and positive values for positive inputs. This zero-centered nature makes tanh particularly advantageous in many scenarios, as it tends to make the average of the output closer to zero, which helps with the convergence during training by making gradient descent more efficient.

Compared to the sigmoid function, tanh has the benefit of being more effective in situations where maintaining the sign of the input data is important. The fact that its output range includes negative values allows it to represent data more naturally in some cases, especially when the data itself is centered around zero.

Python Code Example using TensorFlow/Keras

Here’s how you can implement a neural network layer using the tanh activation function in TensorFlow and Keras:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Define a neural network model with tanh activation functions in the hidden layers
model = Sequential([
    Dense(128, activation='tanh', input_shape=(input_shape,)),  # Using tanh for hidden layers
    Dense(64, activation='tanh'),
    Dense(1, activation='sigmoid')  # Sigmoid function in the output layer for binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

This example illustrates the use of tanh in the hidden layers while maintaining a sigmoid activation function in the output layer for a binary classification problem. This combination takes advantage of tanh’s efficient training characteristics and the sigmoid’s ability to output probabilities.

Application Scenarios

Hidden Layers in Neural Networks

Tanh is particularly useful in the hidden layers of neural networks due to its zero-centered nature, which can lead to faster convergence in some cases. It’s suitable for networks dealing with data that is naturally centered around zero and can be used in both shallow and deep networks.

Pre-Activation Layers

Tanh can serve effectively in layers before a final classification layer or in networks where the goal is to normalize the output of neurons to a range that mirrors the input data distribution.

Complex Valued Mappings

In scenarios where the model needs to learn complex functions that map inputs to outputs in a range that extends in both positive and negative directions, tanh functions can be more appropriate than sigmoid functions due to their wider output range.

Despite its advantages, tanh also shares the vanishing gradient problem with the sigmoid function, albeit to a lesser extent due to its output range. This means that for very high or very low input values, the gradient of the tanh function becomes very small, potentially slowing down learning. However, tanh remains a popular choice for many applications due to its efficiency and the natural way it handles certain types of data. It’s a powerful tool in the neural network toolkit, especially when used judiciously in conjunction with other activation functions tailored to the specific needs of the model and data.

ReLU (Rectified Linear Unit)

The Rectified Linear Unit (ReLU) has become the default activation function for many types of neural networks because it introduces non-linearity with less computational complexity and mitigates some issues found in sigmoid and tanh functions, particularly the vanishing gradient problem.

Explanation and Advantages for Deep Neural Networks

ReLU is defined mathematically as \(f(x) = max(0, x)\), meaning it outputs the input directly if it’s positive; otherwise, it outputs zero. This simplicity leads to several advantages:

• Efficiency: ReLU is computationally efficient, allowing networks to converge faster during training compared to sigmoid and tanh, due to its linear, non-saturating form.
• Mitigation of Vanishing Gradient Problem: ReLU alleviates the vanishing gradient problem because the gradient is either 0 (for negative inputs) or 1 (for positive inputs). This characteristic ensures that during backpropagation, the gradients do not vanish or diminish as they pass through multiple layers, which is especially beneficial for deep networks.
• Sparse Activation: In a network with ReLU activation, only a subset of neurons is activated at a given time, leading to sparse networks that are often more efficient and easier to train. This sparsity can also lead to better feature representation and generalization.

However, ReLU is not without its drawbacks, such as the “dying ReLU” problem, where neurons can sometimes permanently die during training if they stop outputting anything other than 0. This can be mitigated by variations of ReLU, like Leaky ReLU or Parametric ReLU.

Python Code Example using TensorFlow/Keras

Implementing ReLU in TensorFlow and Keras is straightforward. Here’s an example of using ReLU in a neural network model:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

# Define a neural network model with ReLU activation functions in the hidden layers
model = Sequential([
    Dense(128, activation='relu', input_shape=(input_shape,)),  # Using ReLU for hidden layers
    Dense(64, activation='relu'),
    Dense(1, activation='sigmoid')  # Sigmoid function in the output layer for binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

This code snippet demonstrates the use of ReLU in the hidden layers of a model intended for binary classification, with a sigmoid activation function in the output layer for predicting probabilities.

When to Use ReLU

Deep Neural Networks

ReLU is particularly well-suited for deep neural networks due to its ability to facilitate fast convergence and mitigate the vanishing gradient problem. Its simplicity and efficiency make it a go-to choice for hidden layers in most types of neural networks, including convolutional neural networks (CNNs) and deep feedforward networks.

Models Requiring Fast Computation

The computational simplicity of ReLU makes it ideal for models that require fast computation, such as real-time processing applications or models deployed on mobile devices.

Sparse Networks

For models that benefit from sparsity within the neural network, ReLU can be an effective choice. The nature of ReLU leads to the activation of only a subset of neurons at any given time, which can result in more efficient and interpretable models.

Despite its widespread use, it’s important to be aware of situations where ReLU might not be the best choice, such as tasks that require a model to predict negative values or when there is a risk of many neurons dying. In these cases, variations of ReLU or other activation functions might be more appropriate. Nonetheless, for many applications, especially in deep learning, ReLU remains a highly effective and popular choice.

Leaky ReLU

Leaky ReLU is a variant of the Rectified Linear Unit (ReLU) designed to address one of ReLU’s primary shortcomings: the dying ReLU problem. This enhancement allows for a small, non-zero gradient when the unit’s input is less than zero, thereby keeping the neurons alive during the training process.

Understanding Leaky ReLU and its Difference from ReLU

Leaky ReLU modifies the ReLU function by allowing a small, positive gradient when the input is negative. Mathematically, it’s defined as \(f(x) = x\) for \(x > 0\) and \(f(x) = \alpha x\) for \(x \leq 0\), where \(\alpha\) is a small coefficient (e.g., 0.01). This slight adjustment means that instead of outputting zero for all negative inputs, Leaky ReLU allows a tiny, linear component of those inputs to pass through, preventing the neurons from dying.

Advantages Over ReLU

• Prevents Neurons from Dying: The primary advantage of Leaky ReLU over ReLU is its capacity to allow a small gradient when the input is negative, which helps keep neurons alive and ensures that all parts of the neural network can continue to learn.
• Improved Learning for Networks with Negative Inputs: By allowing negative inputs to influence the learning process, Leaky ReLU can lead to improved model performance, especially in cases where the data has a wide range of values on both sides of zero.

Python Code Example using TensorFlow/Keras

TensorFlow and Keras make it easy to implement Leaky ReLU in your neural network models. Here’s an example:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, LeakyReLU

# Define a neural network model with Leaky ReLU activation functions in the hidden layers
model = Sequential([
    Dense(128, input_shape=(input_shape,)),
    LeakyReLU(alpha=0.01),  # Using Leaky ReLU for hidden layers
    Dense(64),
    LeakyReLU(alpha=0.01),
    Dense(1, activation='sigmoid')  # Sigmoid function in the output layer for binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

In this code snippet, Leaky ReLU is used in the hidden layers of a model intended for binary classification. The LeakyReLU layer is added separately after each Dense layer, with an alpha parameter specifying the slope of the function when the input is less than zero.

Comparison with ReLU

While ReLU and Leaky ReLU perform similarly in many scenarios, the choice between them can significantly impact the performance of certain models.

• Performance: Leaky ReLU can outperform ReLU in networks where the dying ReLU problem is significant, as it ensures that all neurons remain active throughout training. However, in many practical scenarios, the performance difference between ReLU and Leaky ReLU may be minimal.
• Computational Complexity: Leaky ReLU introduces slightly more computational complexity than ReLU due to the need to compute the negative slope. However, this difference is often negligible in practice.
• Use Cases: Leaky ReLU is particularly useful in deeper networks or networks that suffer from poor learning due to many neurons dying when using ReLU. It’s also beneficial in tasks where maintaining the flow of gradients for negative inputs is crucial for model performance.

In conclusion, Leaky ReLU offers a simple yet effective modification to the traditional ReLU activation function, mitigating the dying ReLU problem and potentially leading to better learning and model performance in specific scenarios. Its implementation in TensorFlow and Keras is straightforward, making it an accessible option for improving the robustness of neural network models.

Softmax Function

The Softmax function stands out as a pivotal activation function in the realm of neural networks, particularly for tasks involving multi-class classification. Unlike other activation functions that process inputs independently, the Softmax function considers all the neuron outputs in the layer to produce a probability distribution of the classes.

Description and When to Use Softmax

The Softmax function is mathematically represented as \(f(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}\) for each output \(x_i\) in a layer of \(n\) outputs. This function exponentiates each output, then normalizes these values by dividing each by the sum of all exponentials in the outputs. The result is a vector of probabilities that sum up to 1, with each value representing the probability that the input belongs to one of the classes.

When to Use Softmax

• Multi-Class Classification: Softmax is ideally used in the output layer of a neural network when the task is to classify inputs into more than two classes that are mutually exclusive. For instance, recognizing types of fruits in images where each image distinctly represents one fruit type.
• Probabilistic Outputs: Whenever a model’s output requires a probability distribution across multiple classes, Softmax provides a suitable mechanism to interpret the logits (raw predictions of the model) as probabilities. This is particularly useful not just for classification but also in any setting where understanding the model’s confidence across a range of categories is beneficial.

Python Code Example for Multi-Class Classification using TensorFlow/Keras

Implementing the Softmax function in a TensorFlow/Keras model for multi-class classification is straightforward, as shown in the following example:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.datasets import mnist

# Load dataset
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

# Preprocess the data
train_images = train_images / 255.0
test_images = test_images / 255.0

# Define the model
model = Sequential([
    Flatten(input_shape=(28, 28)),  # Flatten the input images
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')  # Softmax function in the output layer for 10 classes
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_images, train_labels, epochs=5)

# Evaluate the model
test_loss, test_acc = model.evaluate(test_images, test_labels)
print('Test accuracy:', test_acc)

In this example, the model is designed for the MNIST dataset, a collection of handwritten digits from 0 to 9, making it a 10-class classification problem. The final layer uses the Softmax function to output a probability distribution over the 10 possible classes. This setup is typical for multi-class classification problems where each input is expected to belong to one and only one class.

The use of sparse_categorical_crossentropy as the loss function complements the Softmax activation in the output layer by calculating the loss between the labels and the predicted probabilities, facilitating effective learning.

In summary, the Softmax function is integral to models tackling multi-class classification problems, converting the raw logits of a neural network into interpretable probabilities. Its implementation in TensorFlow and Keras is both straightforward and efficient, enabling the development of powerful and intuitive classification models.

Swish Function

The Swish function is a relatively recent addition to the activation function repertoire in neural networks, having been identified through automated search techniques for machine learning algorithms. It has attracted attention for its performance benefits over traditional activation functions in certain scenarios.

Introduction to Swish and its Benefits

Swish is defined by the formula \(f(x) = x \cdot sigmoid(\beta x)\), where \(x\) is the input to the function, and \(\beta\) is a constant or trainable parameter. When \(\beta = 1\), the function simplifies to \(f(x) = x \cdot sigmoid(x)\), which is the most common variant used in practice. The Swish function combines aspects of both linear and non-linear functions, allowing small negative values when \(x\) is negative, and behaving similarly to ReLU for positive values of \(x\).

Benefits of Swish

• Smooth Gradient: Unlike ReLU, which has a discontinuity at \(x=0\), Swish is smooth everywhere. This smoothness can help with the optimization process, potentially leading to faster convergence.
• Non-zero Gradient for Negative Inputs: Swish allows small gradients when the input is negative, which helps prevent neurons from dying—a common problem with ReLU.
• Adaptive ReLU: Swish can be seen as an adaptive version of ReLU that can either allow small values to pass when negative or act like ReLU when positive. This adaptiveness makes it versatile.

Python Code Example using TensorFlow/Keras

Implementing Swish in TensorFlow and Keras is straightforward, especially since TensorFlow 2.x includes Swish as a built-in activation function. Here’s an example of how to use it in a neural network model:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten

# Define the model
model = Sequential([
    Flatten(input_shape=(28, 28)),  # Flatten the input layer for image data
    Dense(128, activation='swish'),  # Using Swish activation function
    Dense(10, activation='softmax')  # Softmax for the output layer
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Summary of the model to show layers and use of Swish
model.summary()

This code defines a simple neural network suitable for tasks like image classification, with Swish activation in the hidden layers and Softmax in the output layer for multi-class classification.

Performance Comparison with ReLU and When to Prefer Swish

Performance Comparison

• Empirical Results: Studies and empirical results have shown that Swish often outperforms ReLU on deeper models across a variety of datasets and tasks. It tends to work particularly well in deeper networks where the smoothness of the activation function can aid in optimization.
• Versatility: Swish has demonstrated versatility and robustness in a range of neural network architectures, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

When to Prefer Swish

• Deep Neural Networks: Swish tends to shine in deeper networks where the benefits of its smooth gradient and adaptiveness can be fully leveraged.
• Tasks Requiring Flexibility: For tasks where negative inputs play a significant role, and there’s a need for a more flexible activation function, Swish can provide an advantage over ReLU.
• Exploratory Projects: In scenarios where you’re exploring the model architecture and seeking potential performance improvements, experimenting with Swish as an alternative to ReLU could yield beneficial results.

Despite its benefits, the choice between Swish and ReLU (or other activation functions) should be informed by empirical testing within the specific context of your project. While Swish has shown promise in various research and applications, the optimal activation function can depend on numerous factors, including the specific dataset, network architecture, and training regimen. Therefore, it’s recommended to experiment with Swish and compare performance against ReLU in your specific use case to determine the best option.

Diving deep into non-linear activation functions has unveiled their critical role in shaping powerful neural networks. To build on what you’ve learned, Choosing and Implementing the Right Activation Function in ML Models awaits, packed with advice on making the right choices for your neural network.

For newcomers or those needing a refresher on the basics, our introductory article provides a solid foundation in machine learning and the significance of activation functions, ensuring a well-rounded understanding of these essential technologies.