Optimize to actualize: The impact of hyperparameter tuning on AI

In machine learning, algorithms harness the power to unearth hidden insights and predictions from within data. Central to the effectiveness of these algorithms are hyperparameters, which can be thought of as the architects of a model’s behavior. According to Fortune Business Insights, the global Machine Learning (ML) market size was valued at $19.20 billion in 2022 and is projected to expand to $225.91 billion by 2030—a testament to the significance of advancements in this field.

Hyperparameters, including model complexity, learning rate, and regularization strength, are preset configurations that guide the learning process. Fine-tuning these hyperparameters is an art that combines experience, experimentation, and domain knowledge, much like a conductor skillfully harmonizing each musician’s performance with the subtle intricacies of the data.

This article explores the intricacies of hyperparameter tuning in machine learning, underscoring its pivotal role in the efficacy of models. It discusses various techniques, best practices, and key distinctions between parameters and hyperparameters, equipping you with a comprehensive understanding of this vital process in machine learning.

• Hyperparameter tuning: What does it entail?
• Differences between model parameters and hyperparameters
• Understanding hyperparameter space and distributions
• Categories of hyperparameters
  • Hyperparameter for specific models
  • Hyperparameter for optimization
• The significance of hyperparameter tuning
• Techniques for hyperparameter tuning
• How to perform hyperparameter tuning using Python?
• Best practices for hyperparameter tuning

Hyperparameter tuning: What does it entail?

Hyperparameter tuning is a critical aspect of machine learning, involving configuration variables that significantly influence the training process of a model. These hyperparameters, which are set prior to training, are not to be confused with input data or model parameters. Let’s clarify these terms:

Input data consists of individual records, each with features pertinent to the machine learning problem at hand. During training, the model uses this data to learn patterns and make adjustments to its parameters. Consider a dataset containing images of cats and dogs, where each image is characterized by attributes such as ear shape or fur color. The model leverages these features to distinguish between the two animals, but the data itself does not constitute part of the model’s parameters.

Model parameters are the internal variables that a machine learning algorithm modifies to fit the data. For instance, it might adjust the importance (or weights) it assigns to features like ear shape and fur color to differentiate between cats and dogs more effectively. These parameters evolve as the model learns from the training data.

Hyperparameters, on the other hand, are the settings that oversee the training process. These include decisions like the number of layers in a neural network or the number of neurons in each layer. While they significantly affect how quickly and competently the model learns, they are not derived from the training data itself.

Hyperparameter tuning is the methodical experimentation with various hyperparameter combinations to find the set that maximizes model performance. It’s an iterative process that balances the model’s complexity with its ability to generalize from the training data. This process is essential for enhancing the model’s predictive accuracy.

Differences between model parameters and hyperparameters

Below is a comparison table between model parameters and hyperparameters in the context of machine learning:

Thus, model parameters are internal variables learned during training, specific to the model architecture, and represent its knowledge. In contrast, hyperparameters are external settings that control the learning process and need to be carefully chosen to optimize the model’s performance.

Understanding hyperparameter space and distributions

The hyperparameter space refers to the complete set of potential combinations of hyperparameters that can be used to train a machine learning model. It’s like a multidimensional playground, where each dimension represents a different hyperparameter. For instance, if we have two hyperparameters, such as the learning rate and the number of hidden layers in a neural network, the hyperparameter space would consist of two dimensions—one for the learning rate and another for the number of hidden layers.

Within this hyperparameter space, we have the hyperparameter distribution, which acts as a map showing the range of values each hyperparameter can take on and the likelihood of each value occurring. It helps us understand which values are more probable and less likely.

To tune the hyperparameters for the ML model, we need to search this hyperparameter space. This involves trying out various combinations of hyperparameters to identify the ones that result in the best model performance. The choice of hyperparameter distribution is critical as it influences how we explore this space. It determines the ranges of values we consider and how frequently we test particular values.

Categories of hyperparameters

Hyperparameters in machine learning can be divided into two categories, which are given below:

Hyperparameter for optimization

Hyperparameters are pivotal in optimizing machine learning models, affecting both the speed of convergence toward an optimal solution and the model’s capacity to generalize from training data to unseen data. Some common hyperparameters include:

Learning rate: 

• The learning rate is like a knob that controls how quickly a machine-learning model learns from new data. If the learning rate is too high, the model might not learn properly and could jump over the solutions. On the other hand, if the learning rate is too low, the model will learn very slowly.
• Imagine the model has many settings (parameters) to adjust to perform well. The learning rate controls the magnitude of the update during each step in the learning process. Finding the best settings is like finding the lowest point on a bumpy road (error curve). If the learning rate is set correctly, the model might be better assisted in finding the optimal solution.

So, picking the right learning rate is essential for the model to learn effectively and improve its performance. It’s like finding the best speed for the model’s learning process to navigate smoothly and reach the best solutions.

Batch size: 

• The training data set is divided into smaller batches to accelerate learning. When using the stochastic training procedure, the model is trained, evaluated, and backpropagated on each small batch, allowing the adjustment of hyperparameter values. This process is then repeated for the entire training data set.
• If the batch size is larger, it will increase the learning time, as the model processes more data in each step and requires more memory for matrix multiplication. On the other hand, if the batch size is smaller, there will be more noise in the error calculation because the model learns from a smaller set of data, which can lead to less stable updates.
• Using smaller batches speeds up learning but introduces more noise, while larger batches slow down learning and require more memory for processing.
• Finding the right balance in batch size is important for achieving efficient and accurate model training. The noise introduced with smaller batches can sometimes be beneficial by providing a form of regularization that can help generalize the model.

Number of epochs:

• In machine learning, an epoch represents a complete cycle of the model learning from the data. It refers to the number of times the entire training dataset is passed forward and backward through the neural network. It plays a crucial role in the iterative learning process. We use validation errors to determine the appropriate number of epochs.
• We can increase the number of epochs if the validation error keeps decreasing. However, if the validation error stops improving for consecutive epochs, it’s a sign that we should stop increasing the number of epochs. This approach is known as early stopping, and it helps prevent overfitting and ensures we find the optimal point where the model performs well without unnecessary extra training.

Hyperparameter for specific models

Hyperparameters are influential in structuring machine learning models, and some are integral to the model’s architecture. Below are key hyperparameters to consider:

• Number of hidden units: Choosing the right number of hidden units in layers of deep learning models is crucial. This choice determines the model’s capacity to learn and represent complex data patterns. Excessive hidden units can lead to overfitting, but this risk is mitigated by employing advanced regularization techniques and leveraging large datasets. Thus, the optimal number is influenced by the model’s context, available data, and the complexity of the function being learned.
• Number of layers: The depth of a neural network, indicated by the number of layers, is pivotal for learning hierarchical feature representations. While deeper networks have the potential for superior performance, simply adding layers does not ensure better results. Issues such as vanishing gradients and overfitting must be addressed with proper network design and training strategies. In specific applications like image and language processing, deep networks have set new benchmarks, but each additional layer should be justified with empirical evidence and aligned with the problem’s complexity.

In convolutional neural networks (CNNs), layer depth is especially critical as it allows the network to extract finer levels of detail with each subsequent layer. Yet, the increased performance from additional layers is contingent upon the architecture’s ability to learn from the added complexity efficiently. Technologies like skip connections and batch normalization can aid in harnessing the power of deeper networks. Therefore, the determination of the ideal architecture requires a strategic balance informed by both systematic experimentation and a deep understanding of the task at hand.

The significance of hyperparameter tuning

Hyperparameter tuning in machine learning is vital for several reasons:

Optimizing performance: Fine-tuning hyperparameters can significantly improve model accuracy and predictive power. Small adjustments in hyperparameter values can differentiate between an average and a state-of-the-art model.

Generalization: Optimally tuned hyperparameters enable the model to generalize effectively to new, unseen data. Models that are not properly tuned may exhibit good performance on the training data but often struggle to perform adequately when faced with unseen data.

Efficiency: Properly tuned hyperparameters allow models to converge faster during training, reducing the time and computational resources required to develop a high-performing model.

Overfitting and underfitting: Hyperparameter tuning aids in finding the right balance that prevents overfitting (when the model is too complex and performs well on training data but poorly on test data) or underfitting (when the model is too simplistic and performs poorly on training and test data).

Techniques for hyperparameter tuning

Several techniques are available for hyperparameter tuning, ranging from manual search to automated optimization algorithms. Let’s explore some popular approaches:

Manual search

When manually tuning hyperparameters, we often begin with default recommended values or rules of thumb. We then proceed to explore a range of values through trial and error. However, this manual approach becomes impractical and time-consuming when dealing with numerous hyperparameters with a wide range of possible values.

Grid search

Grid search is a “brute force” hyperparameter tuning method that creates a grid of all possible discrete hyperparameter values. We then train the model with each combination in the grid and record its performance. The combination that yields the best performance is selected as the optimal set of hyperparameters.

While grid search ensures finding the best hyperparameters, it has a drawback – it is slow. Since it involves training the model with every possible combination, it requires significant computation capacity and time. This can be impractical when dealing with many hyperparameters or a wide range of possible values, making grid search less feasible for complex models or datasets with limited resources.

Due to its exhaustive nature, grid search is often very thorough, albeit at the cost of efficiency. Additionally, grid search may not be suitable for hyperparameters that do not have a clear discrete set of values to iterate over.

Random search

As the name suggests, the random search method selects hyperparameter values randomly instead of using a predefined grid of values like the grid search method.

In each iteration, a random search tries a random combination of hyperparameters and records the model’s performance. After several iterations, it comes with the combination that produces the best result. Random search is particularly useful when dealing with multiple hyperparameters with large search spaces. It can still provide reasonably good combinations even with discrete ranges.

The advantage of random search is that it typically takes less time compared to grid search to yield similar results. Additionally, it helps avoid bias towards specific hyperparameter values set by users arbitrarily.

However, the drawback of random search is that it may not always find the absolute best hyperparameter combination. It relies on random chance, and it may miss some promising areas of the search space. Despite this limitation, random search is a popular and efficient choice for hyperparameter tuning, especially when grid search becomes computationally prohibitive or when the exact optimal values are not known in advance.

Bayesian optimization

Bayesian optimization is a technique used by data scientists to find the optimal hyperparameters for a machine learning algorithm, particularly when the optimization problem does not have a closed-form solution. This method turns the search for the best hyperparameters from mere guesswork into a structured, probabilistic optimization process.

The process begins by evaluating a few hyperparameter combinations and observing their performance. It then employs a probabilistic model, such as Gaussian processes, random forest regression, or tree-structured Parzen estimators, to predict the performance of other hyperparameter configurations.

This model acts as a surrogate for the actual evaluation function, estimating the performance of various hyperparameter combinations. Using this approach, Bayesian optimization focuses its search on areas of the hyperparameter space that are expected to yield better results. The probabilistic model is refined continuously with each evaluation, enhancing its ability to guide the search to more promising regions.

One of the key benefits of Bayesian optimization is the efficiency gained by evaluating the cheap-to-compute probabilistic model rather than the expensive objective function—such as model training and validation error—repeatedly. This surrogate model approach reduces the number of times the expensive objective function needs to be run. Through a cycle of iterative updates and informed selection of hyperparameter combinations, Bayesian optimization can quickly hone in on a set of hyperparameters that produces favorable outcomes.

A principal advantage of Bayesian optimization is its efficiency in converging on an optimal or near-optimal solution with fewer iterations than other methods like grid or random search. As the surrogate model becomes more accurate, it increasingly identifies more promising areas of the hyperparameter space, which often results in a faster convergence to the best solution.

Genetic algorithms

Genetic algorithms are optimization methods inspired by the process of natural selection in biology and are utilized to solve various optimization problems, including hyperparameter tuning in machine learning. The method initiates with a diverse population of potential hyperparameter sets and iteratively evolves this population.

Each iteration involves selecting superior individuals from the current population to act as “parents” based on their performance, which is analogous to the fitness function in biological evolution. These parents produce “children” or successors, which inherit hyperparameters from their progenitors, possibly with some random variation, simulating biological mutation.

Through successive generations, the population of hyperparameter sets is refined, with each generation hopefully yielding better model performance than the last. The genetic algorithm is adept at navigating complex, high-dimensional search spaces and can be particularly beneficial when the hyperparameter optimization problem is non-convex or the landscape is rugged with many local optima.

While genetic algorithms are versatile and robust, offering solutions in cases where other methods struggle—like with discontinuous, non-differentiable, stochastic, or highly nonlinear objective functions—they can be computationally demanding. This is due to the need to evaluate the performance of many potential solutions across multiple generations. However, their global search capability is often worth the computational expense, especially when the search space is vast and other optimization techniques fail to provide satisfactory solutions.

Genetic algorithms are just one of the myriad techniques available for hyperparameter tuning in machine learning, each with its unique strengths and applicable contexts.

How to perform hyperparameter tuning using Python?

We will use GridSearchCV from the sklearn.model_selection package to tune all the parameters for our machine learning models. It performs an exhaustive search over a specified hyperparameter grid, evaluating the model’s performance using cross-validation. It helps find the best combination of hyperparameters that optimize the model’s performance. This process is crucial for fine-tuning models and improving their predictive capabilities. You can find the data from the link: https://archive.ics.uci.edu/dataset/17/breast+cancer+wisconsin+diagnostic

Follow these steps for hyperparameter tuning using the support vector machine (SVM) method:

Step-1: Import all the libraries

By importing and harnessing the power of libraries, we can efficiently handle data, create meaningful visualizations, and accurately evaluate our machine-learning model’s effectiveness. In this project, we’ll be using the following essential libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import metrics
from sklearn.model_selection import GridSearchCV

# Importing the dataset
data = pd.read_csv("../input/data.csv")

Step-2: Inspect and clean the data

We will use the following code to inspect and clean the data:

# Printing the 1st 5 columns
data.head()

When you call data.head(), it will show you the top rows of the data frame, which allows you to get a quick glimpse of the data and its structure.

Step-3: Identifying data dimension

In Python, when you call data.shape on a pandas DataFrame; it returns a tuple representing the dimensions of the DataFrame. The tuple contains two values:

1. The first value represents the data frame’s number of rows (instances).
2. The second value represents the data frame’s number of columns (features).

# Printing the dimensions of data
data.shape

Step-4: Viewing column heading

When you call data.columns on a pandas data frame, it returns an Index object that contains the column names of the data frame. You can use the following code:

# Viewing the column heading
data.columns

Step-5: Data diagnosis

Use the following code for this process:

# Inspecting the target variable
data.diagnosis.value_counts()

The code data.diagnosis.value_counts() shows the count of each unique value in the ‘diagnosis’ column of the dataset.

Step-6: Data type

DataFrame data contains various columns representing different attributes related to breast cancer analysis. Use the following code for accurate output:

data.dtypes

Step-7: Identify unique values

This step helps us understand how many different values exist in each column. Use the code below:

# Identifying the unique number of values in the dataset
data.nunique()

data.nunique() counts and returns the number of unique values in each column of the dataset named data

Step-8: Check missing values

This step is used to check for the presence of NULL or missing values in the dataset named data. It will return the number of NULL values present in each column of the dataset.

# Checking if any NULL values are present in the dataset
data.isnull().sum()

The number of null values can be crucial for data cleaning and analysis, as NULL values must be handled appropriately depending on the data context and the analysis being performed. (Here, in the output, there is one section named “Unnamed 32”, which has a missing value.

Step-9: Drop the Id column

Dropping the Unnamed: 32 and the id column since these do not provide any useful information for our models.

data.drop(['Unnamed: 32', 'id'], axis=1, inplace=True)

After this, we will check rows with missing values. Use the following code:

# See rows with missing values
data[data.isnull().any(axis=1)]

Use data.describe() function provides a statistical summary of the dataset data. It computes various descriptive statistics for each numerical column in the dataset, including count, mean, standard deviation, minimum, quartiles and maximum values.

# Viewing the data statistics
data.describe()

Step-10: Finding the correlation

In this step, we calculate the correlation between the features (columns) in the dataset named data and then print the shape of the resulting correlation matrix.

# Finding out the correlation between the features
corr = data.corr()
corr.shape

Step-11: Plotting the heat map

We will create a heatmap of the correlation between features in the dataset. A heatmap is a graphical representation of data where individual values are represented as colors.

# Plotting the heatmap of correlation between features
plt.figure(figsize=(20,20))
sns.heatmap(corr, cbar=True, square= True, fmt='.1f', annot=True, annot_kws={'size':15}, cmap='Greens')
plt.show()

Step-12: Analyzing the target variable

analyze the target variable, specifically related to the count of different cancer types in the dataset. The target variable is represented by the column ‘diagnosis’, and the code creates a count plot to visualize the distribution of different cancer types.

# Analyzing the target variable
plt.title('Count of cancer type')
sns.countplot(data['diagnosis'])
plt.xlabel('Cancer lethality')
plt.ylabel('Count')
plt.show()

Step-13: Correlation between diagnosis and radius

Here, we will create two side-by-side plots to visualize the correlation between the dataset’s target variable (‘diagnosis’) and the ‘radius_mean’ feature.

# Plotting correlation between diagnosis and radius
plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
sns.boxplot(x="diagnosis", y="radius_mean", data=data)
plt.subplot(1,2,2)
sns.violinplot(x="diagnosis", y="radius_mean", data=data)
plt.show()

Step-14: Correlation between diagnosis and concativity

We will create two side-by-side plots to visualize the correlation between the target variable (‘diagnosis’) and the ‘concavity_mean’ feature in the dataset.

# Plotting correlation between diagnosis and concativity
plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
sns.boxplot(x="diagnosis", y="concavity_mean", data=data)
plt.subplot(1,2,2)
sns.violinplot(x="diagnosis", y="concavity_mean", data=data)
plt.show()

Step-15: Create a kernel density estimate (KDE)

Create a kernel density estimate (KDE) plot using Seaborn’s FacetGrid to visualize the distribution of the “radius_mean” feature for each category of the target variable “diagnosis” (benign or malignant).

# Distribution density plot KDE (kernel density estimate)
sns.FacetGrid(data, hue="diagnosis", height=6).map(sns.kdeplot, "radius_mean").add_legend()
plt.show()

Step-16: Visualize the distribution

Create a strip plot to visualize the distribution of the “radius_mean” feature for each category of the target variable “diagnosis” (benign or malignant).

# Plotting the distribution of the mean radius
sns.stripplot(x="diagnosis", y="radius_mean", data=data, jitter=True, edgecolor="gray")
plt.show()

Now, create a grid of 16 scatter plots (bivariate relations) to visualize the relationships between each pair of selected features (“radius_mean”, “concavity_mean”, “smoothness_mean”, and “texture_mean”) with the target variable “diagnosis” (benign or malignant) indicated by different colors.

# Plotting bivariate relations between each pair of features (4 features x4 so 16 graphs) with hue = "diagnosis"
sns.pairplot(data, hue="diagnosis", vars = ["radius_mean", "concavity_mean", "smoothness_mean", "texture_mean"])
plt.show()

Step-17: Split the data

Once the data is cleaned, split the data into training and test sets to prepare it for our machine learning model in a suitable proportion.

# Spliting target variable and independent variables
X = data.drop(['diagnosis'], axis = 1)
y = data['diagnosis']

Step-18: Use the SVM classifier

The following codes can be used to train the model on the SVM classifier:

# SVM Classifier
# Creating scaled set to be used in model to improve the results
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Import Library of Support Vector Machine model
from sklearn import svm
# Create a Support Vector Classifier
svc = svm.SVC()
# Hyperparameter Optimization
parameters = [
  {'C': [1, 10, 100, 1000], 'kernel': ['linear']},
  {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
]
# Run the grid search
grid_obj = GridSearchCV(svc, parameters)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the svc to the best combination of parameters
svc = grid_obj.best_estimator_
# Train the model using the training sets 
svc.fit(X_train,y_train)

# Prediction on test data
y_pred = svc.predict(X_test)

In [39]:linkcode# Calculating the accuracy
acc_svm = round( metrics.accuracy_score(y_test, y_pred) * 100, 2 )
print( 'Accuracy of SVM model : ', acc_svm )

You can check the entire code from: https://www.kaggle.com/datasets/uciml/breast-cancer-wisconsin-data?resource=download

Best practices for hyperparameter tuning

Irrespective of the tuning method employed, adhering to certain best practices can improve the efficiency and efficacy of the hyperparameter tuning process.

Define a reasonable search space: When setting up hyperparameter tuning, it’s important to establish a focused range for each hyperparameter. Choosing too wide a range can make the tuning process less effective and more time-consuming. It’s like searching for treasure in the ocean; if you spread your search too wide, it becomes harder to find the treasure. By narrowing down the search area to the most promising regions, based on expert knowledge or preliminary tests, you can more efficiently find the best settings for your model. This approach not only saves time but also ensures that your model is practical and performs well.

Cross-validation: Use cross-validation to evaluate the performance of different hyperparameter configurations. Cross-validation is a crucial technique for comparing the effectiveness of different hyperparameter settings during the tuning process. It involves dividing the dataset into k subsets, training and evaluating the model k times using different subsets as the test set and the rest as the training set. Cross-validation provides a reliable estimate of the model’s performance for each hyperparameter configuration by summarizing the results over these iterations.

Parallelization: Parallelization means doing multiple things at the same time. If you have enough computer power, it’s a good idea to use parallelization for hyperparameter tuning. This is like having several people searching for something at once instead of just one person doing all the work. By doing this, you can find the best settings for your model much faster.

Focus on promising areas: In certain cases, Bayesian optimization or other intelligent search algorithms can be more efficient by focusing on areas where the performance is likely to improve.

Keep track of results: Maintain a record of hyperparameter configurations and their corresponding performance metrics. This helps in analyzing the tuning process and identifying patterns.

Endnote

Effective hyperparameter tuning is often the difference between an average and an exceptional machine learning model. As we have witnessed throughout this exploration, the right combination of machine learning hyperparameters can unleash a model’s true potential, elevating its performance to new heights and ensuring its adaptability to unseen data. By skillfully tuning hyperparameters, data scientists can build machine learning models that excel in performance and transcend the boundaries of what is achievable with artificial intelligence. The quest for excellence continues, with every refined configuration pushing the boundaries of innovation and unlocking new frontiers of knowledge.

As we continue to advance the frontiers of artificial intelligence and machine learning, the mastery of hyperparameter tuning in machine learning becomes more crucial. It empowers us to harness the full potential of complex models, tackle real-world challenges, and unlock new insights in diverse domains.

Boost your machine learning outcomes with precise hyperparameter tuning. Connect with LeewayHertz for unmatched expertise.