Hyperparameters

What Is Hyperparameter Tuning?

Hyperparameter tuning is the process of choosing the best settings for a machine learning model before training begins.

These settings - unlike model parameters learned during training - control how the model learns. Good tuning improves accuracy and prevents overfitting or underfitting.

Evaluating Model Performance

In supervised learning, we use metrics to judge how well our model is doing - not just overall, but in terms of what it gets right, what it misses, and how confidently it makes decisions.

Accuracy

Accuracy measures how often the model gets things right - the proportion of total predictions that were correct.

Example 1: A spam filter checks 100 emails. If it correctly labels 90 of them (as either spam or not), the accuracy is 90%.

Example 2: In a handwriting recognition task, if the model correctly reads 95 out of 100 handwritten digits, the accuracy is 95%.

Note: Accuracy can be misleading when the dataset is imbalanced - for example, if 95% of emails are not spam, a model that always guesses "not spam" will score 95% accuracy but still be useless.

Precision

Precision answers the question: “Of the items the model labelled as positive, how many actually were?” It tells us how much we can trust the model’s positive predictions.

Example 1: A model predicts 50 emails are spam. If only 40 of those are actually spam, precision is 40/50 = 80%.

Example 2: In a cancer screening tool, if it flags 20 patients as at-risk but only 15 actually have early signs, the precision is 75% - meaning 25% were false alarms.

High precision means fewer false positives (incorrect alerts).

Recall

Recall answers the question: “Of all the actual positive cases, how many did the model catch?” It focuses on coverage - how well the model finds what it’s looking for.

Example 1: There are 60 spam emails in total, and your model correctly identifies 48 of them. Recall is 48/60 = 80%.

Example 2: In a disease detection system, if there are 100 true cases and the model identifies 85, recall is 85% - meaning it missed 15 people who actually had the disease.

High recall means fewer false negatives (missed positives).

F1 Score

The F1 score combines precision and recall into a single number - it’s the harmonic mean of the two. This is especially useful when there’s an imbalance between classes or when both false positives and false negatives matter.

How it works: If precision is 80% and recall is 80%, the F1 score is also 80%. But if one is much lower than the other, the F1 score drops. It only stays high if both precision and recall are high.

Example 1: In a fraud detection model, if precision is 70% and recall is 70%, F1 score is around 70% - showing balanced performance.

Example 2: In a rare disease test, if precision is 90% but recall is only 50%, the F1 score drops to about 64%. It reflects the trade-off between being accurate and catching all cases.

F1 helps when accuracy alone doesn’t tell the full story - especially with unbalanced datasets.

Understanding the F1 Score

The F1 score is not the average of precision and recall - it’s the harmonic mean, which rewards balance and penalises uneven performance.

F1 Score Formula

\[ \text{F1 Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \]

Precision & Recall

In this example:

Precision: 90%
Recall: 60%

High precision means the model is usually right when it says something is positive.

Lower recall means it’s missing many actual positives.

Simple (Arithmetic) Mean

\[ \text{Mean} = \frac{\text{Precision} + \text{Recall}}{2} = \frac{0.9 + 0.6}{2} = 0.75 \]

Seems okay - but hides how poorly the model is in one of the values (in this case, recall).

F1 Score (Harmonic Mean)

\[ F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = 2 \times \frac{0.9 \times 0.6}{0.9 + 0.6} = 2 \times \frac{0.54}{1.5} = 0.72 \]

Lower than the arithmetic mean - highlights that the weaker of the two values (in this case, recall) is limiting performance.

The F1 score is cautious and balanced.

Why Use F1?

When classes are imbalanced or false positives/negatives have different consequences, the F1 score is a fairer, more reliable measure than accuracy or simple averages.

The Role of Hyperparameter Tuning

Hyperparameters guide how the model learns. Proper tuning can make the difference between a model that works and one that fails.

Hyperparameter	Effect on the Model
Learning Rate	Controls how quickly the model learns during training. Too high: Model may overshoot optimal values and fail to converge. Too low: Training becomes very slow and may get stuck in local minima.
Number of Neighbours (K in K-NN)	Defines how many neighbours influence the classification. Too low: Sensitive to noise; model may overfit. Too high: May blur class boundaries; model may underfit.
Max Depth (in Decision Trees)	Restricts how many splits the tree can make. Too deep: Overfits training data by capturing noise. Too shallow: Misses important patterns and underfits.
Number of Epochs	Sets how many times the full dataset is used during training. Too few: Model might not learn enough (underfitting). Too many: Model may learn noise and overfit.
Batch Size	Determines how many samples are processed before the model updates. Too small: Noisy learning with slower convergence. Too large: May miss finer patterns and use more memory.
Regularisation (L1, L2)	Adds a penalty to large weights to discourage complexity. Too weak: Doesn’t prevent overfitting. Too strong: Can oversimplify and lead to underfitting.

Example: Tuning Hyperparameters in Python

Scenario: Choosing the best value for k in K-NN classification.

            from sklearn.neighbors import KNeighborsClassifier

            from sklearn.model_selection import GridSearchCV

            knn = KNeighborsClassifier()

            param_grid = {'n_neighbors': [3, 5, 7, 9, 11]}

            grid_search = GridSearchCV(knn, param_grid, cv=5)

            grid_search.fit(X_train, y_train)

            print("Best value for K:", grid_search.best_params_['n_neighbors'])

Overfitting vs Underfitting

Training vs Validation

We split data so we can tell if a model truly learns patterns or just memorises. Training data teaches the model; validation data checks how well it works on new, unseen examples while we’re still building it.

Set	What it’s for	Does the model see it while learning?	Typical use
Training	To fit the model (adjust its internal settings).	Yes - used directly to learn.	Learn patterns, reduce training loss.
Validation	To check generalisation during development.	No - kept aside from learning.	Compare models, tune hyperparameters, decide early stopping.

Overfitting

Overfitting occurs when a model learns the training data too well - including its noise and outliers. This means it performs very well on the training data but struggles on unseen data because it hasn't learned general patterns - only specific ones.

Example 1: A decision tree that keeps splitting until every training point is perfectly classified - even rare or incorrect cases - but then misclassifies real-world examples.
Example 2: A neural network trained on handwriting that memorises individual writing styles instead of learning general character shapes, resulting in poor predictions on new handwriting samples.

Underfitting

Underfitting happens when a model is too simple to capture the structure of the data. It doesn't perform well even on the training set and is unlikely to improve with new data, as it hasn’t learned enough useful relationships.

Example 1: A model used to predict house prices that only considers the number of bedrooms - missing key variables like location, condition, or size.
Example 2: A shallow decision tree that stops after one or two splits, failing to capture the complexity in the dataset.

When to stop training

You stop training when the validation score stops getting better. This prevents overfitting (the model keeps improving on training data but gets worse on new data) and saves time.

How it works

After each training pass (epoch), compute the validation score.
If it’s the best so far, save the model (“best checkpoint”).
If it doesn’t improve for a few epochs in a row (patience, e.g. 3–5), stop and restore the best checkpoint.

Epoch	Train Acc	Val Acc	Note
6	0.90	0.88	new best → save
7	0.92	0.89	new best → save
8	0.94	0.90	new best → save
9	0.96	0.88	no improvement (1)
10	0.97	0.87	no improvement (2) → stop (patience=2)

What to watch: monitor the metric you care about (e.g. higher accuracy is better, lower loss is better). Tiny wobbles can be noise-patience avoids stopping too early.

Key idea: choose hyperparameters using validation, do early stopping using validation, then report the test score once at the end.

Stopping too soon

If you stop while the validation score is still rising, the model hasn’t learned enough yet → underfitting. Use a small patience window so you only stop after improvements flatten or reverse.

What you see over epochs	What it means	What to do
Train ↑, Val ↑	Still learning useful patterns	Keep training
Train ↑, Val ↔ or ↓	Starting to overfit	Stop after patience, keep the best Val model
Train low, Val low	Underfitting (too simple or under-trained)	Train longer, add capacity/features, or tune LR

Rule of thumb: Stop at the highest validation score, not the highest training score. If both curves are still climbing, don’t stop yet.

How to read model scores

Models are judged at different stages. Each score answers a different question:

Stage	What it measures	When used	Typical role
Training score	Performance on the data the model learned from.	During fitting (often every epoch/iteration).	Checks the model can learn patterns at all. Can look great even if it won’t generalise.
Validation score	Performance on a separate set not used to fit.	During development/tuning.	Choose hyperparameters, compare models, decide early stopping.
Test score	One-time check on a final held-out set.	After you’ve picked everything.	Unbiased estimate before deployment. Don’t touch it during tuning.
Production (live) metrics	Real-world performance on live traffic.	After deployment (ongoing).	Monitor drift/bugs, trigger alerts, decide retraining.

Spotting overfitting vs underfitting with scores

Overfitting signal: very high training score, much lower validation score.
Example: Train 98%, Val 78% → memorised training data; weak on new data.
Underfitting signal: both training and validation scores are low.
Example: Train 65%, Val 62% → model too simple or features not informative.
Healthy fit: high training, slightly lower validation (small gap).
Example: Train 93%, Val 90% → learned useful patterns that generalise.

Tip: If validation improves then starts to drop while training keeps rising, stop training at the best validation point (early stopping). Always report the final test score once, after all choices are fixed.

Key Takeaways

Hyperparameter tuning boosts model accuracy and reliability.
Evaluation metrics like accuracy, precision, recall, and F1 score assess performance.
Helps avoid both overfitting and underfitting.
Techniques like grid search and cross-validation support effective tuning.