Classificaton Techniques
What Is Classification?
Classification is a type of supervised learning used to predict discrete categories. It analyses patterns in labelled data and uses these to assign new, unseen data to predefined classes.
For example, a classification model might label emails as "spam" or "not spam" based on previously seen examples.
K-Nearest Neighbours (K-NN)
K-NN is a simple algorithm that classifies a new data point based on the majority class of its k (how many) nearest neighbours in the training data.
Sample
In a Book Recommendation system, if three of your closest neighbours like science fiction books, the system might recommend similar books to you too.
Positioning the New Member
Before the algorithm checks its neighbours, it first places the new data point on a (conceptual) graph using its features (e.g. weight and texture, or age and preferences). This makes it easy to measure how close it is to other data points - just like dropping a pin on a map and looking at k nearby pins.
Choosing the Right k Value
- Low
k(e.g. 1 or 3): Makes predictions based on just a few neighbours. It can be very responsive to local patterns but also sensitive to noise or outliers (e.g. one odd data point could skew the result). - High
k(e.g. 10 or 20): Takes a broader view by averaging over more neighbours. This reduces the effect of noise but may blur distinct group boundaries, making the model less sensitive to small but meaningful differences.
In practice, finding the best k value is part of hyperparameter tuning - testing different values to see which gives the most accurate results.
K-NN: Key Characteristics
| Aspect | Description |
|---|---|
| Model Type | Uses the entire training set during prediction - no generalised model is built in advance. |
| Prediction Rule | Classifies new points based on the majority class among the k nearest neighbours. |
| Efficiency | Simple to implement but can become slow as the dataset grows. |
K-NN Worked Example: Mammals
We’ll classify a new mammal as Herbivore or Carnivore using two features: Weight (kg) and Ear length (cm). We'll choose k = 3.
Training Data (5 mammals)
| Name | Weight (kg) | Ear Length (cm) | Class |
|---|---|---|---|
| Rabbit | 2 | 7 | Herbivore |
| Fox | 7 | 8 | Carnivore |
| Deer | 90 | 12 | Herbivore |
| Wolf | 45 | 10 | Carnivore |
| Kangaroo | 55 | 11 | Herbivore |
New Mammal (to classify)
| Weight (kg) | Ear Length (cm) |
|---|---|
| 6 | 7 |
Distance: use straight-line distance on the two features (like measuring with a ruler in a 2-D grid).
How we get the distance numbers
K-NN needs a way to measure “closeness.” We’ll use straight-line (Euclidean) distance on two features: Weight and Ear length. For a training animal with values (wt, et) and our new animal (wn, en) = (6, 7):
Two worked examples
√((6 − 7)² + (7 − 8)²) = √(1 + 1) = √2 ≈ 1.41
Rabbit (2 kg, 7 cm):
√((6 − 2)² + (7 − 7)²) = √(16 + 0) = √16 = 4.00
Repeat for the remaining animals
| Neighbour | ΔWeight | ΔEar | (ΔW)²+(ΔE)² | Distance |
|---|---|---|---|---|
| Fox | −1 | −1 | 2 | 1.41 |
| Rabbit | +4 | 0 | 16 | 4.00 |
| Wolf | −39 | −3 | 1530 | 39.12 |
| Kangaroo | −49 | −4 | 2417 | 49.16 |
| Deer | −84 | −5 | 7081 | 84.15 |
Why this matters: the nearest points vote on the class. Because Weight (kg) is much larger in scale than Ear length (cm), it can dominate distance; in real projects, we often standardise features so each contributes fairly.
Step 2 - Take the closest k = 3 neighbours
- #1 Fox (Carnivore)
- #2 Rabbit (Herbivore)
- #3 Wolf (Carnivore)
Step 3 - Majority vote
Prediction: Carnivore (2 votes Carnivore vs 1 vote Herbivore).
Note: Features on very different scales (e.g. weight vs ear length) can dominate the distance. In practice, it helps to standardise features so each contributes fairly. You can also try different k values and compare results on a validation set.
Decision Trees
Decision trees use a flowchart-like structure where each node splits the data based on a feature value, guiding the decision path until a prediction is made.
Example
A veterinary decision tree might begin by asking whether the animal lives in water or on land. If the answer is “land,” the next node might ask whether it has a long neck. Based on the answers, the model could classify the animal as a giraffe, horse, or mouse. Each “yes” or “no” answer moves the decision process down a different branch.
How a Decision Tree Makes Predictions
A decision tree asks a series of yes/no questions based on the data’s features. It starts at the root node and follows the appropriate branches depending on the answers. Eventually, it reaches a leaf node, which gives the final prediction.
Each split aims to:
- Separate the data in a way that increases “purity” - meaning each group contains mostly one category.
- Ask the most useful question first - the one that best divides the data into meaningful groups.
Decision Trees: Advantages and Drawbacks
| Advantages | Drawbacks |
|---|---|
| Trees are easy to understand and visualise. Each question builds toward a logical decision, making the model transparent. | If the tree grows too deep, it can overfit - learning noise instead of useful patterns. Techniques like pruning or limiting the depth can help keep the tree general and accurate. |
In practice: Decision trees are often tuned just like K-NN - by adjusting settings like maximum depth, minimum samples per leaf, or splitting criteria to improve accuracy without overfitting.
Decision Trees: Key Characteristics
| Aspect | Description |
|---|---|
| Prediction Method | Uses a sequence of if-then decision rules to classify data points. |
| Interpretability | Highly visual and easy for humans to follow and understand. |
| Model Risk | Can overfit the training data - especially if the tree is deep. Pruning helps control this. |
Example: K-NN in Python
Scenario: Classifying a fruit as apple or orange based on weight and texture.
from sklearn.neighbors import KNeighborsClassifierX = [[150, 0], [170, 0], [140, 1], [130, 1]]
y = ["Apple", "Apple", "Orange", "Orange"]
knn =
KNeighborsClassifier(n_neighbors=3)knn.
fit(X, y)prediction = knn.
predict([[160, 0]])print("Predicted Category:", prediction)
Example: Decision Tree in Python
Scenario: Diagnosing an illness based on symptoms.
from sklearn.tree import DecisionTreeClassifierX = [[1, 1], [1, 0], [0, 1], [0, 0]]
y = ["Flu", "Cold", "Flu", "Healthy"]
clf =
DecisionTreeClassifier()clf.
fit(X, y)prediction = clf.
predict([[1, 1]])print("Predicted Diagnosis:", prediction)
Real-World Applications of Classification
Healthcare
Classification models help diagnose diseases based on symptoms, test results, and patient history. For example, decision trees can suggest possible conditions based on patient answers to a series of yes/no questions.
Finance
Classification is used to detect fraud by identifying patterns that suggest unusual behaviour. For example, a model can classify a credit card transaction as "fraud" or "legitimate" based on features like transaction amount and location.
E-Commerce
Recommendation systems often use K-NN to classify customer preferences and suggest products based on similarities to other users' buying habits.
Email Filtering
Spam filters use classification to distinguish spam from genuine emails based on keywords, sender history, and message structure.
Limitations of Classification Techniques
- K-NN: Becomes slow with large datasets and is sensitive to irrelevant features.
- Decision Trees: Can overfit unless pruned effectively.
- Training Data Quality: Biased or incomplete data can lead to inaccurate predictions.
Key Takeaways
- Classification predicts categorical outcomes using labelled data.
- K-NN makes decisions based on nearby training examples.
- Decision Trees follow logical decision paths to classify data.
- These techniques are widely used in healthcare, finance, retail, and communication systems.