Artificial Neural Networks

What Are Artificial Neural Networks?

Artificial Neural Networks (ANNs) are computational models inspired by the structure of the human brain. They simulate networks of neurons that learn from data to perform tasks like classification, regression, and pattern recognition.

Each "neuron" processes inputs and passes output to the next layer - forming a powerful system capable of modelling complex patterns.

Structure of a Single Perceptron

A perceptron is the simplest form of a neural unit. It performs a weighted sum of inputs and applies an activation function to decide the output.

Inputs (x₁, x₂, …, xₙ): Data features passed into the neuron.
Weights (w₁, w₂, …, wₙ): Learned parameters indicating the importance of each input.
Bias (b): Shifts the activation function to improve flexibility.
Activation Function: Decides whether the neuron fires.
Output (y): The final result after activation.

Perceptron
Inside: Add the weights (w) of the inputs (x), then add the bias (b)

Multi-Layer Perceptron (MLP)

A multi-layer perceptron (MLP) is an artificial neural network made up of an input layer, multiple hidden layers, and an output layer. It is commonly used in complex tasks like facial recognition, where layered transformations help distinguish unique features of different individuals.

Input Layer: Receives the raw pixel values from a face image (e.g. a 64×64 grayscale image flattened into 4,096 inputs).
Hidden Layers: Extract increasingly abstract features from the image (eg edges → features → face identity).
Output Layer: Classifies the face (e.g. identifies the person from a list of known faces).

Multi-Layer Perceptron
This file is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported license.
This sketch shows an MLP with two hidden layers, each learning progressively abstract features of a face.

What Do the Hidden Layers Learn in Facial Recognition?

Hidden Layer 1: Feature Extraction (Low-Level Features)

Detects edges, corners, and simple shapes like the curve of a jawline or edge of an eye socket. These are the most basic visual patterns in the image.

Hidden Layer 2: Feature Combination (Mid-Level Structures)

Combines low-level features into facial regions such as the shape of an eye, nose, or mouth - learning distinctive arrangements or proportions between features.

Hidden Layer 3: Decision Support (High-Level Identity Encoding)

Encodes the face as a high-dimensional feature vector representing identity. This allows the network to distinguish one individual from another, even if lighting or angles vary.

Training Your Model

Training teaches the network’s weights to make better predictions. Each round follows a simple loop: forward pass → loss → backward pass → update. Repeat over the whole dataset (an epoch), checking validation data to avoid overfitting.

The Flow (One Mini-Batch)

Forward pass: A small batch of inputs goes through the layers to produce predictions.
Loss: Compare predictions with the true labels using a loss function (e.g. cross-entropy for classification).
Backward pass (backpropagation): The network figures out how much each weight contributed to the error.
Update (optimiser): An optimiser (e.g. Adam: Adaptive Moment Estimation) nudges weights to reduce the loss next time.
Repeat: Do this for all batches → that’s 1 epoch. Then check validation performance, adjust if needed.

Forward Pass

Inputs move layer-by-layer. Each neuron applies weights and a bias, then an activation function (e.g. ReLU) before passing values on. The final layer outputs scores/probabilities (e.g. softmax for 10 digits).

Loss Function

Loss measures “how wrong” the predictions are. Lower is better.

Backward Pass (Backprop)

The network works backwards from the loss, attributing error to each layer. This produces gradients (directions for change) for every weight and bias.

Optimiser (Weight Update)

An optimiser (e.g. SGD or Adam) uses gradients to update weights. A key setting is the learning rate: the step size for each update (too big: unstable; too small: slow).

Epochs, Batches, and Iterations

Batch: A small group of examples processed together (e.g. 32). Speeds up training and stabilises gradients.
Iteration (step): One update using one batch.
Epoch: One full pass through the entire training set (many iterations).

Why this matters: You’ll often see settings like “batch_size=32, epochs=5”. That means “update weights per 32 examples; repeat until the model has seen the whole dataset 5 times”.

Validation & Early Stopping

Keep a separate validation set to check generalisation during training. If validation loss stops improving (or gets worse) while training loss still drops, you’re likely overfitting. Early stopping halts training when validation performance plateaus for a few epochs.

Tip: Save the best model weights based on validation score.

Quick Reference (Typical Settings)

Item	Plain Meaning	Typical Choice
Batch size	How many examples per update	32 or 64
Epochs	Full passes over training set	5–20 (watch validation)
Learning rate	Update step size	0.001 (Adam)
Loss (multi-class)	Probability error	Cross-entropy
Metrics	What you report	Accuracy (for classification)

Why these rows? They’re the knobs you’ll set most often in class and projects; sensible defaults keep you productive.

Note: Items like learning rate, batch size, and layer sizes are hyperparameters. Loss/metrics and split/normalisation choices are training/reporting settings.

Common Activation Functions

Activation functions add non-linearity, letting neural networks learn patterns that straight lines cannot. Different activations change how signals flow through a model, so choosing the right one affects speed, stability, and accuracy.

ReLU (Rectified Linear Unit)

ReLU outputs zero for negative inputs and passes positive values through unchanged. This simple rule helps deep networks train quickly because useful gradients flow without getting squeezed too much. It’s the most common choice for hidden layers in CNNs and many feed-forward nets. In practice: start with ReLU in hidden layers unless you have a strong reason not to.

Sigmoid

Sigmoid squashes any input to a value between 0 and 1, which reads naturally as a probability. Because it compresses strongly near 0 and 1, gradients can become tiny, so it’s usually not used in deep hidden layers today. Its sweet spot is the output layer for binary decisions (e.g. spam vs not-spam) or for multi-label problems where each class is an independent yes/no. Think of sigmoid as a “probability dial” for single, independent outcomes.

Softmax

Softmax turns a list of raw scores into a set of probabilities that add up to 1. It’s the standard choice in the output layer for multi-class classification where exactly one class should be selected (e.g. digit 0–9). The function highlights the largest score while still giving every class a probability. You can view softmax as a multi-category cousin of sigmoid: sigmoid handles one independent label, while softmax helps the model pick one best class among many. Use it whenever your task is “pick exactly one.”

Python Project: Train an ANN to Classify Handwritten Digits

There are two scripts and three sample image files. First, train on MNIST. Then test with your own image (e.g. digit.png).

Scenario: Each 28×28 image (flattened to 784 values) is classified into one of 10 digits (0–9).

Training Your ANN (MNIST)

Place all files in the same folder.

                    # TRAINING

                    # Load core modules

                    from tensorflow.keras.models import Sequential

                    from tensorflow.keras.layers import Input, Dense

                    # Load data and preprocessing tools

                    from tensorflow.keras.datasets import mnist

                    from tensorflow.keras.utils import to_categorical

                    # Step 1: Load the dataset (60,000 training and 10,000 test images)

                    (x_train, y_train), (x_test, y_test) = mnist.load_data()

                    # Step 2: Preprocess data

                    # Flatten 28×28 images to 784-element vectors

                    x_train = x_train.reshape(-1, 784).astype("float32") / 255

                    x_test = x_test.reshape(-1, 784).astype("float32") / 255

                    # Convert labels to one-hot encoded format

                    y_train = to_categorical(y_train, 10)

                    y_test = to_categorical(y_test, 10)

                    # Step 3: Build the ANN model

                    model = Sequential([

                        # Input layer: 784 inputs (28×28 image)

                        Input(shape=(784,)),

                        # First hidden layer: learns edges/strokes

                        # ReLU activation function

                        Dense(64, activation='relu'),

                        # Second hidden layer: combines features

                        Dense(32, activation='relu'),

                        # Output layer: 10 digits

                        # Softmax activation function

                        Dense(10, activation='softmax')

                    ])

                    # Step 4: Compile the model

                        # Optimiser: ADAM

                    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

                    # Step 5: Train (5 epochs)

                    model.fit(x_train, y_train, epochs=5, batch_size=32, validation_split=0.2)

                    # Step 6: Evaluate on test data

                    loss, accuracy = model.evaluate(x_test, y_test)

                    print("Test Accuracy:", accuracy)

                    # Save the trained model

                    model.save("my_digit_model.keras")

Testing Your ANN (Your Own Image)

Draw a digit, save as a 28×28 grayscale PNG with a dark digit on a white background (e.g. digit.png), then run:

                    # TESTING

                    from tensorflow.keras.models import load_model

                    # Load and preprocess custom digit image

                    from PIL import Image

                    import numpy as np

                    # Load your trained model from script 1

                    model = load_model("my_digit_model.keras")

                    # Use Your Own Image (e.g. digit.png)

                    img = Image.open("digit.png").convert("L")  # Grayscale

                    img = img.resize((28, 28))                  # Resize to 28x28

                    img_array = np.array(img)

                    img_array = 255 - img_array                 # Invert: dark digit on light bg

                    img_array = img_array / 255.0               # Normalise pixel values

                    img_flat = img_array.reshape(1, 784)        # Flatten to shape (1, 784)

                    prediction = model.predict(img_flat)

                    predicted_digit = prediction.argmax()

                    print("Predicted digit:", predicted_digit)

Tip: Center the digit; use a thick dark stroke on a clean white background.

Sample Images (28×28)

Place these in the same folder as your scripts.

If your own image fails, compare brightness/contrast and size with these samples.

Real-World Applications of Artificial Neural Networks

Image Recognition

Used in tasks like face detection on smartphones.

Input Layer: Pixel data from the image (e.g. 28×28 or 224×224 RGB).
Hidden Layers:
- Early layers detect edges and textures (e.g. lines, corners).
- Deeper layers learn facial features or object shapes.
Output Layer: Probability scores for classes (e.g. face detected / not detected).

Speech Recognition

Used by assistants like Siri or Alexa to convert speech to text.

Input Layer: Audio features like MFCCs (Mel-frequency cepstral coefficients).
Hidden Layers:
- Recurrent or convolutional layers detect speech patterns and phonemes.
- Later layers model words or phrases from temporal patterns.
Output Layer: Predicted words or characters from spoken input.

Medical Imaging

Supports radiologists by detecting signs of illness in X-rays or MRIs.

Input Layer: Medical scan images (e.g. grayscale pixel arrays).
Hidden Layers:
- Initial layers pick up tissue texture and structural features.
- Deep layers highlight anomalies (e.g. tumours, blockages).
Output Layer: Diagnostic labels or risk scores (e.g. “tumour likely”).

Autonomous Vehicles

Neural networks help self-driving cars interpret their surroundings.

Input Layer: Visual feeds, lidar scans, or radar data.
Hidden Layers:
- Image layers detect road edges, signs, pedestrians, and other vehicles.
- Fusion layers integrate sensor types to understand the driving context.
Output Layer: Control signals for actions like braking, turning, or accelerating.

Challenges in Training ANNs

Overfitting: The model memorises training data rather than generalising.
Vanishing Gradient: Gradient updates become too small for deep networks.
Training Time: High computational cost, especially with large data or deep networks.

Key Takeaways

ANNs simulate how neurons in the brain process information to perform classification and pattern recognition.
A perceptron models a single decision-making unit with inputs, weights, bias, and activation.
MLPs stack multiple layers to detect and learn from complex data relationships.
Activation functions enable non-linear learning essential for complex tasks.