Neural Network Architectures and Learning Algorithms

Posted on Jun 17, 2026 in Civil Engineering Technology

Convolutional Neural Networks (CNN)

CNN is a deep learning model mainly used for processing image data. It automatically extracts features such as edges, textures, and shapes from images.

CNN Architecture

Input Image
|
Convolution Layer
|
Activation (ReLU)
|
Pooling Layer
|
Convolution + Pooling
|
Fully Connected Layer
|
Output Layer

Working of CNN

  1. Input Layer: Receives the image.
  2. Convolution Layer: Extracts features using filters.
  3. ReLU Layer: Introduces non-linearity.
  4. Pooling Layer: Reduces image dimensions.
  5. Fully Connected Layer: Performs classification.
  6. Output Layer: Gives final prediction.

Applications of CNN

  • Image Classification
  • Face Recognition
  • Object Detection
  • Medical Image Analysis
  • Self-driving Cars
  • Handwritten Digit Recognition
  • Video Analysis

LeNet Architecture

LeNet is one of the earliest CNN architectures developed by Yann LeCun for handwritten digit recognition.

LeNet Diagram

Input Image (32×32)
|
Conv1 (6 filters)
|
Average Pooling
|
Conv2 (16 filters)
|
Average Pooling
|
Fully Connected Layer
|
Fully Connected Layer
|
Output (10 Classes)

Features of LeNet

  • Uses convolution and pooling layers.
  • Reduces image size gradually.
  • Classifies digits from 0–9.
  • Simple and efficient architecture.

Recurrent Neural Networks (RNN)

RNN is a neural network designed for sequential data where previous outputs influence future outputs.

Features

  • Has memory of previous inputs.
  • Suitable for sequence prediction.
  • Processes data step by step.

Bidirectional RNN (Bi-RNN)

A Bidirectional RNN processes information in both forward and backward directions.

Bi-RNN Diagram

Forward RNN —>

Input Sequence
x1 -> x2 -> x3 -> x4

Backward RNN <—

Outputs Combined
|
Final Output

Long Short-Term Memory (LSTM)

LSTM is a special type of RNN that solves the vanishing gradient problem and can learn long-term dependencies.

LSTM Components

  1. Forget Gate: Removes unnecessary information.
  2. Input Gate: Adds new information.
  3. Cell State: Stores long-term memory.
  4. Output Gate: Produces output.

Activation Functions in LSTM

  • Sigmoid Function: Output range 0 to 1; used in gates to control information flow.
  • Tanh Function: Output range -1 to 1; used for updating cell state and normalizing values.

Deep Recurrent Networks

Deep RNN is an extension of RNN that contains multiple hidden recurrent layers stacked one above another.

Features

  • Learns complex sequential patterns.
  • Greater representation power.
  • Better accuracy than simple RNN.

Artificial Neural Networks (ANN)

ANN is a computational model inspired by the structure and working of the human brain. It consists of interconnected neurons that process information and learn patterns from data.

Structure of ANN

Input Layer → Hidden Layer(s) → Output Layer

Perceptron Learning Algorithm

A perceptron is the simplest form of neural network used for binary classification.

Algorithm Steps

  1. Initialize weights and bias randomly.
  2. Provide training input and desired output.
  3. Calculate output:
    wSaTDJmpm8fAAAAAElFTkSuQmCC
  4. Compare actual output with desired output.
  5. Compute error: Error = Target − Output
  6. Update weights:
    AAAAAElFTkSuQmCC
  7. Repeat until error becomes minimum.

Backpropagation

Backpropagation is a supervised learning algorithm used to train multilayer neural networks by minimizing error.

Steps

  1. Initialize Weights: Assign small random values.
  2. Forward Propagation: Input data passes through layers.
  3. Calculate Error: Error = Target Output − Actual Output.
  4. Compute Gradients: Determine contribution of each weight to the error.
  5. Backward Propagation: Error is propagated from output to hidden layers.
  6. Update Weights: Adjusted using gradient descent.

Feed Forward Neural Network (FFNN)

FFNN is the simplest neural network where information flows only in one direction—from input layer to output layer.

Activation Functions

Sigmoid Function

Maps values between 0 and 1.
l4mIzrE9A1+ASJN0k57htURAAAAAElFTkSuQmCC

Tanh Function

Maps values between -1 and 1.
gJIpKKWYGQcrZPQnvDIKTKCqlmP8DGeZ2eBioi8MAAAAASUVORK5CYII=

ReLU (Rectified Linear Unit)

AAWAW5ZkZ99LAAAAAElFTkSuQmCC

Softmax Function

Used in multi-class classification.
wNogeF756mBfwAAAABJRU5ErkJggg==

Regularization Techniques

Regularization is a technique used to reduce overfitting and improve the generalization ability of a neural network.

Types of Regularization

  • L1 Regularization (Lasso): Adds absolute values of weights to the loss function.
  • L2 Regularization (Ridge): Adds squared values of weights to the loss function.
  • Dropout: Randomly disables neurons during training.
  • Early Stopping: Stops training when validation error starts increasing.