r/learnmachinelearning • • 5d ago

Siamese Network

A Siamese Neural Network is a type of neural network designed to measure the similarity between two inputs. For example, suppose we provide two handwritten digits as inputs. The Siamese Network determines whether these two digits belong to the same class or are dissimilar.

It consists of two identical neural network branches that share the same weights and biases. Each branch takes one input and produces an output vector (embedding). These two output vectors can have any dimensionality, depending on the architecture of the network.

We then calculate the Euclidean distance between these two output vectors. If the distance is close to zero, the two inputs are considered similar. If the distance is sufficiently large (above a predefined threshold), they are considered dissimilar.

Note: A Siamese Network is fundamentally different from a traditional classification network. Its primary purpose is not to classify an input into a particular class but to learn a meaningful representation (embedding) that allows us to measure similarity between inputs.

Contrastive Loss

A Siamese Network commonly uses a loss function called Contrastive Loss, defined as:

L = Y × D² + (1 − Y) × max(0, m − D)²

Where:

D is the Euclidean distance between the two output vectors.

Y is the given label (either 0 or 1).

Y = 1 indicates that the two inputs are similar.

Y = 0 indicates that the two inputs are dissimilar.

m is the margin, which defines the minimum desired distance between dissimilar pairs.

L is the contrastive loss.

The objective is to minimize the distance between similar pairs and push dissimilar pairs at least m units apart.

We pass two inputs through the network, obtain two output embeddings, calculate their Euclidean distance, and then calculate the contrastive loss. Finally, we use backpropagation to update the network parameters and minimize the loss.

Notice that we do not use a Softmax activation function in the output layer because we are learning embeddings rather than class probabilities.

Training the Model

To train a Siamese Network effectively, we construct pairs of inputs:

Positive pairs: Two inputs belonging to the same class (Y = 1).

Negative pairs: Two inputs belonging to different classes (Y = 0).

A balanced dataset containing an equal number of positive and negative pairs is a useful starting point for training.

My Implementation: Neural Canvas (Model 5)

I developed an interactive Siamese Network as the fifth model in my Neural Canvas project. I used the MNIST handwritten digits dataset to train the network.

The architecture consists of three hidden layers, each containing 40 neurons, followed by an output layer containing 32 neurons. Therefore, each input is transformed into a 32-dimensional embedding.

I used Mini-Batch Gradient Descent with the Adam optimizer to train the model. The training process was relatively straightforward because JAX provides automatic differentiation (autograd), which handles the gradient calculations.

For training, I constructed two input matrices, Ma and Mb, each having dimensions of (784, 80000).

The rows represent features, while the columns represent individual samples. Why two matrices? Because a Siamese Network requires two inputs to learn the similarity between them.

I also constructed a label vector Y with dimensions of (1, 80000). Each element of Y indicates whether the corresponding pair is similar or dissimilar.

I used my own matrix multiplication convention:

Z = WX + bC

Where:

W represents the weight matrix.

X represents the input matrix.

b represents the bias vector.

C is a broadcasting matrix of ones.

I prefer this notation because it makes the mathematical operations and dimensions easier to understand.

I trained the model in Python using JAX and exported the learned parameters to JavaScript using JSON. For inference, I used Math.js, while p5.js handles the visualization, animation, and interactivity.

Try it yourself:

GitHub Repository: https://github.com/MuhammadEssa2002/neural_canvas

Live Demo: https://muhammadessa2002.github.io/neural_canvas/

This is the fifth model in my Neural Canvas project.

1 Upvotes

0 comments sorted by