Understanding ML: Introduction

Last Updated: July 18, 2026

Probabilistic Machine Learning: An Introduction by Kevin P. Murphy

Contact

• LinkedIn: Natan Asrat
• Gmail: nathanyilmaasrat@gmail.com
• Telegram: Natan Asrat
• X: Natan Asrat Yilma
• Youtube: Natville

What is Machine Learning

The definition of ML, according to Tom Mitchell, is as follows:

A computer program is said to learn from experience E with respect to some class of tasks T, and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Supervised Learning

The task $T$ is to learn a mapping $f$ from inputs $x \in X$ to outputs $y \in Y$. The inputs $x$ are also called the features, covariates, or predictor. The output $y$ is also known as the label, target, or response.

The experience $E$ is given in the form of a set of $N$ input-output pairs $\mathcal{D} = \{(x_n, y_n)\}_{n=1}^N$, known as the training set (where $N$ is called the sample size). The performance measure P depends on the type of output we are predicting.

Classification

The output space is a set of $C$ unordered and mutually exclusive labels known as classes, $\mathcal{Y} = \{1, 2, \dots, C\}$.

The problem of predicting the class label given an input is also called pattern recognition. If there are just two classes, often denoted by $y \in \{0, 1\} or y \in \{-1, +1\}$, it is called binary classification.

When we have small datasets of features, it is common to store them in an $N \times D$ matrix, in which each row represents an example, and each column represents a feature. This is known as a design matrix.

Exploratory Data Analysis

Before tackling a problem with ML, it is usually a good idea to perform exploratory data analysis, to see if there are any obvious patterns (which might give hints on what method to choose), or any obvious problems with the data (e.g., label noise or outliers).

For tabular data with a small number of features, it is common to make a pair plot. For higher-dimensional data, it is common to first perform dimensionality reduction, and then to visualize the data in 2d or 3d.

Learning a Classifier

Classifiers divide data into different regions using a decision boundary. A decision boundary is the “line in the sand” that a classifier draws to separate different classes of data. Decision Trees are built by repeatedly splitting these regions with “if/then” rules to fix classification errors.

Tree Parameters ($\theta$) are simply the specific features and cutoff values (thresholds) used to make those splits.

Empirical Risk Minimization

Empirical Risk Minimization (ERM): A training method where the goal is to find parameters $\theta$ that minimize the average loss on the training set.

Loss Function ($\ell$): Measures the cost of a prediction error.

• Zero-One Loss ($\ell_{01}$): Treats all mistakes equally (returns 1 for any incorrect prediction, 0 for correct): $\ell_{01}(y^*, \hat{y}) = \mathbb{I}(y^* \ne \hat{y})$. The average of this loss is the misclassification rate.
• Asymmetric Loss: Assigns different costs to different types of errors (e.g., misclassifying a poisonous flower as edible is penalized more heavily).

Empirical Risk ($L(\theta)$): The average loss of the model across all $N$ training examples.

While ERM minimizes training loss, the true objective is generalization (minimizing expected loss on unseen future data).

Uncertainty

In many cases, we will not be able to perfectly predict the exact output given the input, due to lack of knowledge of the input-output mapping (this is called epistemic uncertainty or model uncertainty), and/or due to intrinsic (irreducible) stochasticity in the mapping (this is called aleatoric uncertainty or data uncertainty).

Conditional Probability: To capture model uncertainty, we predict a probability distribution over the $C$ possible classes: $p(y = c \mid x; \theta) = f_c(x; \theta)​$.

Logits and Softmax: Instead of forcing the model $f(x; \theta)$ to output valid probabilities directly (which must be between 0 and 1, and sum to 1), we let it output raw, unnormalized log-probabilities called logits ($a$). We then pass these through the softmax function to convert them into a valid probability distribution: $p(y = c \mid x; \theta) = \operatorname{softmax}_c(f(x; \theta))$.

import tensorflow as tf

def softmax_tf(logits_input):
    return tf.nn.softmax(tf.constant(logits_input, dtype=tf.float32))

# N = 2 samples, C = 3 possible events/classes (Raw, unnormalized scores)
logits = [
    [1.0, 2.0, 3.0],  # Sample 1 raw scores
    [2.0, 4.0, 2.0]   # Sample 2 raw scores
]

probabilities = softmax_tf(logits)

print("TF Softmax Probabilities:\n", probabilities.numpy())
# Output: 
# TF Softmax Probabilities:
# [[0.09003057 0.24472848 0.66524094]
# [0.10650697 0.786986   0.10650697]]

import torch

def softmax_torch(logits_input):
    logits_tensor = torch.tensor(logits_input, dtype=torch.float32)
    # dim=-1 explicitly specifies softmax over the final class columns axis
    return torch.nn.functional.softmax(logits_tensor, dim=-1)

# N = 2 samples, C = 3 possible events/classes (Raw, unnormalized scores)
logits = [
    [1.0, 2.0, 3.0],
    [2.0, 4.0, 2.0]
]

probabilities = softmax_torch(logits)

print("PyTorch Softmax Probabilities:\n", probabilities.numpy())
# Output: 
# PyTorch Softmax Probabilities:
# [[0.09003057 0.24472848 0.66524094]
# [0.10650697 0.786986   0.10650697]]

import numpy as np

def softmax_numpy(logits_input):
    logits_arr = np.asarray(logits_input)
    
    # Subtracting the max value per row prevents system numerical overflow errors
    stable_logits = logits_arr - np.max(logits_arr, axis=-1, keepdims=True)
    
    exponentiated = np.exp(stable_logits)
    return exponentiated / np.sum(exponentiated, axis=-1, keepdims=True)

# N = 2 samples, C = 3 possible events/classes (Raw, unnormalized scores)
logits = [
    [1.0, 2.0, 3.0],
    [2.0, 4.0, 2.0]
]

probabilities = softmax_numpy(logits)

print("NP Softmax Probabilities:\n", probabilities)
# Output: 
# NP Softmax Probabilities:
# [[0.09003057 0.24472847 0.66524096]
# [0.10650698 0.78698604 0.10650698]]

p(y = c \mid x; \theta) = \operatorname{softmax}_c(f(x; \theta))

Logistic Regression: A special case where $f$ is an affine function. In Machine Learning, the parameters $\theta = (b, w)$ are called the bias ($b$) and the weights $w: f(x; \theta) = b + w^T x​$.

Notation Simplification: To simplify formulas, we can absorb the bias $b$ into the weights vector $w$ by prepending a 1 to our input vector $x$ (making it $\tilde{x} = [1, x_1, \dots, X_D]^T$. This turns the affine function into a simple linear dot product: $f(x; w) = w^T x$.

Maximum Likelihood Estimation

Negative Log Probability (Loss Function): When using probabilistic models, the loss function is typically defined as the negative log of the predicted probability of the true label: $\ell(y, f(x; \theta)) = -\log p(y \mid f(x; \theta))$.

Negative Log Likelihood (NLL): The empirical risk (average loss) of this model over the entire training set of $N$ examples:

import tensorflow as tf

def nll_logits_tf(y_true, logits):
    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(
        from_logits=True,
        reduction=tf.keras.losses.Reduction.NONE
    )

    return loss_fn(
        tf.convert_to_tensor(y_true),
        tf.convert_to_tensor(logits, dtype=tf.float32)
    )


# N = 2 samples, C = 3 classes
y_true = [1, 2]

# Raw logits (not probabilities)
logits = [
    [1.0, 2.5, 1.2],  # predicts class 1
    [0.8, 0.5, 2.0]   # predicts class 2
]

loss_per_sample = nll_logits_tf(y_true, logits)
total_loss = tf.reduce_mean(loss_per_sample)

print("TF NLL per sample:", loss_per_sample.numpy())
print("TF NLL (Final Average):", total_loss.numpy())
# Output: 
# TF NLL per sample: [0.4025689  0.42155132]
# TF NLL (Final Average): 0.4120601

import torch

def nll_logits_torch(y_true, logits):
    y_true_tensor = torch.tensor(y_true, dtype=torch.long)
    logits_tensor = torch.tensor(logits, dtype=torch.float32)

    loss_individual = torch.nn.functional.cross_entropy(
        logits_tensor,
        y_true_tensor,
        reduction="none"
    )

    return loss_individual


# N = 2 samples, C = 3 classes
y_true = [1, 2]

# Raw logits
logits = [
    [1.0, 2.5, 1.2],
    [0.8, 0.5, 2.0]
]

loss_per_sample = nll_logits_torch(y_true, logits)
total_loss = torch.mean(loss_per_sample)

print("PyTorch NLL per sample:", loss_per_sample.numpy())
print("PyTorch NLL (Final Average):", total_loss.item())
# Output: 
# PyTorch NLL per sample: [0.4025689  0.42155132]
# PyTorch NLL (Final Average): 0.41206011176109314

import numpy as np

def nll_logits_numpy(y_true, logits):
    y_true_arr = np.asarray(y_true)
    logits_arr = np.asarray(logits)

    # Stable log-softmax
    logits_shifted = logits_arr - np.max(
        logits_arr,
        axis=-1,
        keepdims=True
    )

    log_probs = (
        logits_shifted
        - np.log(np.sum(np.exp(logits_shifted), axis=-1, keepdims=True))
    )

    # Pick log probability of true class
    loss_individual = -log_probs[
        np.arange(len(y_true_arr)),
        y_true_arr
    ]

    return loss_individual


y_true = [1, 2]

logits = [
    [1.0, 2.5, 1.2],
    [0.8, 0.5, 2.0]
]

loss_per_sample = nll_logits_numpy(y_true, logits)
total_loss = np.mean(loss_per_sample)

print("NP NLL per sample:", loss_per_sample)
print("NP NLL (Final Average):", total_loss)
# Output: 
# NP NLL per sample: [0.40256889 0.42155128]
# NP NLL (Final Average): 0.41206008203499644

\text{NLL}(\theta) = -\frac{1}{N} \sum_{n=1}^N \log p(y_n \mid f(x_n; \theta))

When the data contains only two classes ($y \in \{0,1\}$):

Because $y_n$ must be either 0 or 1, one half of the equation cancels out completely for every sample, isolating the log probability of the correct target.

import tensorflow as tf

def bce_logits_tf(y_true, logits_pred):
    y_true_tensor = tf.convert_to_tensor(y_true, dtype=tf.float32)
    logits_tensor = tf.convert_to_tensor(logits_pred, dtype=tf.float32)

    bce_loss = tf.keras.losses.BinaryCrossentropy(
        from_logits=True,
        reduction=tf.keras.losses.Reduction.NONE
    )

    return bce_loss(y_true_tensor, logits_tensor)


# N = 2 separate samples
y_true = [[1.0], [1.0]]

# Raw logits (not probabilities)
logits_pred = [[1.386], [2.197]]

loss_per_sample = bce_logits_tf(y_true, logits_pred)
total_loss = tf.reduce_mean(loss_per_sample)

print("TF BCE (Logits) per sample:", loss_per_sample.numpy())
print("TF BCE (Logits) (Final Average):", total_loss.numpy())
# Output: 
# TF BCE (Logits) per sample: [0.2232024  0.10538297]
# TF BCE (Logits) (Final Average): 0.1642927

import torch

def bce_logits_torch(y_true, logits_pred):
    y_true_tensor = torch.tensor(y_true, dtype=torch.float32)
    logits_tensor = torch.tensor(logits_pred, dtype=torch.float32)

    bce_loss = torch.nn.BCEWithLogitsLoss(
        reduction="none"
    )

    return bce_loss(logits_tensor, y_true_tensor)


# N = 2 separate samples
y_true = [[1.0], [1.0]]

# Raw logits (not probabilities)
logits_pred = [[1.386], [2.197]]

loss_per_sample = bce_logits_torch(y_true, logits_pred)
total_loss = torch.mean(loss_per_sample)

print("PyTorch BCE (Logits) per sample:", loss_per_sample.numpy())
print("PyTorch BCE (Logits) (Final Average):", total_loss.numpy())

# Output: 
# PyTorch BCE (Logits) per sample: [[0.22320242]
# [0.10538297]]
# PyTorch BCE (Logits) (Final Average): 0.1642927

import numpy as np

def bce_logits_numpy(y_true, logits_pred):
    y_true_arr = np.asarray(y_true, dtype=np.float32)
    logits_arr = np.asarray(logits_pred, dtype=np.float32)

    bce_loss = np.maximum(logits_arr, 0) - (y_true_arr * logits_arr) + np.log1p(
        np.exp(-np.abs(logits_arr))
    )

    return bce_loss


# N = 2 separate samples
y_true = [[1.0], [1.0]]

# Raw logits (not probabilities)
logits_pred = [[1.386], [2.197]]

loss_per_sample = bce_logits_numpy(y_true, logits_pred)
total_loss = np.mean(loss_per_sample)

print("NP BCE (Logits) per sample:", loss_per_sample.squeeze())
print("NP BCE (Logits) (Final Average):", total_loss)

# Output: 
# NP BCE (Logits) per sample: [0.2232024  0.10538296]
# NP BCE (Logits) (Final Average): 0.1642927

\mathcal{L}=-\frac{1}{N}\sum_{n=1}^N\left[ y_n \log(p_n) + (1-y_n)\log(1-p_n)\right]

Maximum Likelihood Estimate (MLE): The specific parameter settings $\hat{\theta}_{\text{mle}}$ found by minimizing the NLL: $\hat{\theta}_{\text{mle}} = \arg\min\limits_{\theta} \text{NLL}(\theta)$.

Regression

Now suppose that we want to predict a real-valued quantity $y \in \mathbb{R}$ instead of a class label $y \in \{1, \dots, C\}$; this is known as regression.

For regression, the most common choice is to use quadratic loss, or $\ell_2$ loss: $\ell_2(y, \hat{y}) = (y – \hat{y})^2$.

The empirical risk when using quadratic loss is equal to the mean squared error or MSE:

import tensorflow as tf

def empirical_risk_tf(y_true, y_pred):
    return tf.keras.losses.MSE(y_true, y_pred)

print("TF MSE: ", empirical_risk_tf([2, 1], [4, 5]))
# Output: TF MSE:  tf.Tensor(10.0, shape=(), dtype=float32)

import torch

def empirical_risk_torch(y_true, y_pred):
    return torch.nn.functional.mse_loss(y_true, y_pred)

y_true = torch.tensor([2.0, 1.0])
y_pred = torch.tensor([4.0, 5.0])

print("PyTorch MSE: ", empirical_risk_torch(y_true, y_pred).item())
# Output: PyTorch MSE:  10.0

import numpy as np

def empirical_risk_numpy(y_true, y_pred):
    return np.mean(np.square(np.subtract(y_true, y_pred)))

print("NP MSE:", empirical_risk_numpy([2, 1], [4, 5])) 
# Output: NP MSE: 10.0

\text{MSE}(\theta) = \frac{1}{N} \sum_{n=1}^N (y_n - f(x_n; \theta))^2

In regression problems, it is common to assume the output distribution is a Gaussian or normal:

In the context of regression, we can make the mean depend on the inputs by defining $\mu = f(x_n; θ)$. We therefore get the following conditional probability distribution:

If we assume that the variance $\sigma^2$ is fixed (for simplicity), the corresponding average (per-sample) negative log likelihood becomes:

We see that the $\text{NLL}$ is proportional to the $\text{MSE}$. Hence computing the maximum likelihood estimate of the parameters will result in minimizing the squared error, which seems like a sensible approach to model fitting.

Linear Regression

We can fit a 1D data using a simple linear regression model of the form: $f(x;\theta) = b + wx$, where $w$ is the slope, $b$ is the offset, and $\theta = (w,b)$ are all the parameters of the model.

If we have multiple input features, we can write: $f(x;\theta) = b + w_1x_1 + \dots + w_Dx_D = b + w^Tx$. This is called multiple linear regression.

Polynomial Regression

We can improve the fit by using a polynomial regression model of degree $D$. This has the form $f(x; w) = w^T\phi(x)$, where $\phi(x)$ is a feature vector derived from the input, which has the following form: $\phi(x) = \left[ 1, x, x^2, \dots, x^D \right]$.

import sklearn
import numpy as np

poly = sklearn.preprocessing.PolynomialFeatures(degree=3, include_bias=True)

# Scikit-learn expects a 2D grid. Here, we have a batch of 3 samples, each with 1 feature.
x = np.array([[1.0], 
              [2.0], 
              [3.0]])

phi_x = poly.fit_transform(x)

print("Scikit-learn Polynomial Features Matrix:\n", phi_x)
# Output: Scikit-learn Polynomial Features Matrix:
#  [[ 1.  1.  1.  1.]
#  [ 1.  2.  4.  8.]
#  [ 1.  3.  9. 27.]]
# Meaning:
# phi(1) => phi_x[0] = [ 1.  1.  1.  1.]
# phi(2) => phi_x[1] = [ 1.  2.  4.  8.]
# phi(3) => phi_x[2] = [ 1.  3.  9. 27.]

\phi(x) = \left[ 1, x, x^2, \dots, x^D \right]

i.e. Linear: $\hat{f}(x) = w_0 + w_1x_1 + w_2x_2$; Polynomial: $\hat{f}(x) = w_0 + w_1x_1 + w_2x_2 + w_3x_1^2 + w_4x_2^2$.

This is a simple example of feature preprocessing, also called feature engineering.

Note that the above models still use a prediction function that is a linear function of the parameters $w$, even though it is a nonlinear function of the original input $x$.

Deep Neural Networks

We had manually specified the transformation of the input features, namely polynomial expansion, $\phi(x) = \left[ 1, x_1, x_2, x_1^2, x_2^2, \dots\right]$. We can create much more powerful models by learning to do such nonlinear feature extraction automatically. If we let $\phi(x)$ have its own set of parameters, say $\mathbf{V}$, then the overall model has the form: $f(x; w, V) = w^T\phi(x;V)$.

We can recursively decompose the feature extractor $\phi(x; V)$ into a composition of simpler functions. The resulting model then becomes a stack of $L$ nested functions:

where $f_{\ell}(x) = f(x; \theta_{\ell})$ is the function at layer $\ell$.

Overfitting and Generalization

We can rewrite the empirical risk:

in the following equivalent way:

where $|D_{train}|$ is the size of the training set $D_{train}$. This formulation is useful because it makes explicit which dataset the loss is being evaluated on.

A model that perfectly fits the training data, but which is too complex, is said to suffer from overfitting.

To detect if a model is overfitting, let us assume (for now) that we have access to the true (but unknown) distribution $p^\ast(x,y)$ used to generate the training set. Then, instead of computing the empirical risk we compute the theoretical expected loss or population risk:

The difference $\mathcal{L}(\theta; p^\ast) – \mathcal{L}(\theta; D_{train})$ is called the generalization gap. If a model has a large generalization gap (i.e., low empirical risk but high population risk), it is a sign that it is overfitting.

In practice we don’t know $p^*$. However, we can partition the data we do have into two subsets, known as the training set and the test set. Then we can approximate the population risk using the test risk:

In practice, we need to partition the data into three sets, namely the training set, the test set and a validation set; the latter is used for model selection, and we just use the test set to estimate future performance (the population risk), i.e., the test set is not used for model fitting or model selection.

No Free Lunch Theorem

Given the large variety of models in the literature, it is natural to wonder which one is best. Unfortunately, there is no single best model that works optimally for all kinds of problems. The best way to pick a suitable model is based on domain knowledge, and/or trial and error. For this reason, it is important to have many models and algorithmic techniques in one’s toolbox to choose from.

Unsupervised Learning

An arguably much more interesting task is to try to “make sense of” data, as opposed to just learning a mapping. That is, we just get observed “inputs” $D = \left\{ x_n: n=1: N \right\}$ without any corresponding “outputs” $y_n$. This is called unsupervised learning.

From a probabilistic perspective, we can view the task of unsupervised learning as fitting an unconditional model of the form $p(x)$, which can generate new data $x$, whereas supervised learning involves fitting a conditional model, $p(y|x)$, which specifies (a distribution over) outputs given inputs.

Clustering

A simple example of unsupervised learning is the problem of finding clusters in data. The goal is to partition the input into regions that contain “similar” points.

Discovering Latent “Factors of Variation”

In the real world, data often has too many variables (high-dimensional, $x_n \in \mathbb{R}^D$). However, we assume that this complex, observed data is actually driven by a much smaller set of hidden, underlying concepts called latent variables ($z_n \in \mathbb{R}^K$).

Imagine you have a high-definition video ($x$) of a person’s 3D face consisting of millions of pixels ($D$). In reality, those millions of pixels are completely controlled by just a few facial muscles ($z$, size $K$). If you can find $z$, you capture the “essence” of the video in a fraction of the size.

The Generative Direction ($z_n \to x_n$): We model the world as a generative process. First, nature chooses some latent factors $z_n$ (e.g., facial muscle positions) from a simple prior distribution like a Gaussian $p(z_n)$. Then, those factors generate our observed data $x_n$ (the pixels) through some likelihood function $p(x_n \mid z_n)$.

The simplest example is when we use a linear model, $p(x_n \mid z_n; \theta) = \mathcal{N}(x_n \mid Wz_n + \mu, \Sigma)$.

Here, we say that the observed data $x_n$ is a linear transformation $(Wz_n + \mu)$ of our hidden factors, plus some added Gaussian noise (represented by the covariance matrix $\Sigma$).

You have to figure out both the inputs $z$ and the weights $W$ at the same time.

In Factor Analysis, the noise $\Sigma$ can be different for each observed dimension (a diagonal matrix with different values on the diagonal). It means some pixel sensors might be noisier than others.

In Probabilistic PCA, we assume the noise is exactly the same in every direction ($\Sigma = \sigma^2 I$, where $I$ is the identity matrix). This makes the math cleaner and directly aligns with classic geometric Principal Component Analysis (PCA).

While linear models are elegant, the real world is rarely linear. To fix this, we can replace the linear mapping $Wz_n$ with a nonlinear function, $f(z_n; \theta$):

$f(z_n; \theta)$ is a deep neural network. This network acts as a decoder. It takes a tiny latent vector $z$ and learns a highly complex, nonlinear way to reconstruct the high-dimensional data $x$. Estimating the parameters ($\theta$) of the neural network is incredibly hard when you don’t even know what the inputs ($z$) are. This is where the Variational Autoencoder (VAE) comes in. It solves this optimization nightmare by training a second neural network (the encoder, or inference network) that learns to guess the latent variables $z$ directly from the inputs $x$.

Self-supervised Learning

In this approach, we create proxy supervised tasks from unlabeled data.

It uses the exact same math, loss functions, and optimization algorithms as supervised learning, but the training targets are automatically extracted from the raw data itself.

We do not actually care about the model’s performance on the proxy task (e.g., predicting missing words). We only care about the high-quality, reusable feature representations the model is forced to develop in order to solve that task.

Evaluating Unsupervised Learning

Evaluating unsupervised learning is incredibly hard because there are no correct labels (“ground truth”) to compare the results against. To solve this, researchers use three main approaches to measure success:

• Density Estimation (Assigning Probabilities): We measure how “surprised” the model is by brand-new, unseen test data. We calculate this using negative log-likelihood: $\mathcal{L}(\theta; \mathcal{D}) = -\frac{1}{\vert{}\mathcal{D}\vert{}} \sum_{x \in \mathcal{D}} \log p(x \mid \theta)$.
• Downstream Performance (Sample Efficiency): Instead of evaluating the model directly, we use the features it learned as the inputs for a supervised task (like classification).
• Interpretability: A great unsupervised model should explain why the data looks the way it does by uncovering its true, underlying structure.

Reinforcement Learning

The system or agent has to learn how to interact with its environment. This can be encoded by means of a policy $a = \pi(x)$, which specifies which action to take in response to each possible input $x$ (derived from the environment state).

The difference from supervised learning (SL) is that the system is not told which action is the best one to take (i.e., which output to produce for a given input). Instead, the system just receives an occasional reward (or punishment) signal in response to the actions that it takes.

To compensate for the minimal amount of information coming from the reward signal, it is common to use other information sources, such as expert demonstrations, which can be used in a supervised way, or unlabeled data, which can be used by an unsupervised learning system to discover the underlying structure of the environment.

Data

The nature and quality of the training data also plays a vital role in the success of any learned model.

Some Common Image Datasets

Small Image Datasets

• MNIST (Modified National Institute of Standards): 60k training images and 10k test images, each of size 28 ×28 (grayscale), illustrating handwritten digits from 10 categories.
• EMNIST (Extended MNIST): includes lower and upper case letters. There are 62 classes.
• Fashion-MNIST: each image is the picture of a piece of clothing.
• CIFAR (Canadian Institute For Advanced Research): dataset of 60k color images, each of size 32 ×32 ×3, representing everyday objects from 10 or 100 classes;

ImageNet

This is a dataset of ∼14M images of size 256 ×256 ×3 illustrating various objects from 20,000 classes; was used as the basis of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), which ran from 2010 to 2018.

Some Common Text Datasets

Machine learning is often applied to text to solve a variety of tasks. This is known as natural language processing or NLP.

Text Classification

IMDB (Internet Movie Database): contains 25k labeled examples for training, and 25k for testing. Each example has a binary label, representing a positive or negative rating.

Machine Translation

• English-French pairs from the Canadian parliament.
• Europarl: from the European Union.
• WMT (Workshop on Machine Translation): English-German pairs; a subset of Europarl.

Other seq2seq Tasks

A generalization of machine translation is to learn a mapping from one sequence $x$ to any other sequence $y$. This is called a seq2seq model, and can be viewed as a form of high-dimensional classification. Includes tasks:

• Document Summarization
• Question Answering

Language Modeling

Refers to the task of creating unconditional generative models of text sequences, $p(x_1,…,x_T)$. This only requires input sentences $x$, without any corresponding “labels” $y$. We can therefore think of this as a form of unsupervised learning. If the language model generates output in response to an input, as in seq2seq, we can regard it as a conditional generative model.

Preprocessing Discrete Input Data

One-hot Encoding

When we have categorical features, we need to convert them to a numerical scale, so that computing weighted combinations of the inputs makes sense.

If a variable $x$ has $K$ values, we will denote its dummy encoding as follows: $\text{one-hot}(x) = \left[ \mathbb{I}(x=1), \dots, \mathbb{I}(x=K) \right]$. For example, if there are 3 colors (say red, green and blue), the corresponding one-hot vectors will be $\text{one-hot}(red) = [1,0,0], \text{one-hot}(green) = [0, 1,0], \text{one-hot}(blue) = [0,0,1]$.

Feature Crosses

A linear model using a dummy encoding for each categorical variable can capture the main effects of each variable, but cannot capture interaction effects between them.

For example, suppose we want to predict the fuel efficiency of a vehicle given two categorical input variables: the type (say SUV, Truck, or Family car), and the country of origin (say USA or Japan). If we concatenate the one-hot encodings for the ternary and binary features, we get the following input encoding:

where $x_1$ is the type and $x_2$ is the country of origin.

We can fix this by computing explicit feature crosses:

$f(x; w) = w^T\phi(x) \\ = w_0 + w_1\mathbb{I}(x_1=S) + w_2\mathbb{I}(x_1=T) + w_3\mathbb{I}(x_1=F) + \\ w_4\mathbb{I}(x_2=U) + w_5\mathbb{I}(x_2=J) + w_6\mathbb{I}(x_1=S,x_2=U) + \\ w_7\mathbb{I}(x_1=T,x_2=U) + w_8\mathbb{I}(x_1=F,x_2=U) + \\ w_9\mathbb{I}(x_1=S,x_2=J) + w_{10}\mathbb{I}(x_1=T,x_2=J) + \\ w_{11}\mathbb{I}(x_1=F,x_2=J)$

We can see that the use of feature crosses converts the original dataset into a wide format, with many more columns:

$\phi(x) = \left[ \begin{aligned} 1,\mathbb{I}(x_1=S), \mathbb{I}(x_1=T),\mathbb{I}(x_1=F), \mathbb{I}(x_2=U),\mathbb{I}(x_2=J), \\ \mathbb{I}(x_1=S,x_2=U) ,\mathbb{I}(x_1=T,x_2=U),\mathbb{I}(x_1=F,x_2=U),\\ \mathbb{I}(x_1=S,x_2=J),\mathbb{I}(x_1=T,x_2=J), \mathbb{I}(x_1=F,x_2=J) \end{aligned} \right]$

import pandas as pd

df = pd.DataFrame({
    'col1': [1, 2, 3, 1], # S, T, F, S
    'col2': [1, 2, 1, 2]  # U, J, U, J
})

combined_series = df['col1'].astype(str) + "_" + df['col2'].astype(str)

df['combined_col'] = combined_series.astype('category').cat.codes + 1

print("Pandas DataFrame:\n ",df)
# Output: Pandas DataFrame:
#    col1  col2  combined_col
# 0     1     1             1
# 1     2     2             3
# 2     3     1             4
# 3     1     2             2

df['col1'] = pd.Categorical(df['col1'], categories=[1, 2, 3])
df['col2'] = pd.Categorical(df['col2'], categories=[1, 2])
df['combined_col'] = pd.Categorical(df['combined_col'], categories=[1, 2, 3, 4, 5, 6])

one_hots_df = pd.get_dummies(df, columns=['col1', 'col2', 'combined_col'], dtype=float)

one_hots_df.insert(0, 'bias', 1.0)

phi_x = one_hots_df.to_numpy()
print("phi:\n", phi_x)
# Output: 
# phi:
# [[1. 1. 0. 0. 1. 0. 1. 0. 0. 0. 0. 0.]
#  [1. 0. 1. 0. 0. 1. 0. 0. 1. 0. 0. 0.]
#  [1. 0. 0. 1. 1. 0. 0. 0. 0. 1. 0. 0.]
#  [1. 1. 0. 0. 0. 1. 0. 1. 0. 0. 0. 0.]]

\phi(x) = \left[ \begin{aligned} 1,\mathbb{I}(x_1=S), \mathbb{I}(x_1=T),\mathbb{I}(x_1=F), \mathbb{I}(x_2=U),\mathbb{I}(x_2=J), \\ \mathbb{I}(x_1=S,x_2=U) ,\mathbb{I}(x_1=T,x_2=U),\mathbb{I}(x_1=F,x_2=U),\\ \mathbb{I}(x_1=S,x_2=J),\mathbb{I}(x_1=T,x_2=J), \mathbb{I}(x_1=F,x_2=J)  \end{aligned} \right]

Preprocessing Text Data

To feed text data into a classifier, we need to tackle various issues. First, documents have a variable length, and are thus not fixed-length feature vectors, as assumed by many kinds of models. Second, words are categorical variables with many possible values (equal to the size of the vocabulary), so the corresponding one-hot encodings will be very high-dimensional, with no natural notion of similarity. Third, we may encounter words at test time that have not been seen during training (so-called out-of-vocabulary or OOV words).

Bag of Words Model

A simple approach to dealing with variable-length text documents is to interpret them as a bag of words, in which we ignore word order. To convert this to a vector from a fixed input space, we first map each word to a token from some vocabulary.

Let $x_{nt}$ be the token at location $t$ in the $n$’th document. If there are $D$ unique tokens in the vocabulary, then we can represent the $n$’th document as a $D$-dimensional vector $\tilde{x}_n$, where $\tilde{x}_{nv}$ is the number of times that word $v$occurs in document $n$:

import tensorflow as tf

text_data = [
    "The cat sat on the mat",
    "The dog lay on the rug"
]

max_tokens = 10
vectorizer = tf.keras.layers.TextVectorization(
    max_tokens=max_tokens, 
    output_mode="count"
)

vectorizer.adapt(text_data)

bow_representation = vectorizer(text_data)

print("Learned Vocabulary Indices:\n", vectorizer.get_vocabulary())
print("Abstract BoW Matrix:\n", bow_representation.numpy())
# Output: 
# Learned Vocabulary Indices:
#  ['[UNK]', np.str_('the'), np.str_('on'), np.str_('sat'), np.str_('rug'), np.str_('mat'), np.str_('lay'), np.str_('dog'), np.str_('cat')]
# Abstract BoW Matrix:
#  [[0 2 1 1 0 1 0 0 1]
#  [0 2 1 0 1 0 1 1 0]]

from sklearn.feature_extraction.text import CountVectorizer

text_data = [
    "The cat sat on the mat",
    "The dog lay on the rug"
]

vectorizer = CountVectorizer(max_features=5, lowercase=True)

bow_matrix = vectorizer.fit_transform(text_data)

print("Learned Feature Names (Vocabulary Blueprint):")
print(vectorizer.get_feature_names_out())

print("Raw Count Matrix for Training Data:")
print(bow_matrix.toarray())
# Output:
# Learned Feature Names (Vocabulary Blueprint):
#  ['cat' 'dog' 'mat' 'on' 'the']
# Raw Count Matrix for Training Data:
# [[1 0 1 1 2]
#  [0 1 0 1 2]]

TF-IDF

One problem with representing documents as word count vectors is that frequent words may have undue influence, just because the magnitude of their word count is higher, even if they do not carry much semantic content. A common solution to this is to transform the counts by taking logs, which reduces the impact of words that occur many times within a single document.

from sklearn.feature_extraction.text import TfidfVectorizer

text_data = ["The cat sat on the mat", "The dog lay on the rug"]

vectorizer = TfidfVectorizer(lowercase=True)
tfidf_matrix = vectorizer.fit_transform(text_data)

print("Vocabulary Blueprint:")
print(vectorizer.get_feature_names_out())

print("\nTF-IDF Weighted Matrix:")
print(tfidf_matrix.toarray())
# Output:
# Vocabulary Blueprint:
#  ['cat' 'dog' 'lay' 'mat' 'on' 'rug' 'sat' 'the']
# TF-IDF Weighted Matrix:
#  [[0.42519636 0.         0.         0.42519636 0.30253071 0.
#   0.42519636 0.60506143]
#  [0.         0.42519636 0.42519636 0.         0.30253071 0.42519636
#   0.         0.60506143]]

Word Embeddings

We map each sparse one-hot vector, $x_{nt} \in \{0, 1 \} ^ V$, to a lower-dimensional dense vector, $e_{nt} \in \mathbb{R}^K$ using $e_{nt} \in \text{E}x_{nt}$, where $\text{E} \in \mathbb{R}^{K \times V}$ is learned such that semantically similar words are placed close by.

Once we have an embedding matrix, we can represent a variable-length text document as a bag of word embeddings. We can then convert this to a fixed length vector by summing (or averaging) the embeddings: $\overline{e}_n = \sum_{t=1}^T e_{nt} = \mathbf{E}\tilde{x}_n$, where $\tilde{x}_n$ is the bag of words representation.

We can then use this inside of a logistic regression classifier. The overall model has the form:

Dealing with Novel Words

Such novel words are bound to occur, because the set of words is an open class. For example, the set of proper nouns (names of people and places) is unbounded.

A standard heuristic to solve this problem is to replace all novel words with the special symbol UNK, which stands for “unknown”. It is better to leverage the fact that words have substructure, and then to take as input subword units or wordpieces; these are often created using a method called byte-pair encoding, which is a form of data compression that creates new symbols to represent common substrings.

Handling Missing Data

Sometimes we may have missing data, in which parts of the input $x$ or output $y$ may be unknown.

To mathematically track what is missing and what is not, we create a “mask” matrix, $M$.

$M$ (The Mask Matrix): A binary grid of the same size as our dataset ($N$ examples $\times D$ features).

• $M_{nd} = 1$ means the data is hidden / missing ($X_h$).
• $M_{nd} = 0$ means the data is visible / observed ($X_v$).

The Three Types of Missing Data:

• Missing Completely at Random (MCAR): The missingness is pure noise and has absolutely nothing to do with any of the visible or hidden data. $p(M \mid X_v, X_h, Y) = p(M)$
• Missing at Random (MAR): The missingness depends only on the visible data we have collected, not on the actual hidden values that are missing. $p(M \mid X_v, X_h, Y) = p(M \mid X_v, Y)$
• Not Missing at Random (NMAR): The missingness directly depends on the unobserved/hidden value itself. The fact that the data is missing is informative.