Linear Algebra for Deep Learning

From scalars to PCA · Chapter 2 + reproducible Google Colab labs

Juan Sebastian Laverde

2026-01-01

The question that guides the class

How does a computer turn numbers, images, and text into mathematical objects that an AI model can process?

Real-world data

→

Numbers

→

Tensors

→

Operations

→

AI model

Do not start with formulas. Ask the group, ‘What does a computer see when we show it a photo?’ Guide them toward the idea of numerical representation.

Learning objectives

By the end, learners will be able to:

1. Identify scalars, vectors, matrices, and tensors in real-world data.
2. Interpret shape, ndim, batches, and channels.
3. Understand why matrix multiplication appears in neural networks.
4. Explain inverses, rank, norms, eigenvectors, SVD, and PCA intuitively.
5. Reproduce each section from 2.1–2.12 in Google Colab.
6. Move tensors among NumPy, TensorFlow, and PyTorch, including CPU/GPU.

Three-hour roadmap

45 min

2.1–2.4 · objects, products, inverse, span

35 min

2.5–2.7 · norms, special matrices, eigen

45 min

2.8–2.12 · SVD, pseudoinverse, trace, determinant, PCA

35 min

Colab · datasets + frameworks + CPU/GPU

+ 20 min spread across questions, transitions, and a short break.

2.1 Scalars, vectors, matrices, and tensors

Scalars

Scalars

A scalar is the simplest form of data: a single numerical or symbolic value. It can represent a constant or a univariate variable.

In machine learning, we usually work with real-valued scalars:

\[ a \in \mathbb{R} \]

Represents a single value

It has no dimensions or direction

It can be real or complex

Although they seem simple, scalars are fundamental: many model hyperparameters, such as learning rate \(\lambda\), the number of epochs, or a threshold, are often expressed as scalar values.

5

🌡️

24°C

λ

= 0.01

x

= 3.5

A scalar = one value

Vectors

Matrices

Vectors

Four objects, one idea

Object	Simple explanation	Data example	Typical shape	Tensor dimension / rank
Scalar	one number	age	()	Tensor 0D
Vector	ordered list	one person	(4,)	Tensor 1D
Matrix	2D table	100 people × 4 variables	(100, 4)	Tensor 2D
Tensor 3D	array with 3 axes	RGB image	(224, 224, 3)	Tensor 3D
Tensor 4D	array with 4 axes	batch of RGB images	(32, 224, 224, 3)	Tensor 4D
Tensor 5D	array with 5 axes	batch of videos	(8, 30, 224, 224, 3)	Tensor 5D
Tensor 6D	array with 6 axes	batch of video sequences	(4, 10, 30, 224, 224, 3)	Tensor 6D

Avoid saying that tensor ‘always means 3D’.

In modern frameworks, tensor is the general term for n-dimensional arrays.

A scalar can be viewed as a rank-0 tensor.

Explain the progression: - Scalar → tensor 0D - Vector → tensor 1D - Matrix → tensor 2D - RGB image → tensor 3D - Batch of images → tensor 4D - Video → tensor 4D - Batch of videos → tensor 5D

Broadcasting

Open lab 2.1 in Colab

2.2 Matrix and Vector Multiplication

Open lab 2.2

2.3 Identity and Inverse Matrices

Open lab 2.3

2.4 Linear Dependence and Span

Open lab 2.4

2.5 Norms

Open lab 2.5

2.6 Special Types of Matrices and Vectors

Open lab 2.6

2.7 Eigendecomposition

Open lab 2.7

2.8 Singular Value Decomposition (SVD)

Open lab 2.8

2.9 Moore–Penrose Pseudoinverse

Open lab 2.9

2.10 The Trace Operator

Open lab 2.10

2.11 Determinant

Open lab 2.11

2.12 Example: Principal Components Analysis (PCA)

Open lab 2.12

The algebra stays the same; where it runs changes

               same operation
                    A @ B
                      │
        ┌─────────────┼─────────────┐
        ↓             ↓             ↓
      NumPy      TensorFlow      PyTorch
       CPU        CPU / GPU       CPU / GPU

Repository and labs

Structure prepared for GitHub Pages:

linear-algebra-deep-learning/
├── en/
│   ├── slides/
│   └── exercises/
├── es/
│   ├── slides/
│   └── exercises/
├── shared/
├── _quarto.yml
└── .github/workflows/publish.yml

Expected URL:

https://laverde97.github.io/linear-algebra-deep-learning/

Sources and credits

• Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning, Chapter 2: Linear Algebra. MIT Press (Goodfellow et al. 2016). https://www.deeplearningbook.org/contents/linear_algebra.html
• Quarto RevealJS documentation: https://quarto.org/docs/presentations/revealjs/
• Quarto GitHub Pages publishing: https://quarto.org/docs/publishing/github-pages.html
• MNIST via TensorFlow/Keras (LeCun et al. 1998).
• IMDB Reviews via TensorFlow/Keras (Maas et al. 2011).
• ImageNet-v2 documentation via TensorFlow Datasets (Recht et al. 2019); for the live class the notebook uses a lightweight ImageNet-format demonstration and leaves ImageNet-v2 as optional due to download size.

Original teaching content based on the chapter concepts; it does not reproduce the full text of the book.

Key takeaway

Deep learning works with representations. Linear algebra gives us the language to describe, transform, measure, and compress them.

Understand

→

Run

→

Visualize

→

Connect to AI

Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. https://www.deeplearningbook.org/.

LeCun, Yann, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86 (11): 2278–324.

Maas, Andrew L., Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. “Learning Word Vectors for Sentiment Analysis.” Proceedings of ACL.

Recht, Benjamin, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. “Do ImageNet Classifiers Generalize to ImageNet?” International Conference on Machine Learning.