You Hated Matrix Math in College. Turns Out It Runs Every AI Model You’ve Ever Used.

There’s a specific kind of pain reserved for sitting in a linear algebra lecture, staring at a matrix multiplication problem, and thinking: when will I ever use this?

Your professor said “it’s foundational.” You filed that under “things adults say when they can’t explain themselves.” You memorized the rules, passed the exam, and quietly assumed you’d never multiply two matrices again.

Then large language models showed up.

It turns out linear algebra isn’t a prerequisite for understanding AI the way calculus is a prerequisite for physics — where you need it to get in the door but rarely touch it again. Linear algebra is the AI. Every token that flows through a transformer, every attention score, every embedding comparison, every layer of every neural network — it’s all matrix operations, all the way down.

Here’s the vindication tour your professor never gave you.

Matrix Multiplication: The Forward Pass

Let’s start with the most basic operation and why it’s everywhere.

When text enters a language model, each token gets converted into a vector — a list of numbers, typically 4,096 or more dimensions long for a large model. Think of it as a coordinate in a very high-dimensional space, where the model has learned to place tokens with similar meanings near each other.

Now the model needs to transform that vector — to process it, mix it with other information, project it into different representations. How? Matrix multiplication.

Every layer of a transformer is fundamentally this operation:

output = input × W\n

Where input is your token vector, W is a learned weight matrix, and output is the transformed representation. Do this across dozens of layers, with different weight matrices at each step, and you get a model that can translate languages, write code, and explain quantum physics.

The weight matrices are what training learns. When you “train” a neural network, you are iteratively adjusting the numbers inside millions of matrices until the outputs match what you want. The architecture is fixed. The matrices are what change.

This is why people say neural networks are “just matrix multiplication” — it’s reductive, but it’s not wrong.

Dot Products: The Attention Mechanism

Here’s where the math you suffered through becomes genuinely elegant.

The transformer’s core innovation — the thing that made modern LLMs possible — is the attention mechanism. The idea: when processing any token in a sequence, the model should be able to “look at” every other token and decide how much each one matters for understanding the current one.

When you read the sentence “The animal didn’t cross the street because it was too tired,” your brain knows “it” refers to “animal” and not “street.” Attention is how the model does this computationally.

The mechanism produces three vectors per token: Query (Q), Key (K), and Value (V). Here’s what they represent intuitively:

Q is the token asking “what should I pay attention to?”
K is each token announcing “here’s what I contain”
V is the actual information each token contributes if attended to

The attention score between two tokens is computed as a dot product:

score = Q · K

A dot product is just element-wise multiplication followed by summation:

[1, 2, 3] · [4, 5, 6] = (1×4) + (2×5) + (3×6) = 32

Geometrically, the dot product measures how aligned two vectors are. High score = the Query and Key are pointing in similar directions = this token is relevant to what we’re looking for. Low or negative score = not relevant.

That number — which you computed by hand in a calculus class and immediately forgot — is how “it” gets linked back to “animal” in every transformer running today.

Softmax: Turning Scores Into Probabilities

After computing attention scores, the model has a list of raw numbers — one per token in the context. But raw numbers aren’t weights you can use directly. You need them to sum to 1, so they can be interpreted as “how much of my attention does each token deserve.”

Enter softmax:

softmax(x_i) = e^x_i / Σ e^x_j

Exponentiate each score, divide by the sum of all exponentiated scores. The result: every score is positive, and they all sum to 1. The largest scores get amplified disproportionately — softmax sharpens the distribution, making the model commit more decisively to the most relevant tokens.

You probably saw this in statistics or probability. It’s doing the same thing here: converting arbitrary numbers into a valid probability distribution. That distribution tells the model how to weight the Value vectors when constructing its output — which is the weighted sum that actually gets passed to the next layer.

Transpose: The Unsung Hero

Here’s one that gets zero credit. The attention score computation at scale is:

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

That K^T is the transpose of K — flipping the matrix so rows become columns. Without it, the matrix dimensions don’t align for multiplication. It’s the operation that makes the dot product between Q and every K simultaneously computable as a single matrix multiply rather than a loop.

You transposed matrices in homework because the problem told you to. Transformers transpose them because it’s the only way to make the attention computation efficient enough to run at all.

Vector Spaces: Where Meaning Lives

Every time you use a RAG system, a semantic search, or any embedding-based application, you’re doing geometry in high-dimensional space.

An embedding model takes text and produces a vector. Similar texts get similar vectors — “king” and “queen” end up nearby. “King” and “carburetor” end up far apart. The model has learned a geometry of meaning.

To find the most relevant documents to a query, you compute the similarity between the query vector and every document vector. The standard measure is cosine similarity:

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

It’s the dot product normalized by the magnitudes of both vectors. The result ranges from -1 to 1, where 1 means identical direction (maximally similar), 0 means perpendicular (unrelated), and -1 means opposite.

The reason cosine similarity rather than Euclidean distance: you care about the direction vectors point, not how far from the origin they are. A short document and a long document about the same topic should register as similar even if one has a larger magnitude.

This is the geometry your linear algebra professor was teaching when they explained the dot product and unit vectors. It felt abstract. It is now the retrieval mechanism inside every vector database running in production.

Eigenvalues: The One That Actually Comes Back

Eigenvalues are where most students check out. The intuition — that some vectors only get scaled (not rotated) when a matrix is applied to them — feels like a mathematical curiosity with no obvious use.

Two places it shows up in AI:

PCA (Principal Component Analysis) — used to reduce the dimensionality of embeddings without losing the most important structure. The principal components are the eigenvectors of the covariance matrix. When researchers visualize high-dimensional embedding spaces in 2D or 3D to understand what the model has learned, PCA (or its cousin t-SNE) is doing eigenvector decomposition under the hood.
Understanding attention heads — research into what individual attention heads learn has found that some heads specialize: one might track syntactic dependencies, another coreference. Analyzing the eigenstructure of attention weight matrices is one of the tools researchers use to reverse-engineer what the model has learned. It’s not something you do in production, but it’s the method behind “mechanistic interpretability” — the field trying to understand what’s actually happening inside these models.

The Full Picture

Here’s the vindication:

Math concept Where it runs Matrix multiplication Every layer of every neural network, every forward pass Dot product Attention scores, cosine similarity Transpose Making attention computation efficient Softmax Converting scores to probability distributions Vector spaces Embeddings, semantic search, RAG Cosine similarity Vector database retrieval Eigenvalues/PCA Dimensionality reduction, interpretability research

The professor who told you it was foundational was right. They just couldn’t tell you that the application would be systems that generate human language at scale. That wasn’t on the curriculum in 2005.

The other thing worth noting: this is still the math. There are more sophisticated operations — layer normalization, feed-forward sublayers, positional encodings — but they’re built on the same substrate. If you understand matrix multiplication and dot products at the level of “what they’re computing and why,” you understand the mechanical core of how transformers work.

The gap between “I passed linear algebra” and “I understand what’s happening inside an LLM” turns out to be much smaller than the AI hype cycle makes it seem. The operations are not exotic. The scale is.

Your matrices are running. They have been this whole time.

Under The Hood covers the mechanics that the AI product ecosystem hand-waves. If this made your linear algebra credits feel retroactively justified, share it with whoever told you math wasn’t practical.

Thanks for reading Under The Hood! Subscribe for free to receive new posts and support my work.