The Khatri-Rao Product Visualized

The Khatri-Rao product takes two matrices with the same number of columns and pairs their columns up, replacing each pair by its Kronecker product. The result is a tall matrix with the same number of columns as the inputs. As a result it keeps the width fixed while the heights are multiplied.

1. Definition: the column-wise Kronecker product

Write $\textcolor{#0d9488}{A} = \begin{bmatrix} \textcolor{#0d9488}{a_1} & \textcolor{#0d9488}{a_2} & \cdots & \textcolor{#0d9488}{a_K} \end{bmatrix}$ and $\textcolor{#b45309}{B} = \begin{bmatrix} \textcolor{#b45309}{b_1} & \textcolor{#b45309}{b_2} & \cdots & \textcolor{#b45309}{b_K} \end{bmatrix}$ as lists of columns, with $\textcolor{#0d9488}{a_k} \in \mathbb{R}^{I}$ and $\textcolor{#b45309}{b_k} \in \mathbb{R}^{J}$. On a matrix, a subscript picks a column and a superscript picks a row: $\textcolor{#0d9488}{a_k}$ is the $k$-th column of $\textcolor{#0d9488}{A}$, and $\textcolor{#0d9488}{a^i}$ (section 3) is its $i$-th row. The Khatri-Rao product is defined column by column:

Column $k$ of the output depends only on column $k$ of each input. Nothing mixes across columns.

The two inputs have $K = 2$ columns each. In the figure below, matching columns share a color: the entries of $A$ are circles, the columns of $B$ are blocks.

An example with $2 \times 2$ inputs

The second step is a $2 \times 2$ grid of blocks, not of numbers: each block is one whole column of $\textcolor{#b45309}{B}$, written out as a stacked pair and scaled by a single entry of $\textcolor{#0d9488}{A}$. Column $1$ of the answer therefore only ever sees $\textcolor{#b45309}{b_1}$, and column $2$ only ever sees $\textcolor{#b45309}{b_2}$, never the whole of $\textcolor{#b45309}{B}$. That single difference is what separates $\odot$ from $\otimes$.

2. One column at a time: the Kronecker product of two columns

Zoom in on a single column pair $\textcolor{#0d9488}{a}$, $\textcolor{#b45309}{b}$. Inside this section a subscript indexes an entry of that one column, not a column of a matrix. The Kronecker product $\textcolor{#0d9488}{a} \otimes \textcolor{#b45309}{b}$ stacks one scaled copy of $\textcolor{#b45309}{b}$ per entry of $\textcolor{#0d9488}{a}$:

Laying those same three blocks side by side instead of stacking them gives the outer product $\textcolor{#b45309}{b}\textcolor{#0d9488}{a^T}$:

Its columns are $\textcolor{#0d9488}{a_1}\textcolor{#b45309}{b}$, $\textcolor{#0d9488}{a_2}\textcolor{#b45309}{b}$ and $\textcolor{#0d9488}{a_3}\textcolor{#b45309}{b}$, the very blocks that the Kronecker product stacks.

Three copies of $\textcolor{#b45309}{b}$, each tinted by the entry of $\textcolor{#0d9488}{a}$ that scales it. The output length is the product of the two input lengths.

Both layouts hold the same numbers in the same order, because $\operatorname{vec}$ reads a matrix one column at a time and stacks those columns into a single long vector. That is why the Khatri-Rao product has a natural interpretation as an outer product in disguise:

Each column of $\textcolor{#0d9488}{A} \odot \textcolor{#b45309}{B}$ is a flattened outer product matrix.

3. The row-wise variant

Swapping rows for columns gives the face-splitting product. For $\textcolor{#0d9488}{A} \in \mathbb{R}^{I \times J}$ and $\textcolor{#b45309}{B} \in \mathbb{R}^{I \times K}$, the matrix $\textcolor{#0d9488}{A} \bullet \textcolor{#b45309}{B} \in \mathbb{R}^{I \times JK}$ has rows $\textcolor{#0d9488}{a^i} \otimes \textcolor{#b45309}{b^i}$, where $\textcolor{#0d9488}{a^i}$ is the $i$-th row of $\textcolor{#0d9488}{A}$. The two are transposes of each other:

Its defining property is that it turns a Kronecker product of inputs into an elementwise product of outputs:

where $*$ is the elementwise (Hadamard) product.

Proof

The identity holds row by row, so watch a single row $i$, with $\textcolor{#0d9488}{a}$ short for $\textcolor{#0d9488}{a^i}$ and $\textcolor{#b45309}{b}$ short for $\textcolor{#b45309}{b^i}$.

Both Kronecker products are indexed by the pair $(j,k)$: entry $(j,k)$ is $\textcolor{#0d9488}{a_j}\textcolor{#b45309}{b_k}$ on one side and $\textcolor{#0d9488}{x_j}\textcolor{#b45309}{y_k}$ on the other:

4. Three properties worth remembering

4.1 The Gram matrix collapses to a Hadamard product

Proof

Entry $(k,l)$ of the left side is the dot product of column $k$ with column $l$, and the two families separate:

A $K \times K$ answer that never touches the $IJ \times K$ matrix $\textcolor{#0d9488}{A} \odot \textcolor{#b45309}{B}$. This is the single most useful fact about $\odot$.

4.2 It expresses a weighted sum of outer products as a matrix-vector product

Proof

Read the left side one column at a time, then replace each Kronecker column by the outer product it flattens:

The last step is the outer-product reading of a matrix product: $\operatorname{diag}(x)$ hands weight $x_k$ to the $k$-th pairing of a column of $\textcolor{#b45309}{B}$ with a row of $\textcolor{#0d9488}{A^T}$.

4.3 The mixed product rule carries over

Proof

Column $k$ of $\textcolor{#0d9488}{C} \odot \textcolor{#b45309}{D}$ is itself a Kronecker product, so the mixed product rule for $\otimes$ applies to it directly:

The Khatri-Rao product is also associative: $(A \odot B) \odot C = A \odot (B \odot C)$.