• 1. Binary representations of integers
  • 2. Anatomy of a floating point number
  • 3. Block scaling in MXFP4 and NVFP4
  • 4. Adding two floating point numbers
  • 5. Why floating point addition isn't associative
  • 6. Exponentiation by squaring
unitgridOpacitycanvasColororiginXoriginY

Exponentiation by squaring

  • The bits of nnn pick the powers
  • The loop
  • Cost
  • Fibonacci numbers from matrix powers

Finally a small trick similar to how Fibonacci numbers can be calculated efficiently. Computing xnx^nxn by multiplying xxx by itself takes n−1n - 1n−1 multiplications. A clever way is to use squaring to get there in about log⁡2n\log_2 nlog2​n steps instead.


The bits of nnn pick the powers

Any nnn is a sum of powers of 222, its binary digits, so xnx^nxn is a product of those squares. For n=13=11012=8+4+1n = 13 = 1101_2 = 8 + 4 + 1n=13=11012​=8+4+1:

x13=x8⋅x4⋅x1x^{13} = x^{8} \cdot x^{4} \cdot x^{1}x13=x8⋅x4⋅x1
NBITS = 6  # n from 1 to 63

def powers_latex(n):
    """n's binary digits, and x^n as the product of the squares they pick."""
    n = int(n)
    ks = [k for k in reversed(range(NBITS)) if n >> k & 1]
    return (rf'{n} = {n:b}_2 = ' + ' + '.join(str(2 ** k) for k in ks) + r' \qquad '
            + rf'x^{{{n}}} = ' + r' \cdot '.join(f'x^{{{2 ** k}}}' for k in ks))

pie.clear()
n = pie.scalar(13, out='pow_n')

pie.ui.slider(n, min=1, max=2 ** NBITS - 1, step=1, label='n:')
pie.ui.latex(pie.lift(powers_latex, pie.Text)(n, out='pow_tex'))

The loop

The loop reads the bits of nnn from the right. Note that M on pass kkk is x2kx^{2^k}x2k.

def power(x, n):
    result = 1
    M = x
    while n:
        if n & 1:
            result = result * M
        M = M * M
        n >>= 1
    return result

print(power(3, 13), 3 ** 13)

Cost

Each bit of nnn costs at most two multiplications, so the loop takes at most about 2log⁡2n2 \log_2 n2log2​n, against n−1n - 1n−1.

nnn Bits 111 bits Loop: squarings + multiplications Plain: n−1n - 1n−1
131313 444 333 777 121212
1,0001{,}0001,000 101010 666 161616 999999999
10,00010{,}00010,000 141414 555 191919 9,9999{,}9999,999
1,000,0001{,}000{,}0001,000,000 202020 777 272727 999,999999{,}999999,999

The loop squares M once more than it needs to on its last pass. That is one wasted multiplication, kept here because it makes the loop shorter.


Fibonacci numbers from matrix powers

The loop works for anything that can be multiplied, matrices included. Powers of M=[1110]M = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}M=[11​10​] hold the Fibonacci numbers F0=0,F1=1,F2=1,F3=2,…F_0 = 0, F_1 = 1, F_2 = 1, F_3 = 2, \dotsF0​=0,F1​=1,F2​=1,F3​=2,…:

Mn=[Fn+1FnFnFn−1]M^n = \begin{bmatrix} F_{n+1} & F_n \\ F_n & F_{n-1} \end{bmatrix}Mn=[Fn+1​Fn​​Fn​Fn−1​​]

So FnF_nFn​ takes about 2log⁡2n2 \log_2 n2log2​n matrix multiplications. Run the editor: it prints a few FnF_nFn​ and the number of digits of F10,000F_{10{,}000}F10,000​.

def matmul(A, B):
    return [[sum(a * b for a, b in zip(row, col)) for col in zip(*B)] for row in A]

def matpow(M, n):
    result = [[1, 0], [0, 1]]
    while n:
        if n & 1:
            result = matmul(result, M)
        M = matmul(M, M)
        n >>= 1
    return result

M = [[1, 1],
     [1, 0]]
for n in (10, 20, 50, 100):
    print(f'F({n}) =', matpow(M, n)[0][1])
f = matpow(M, 10_000)[0][1]
print(f'F(10000) has {len(str(f))} digits')