Matrix Multiplication

1 Introduction

I am working through a couple of linear algebra books/courses as I write this. All content in here will be heavily inspired by those resources:

• Gilbert Strang. 18.06 Linear Algebra. Spring 2010. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.
• Applied Linear Algebra with A. P. L. by Garry Helzer (Author)

I have 2 main goals: + Learn Dyalog APL: APL works very differently than other languages I have done. By learning it I will learn another way of thinking and approaching problems. By having more ways to think and approach problems I become smarter. I want to be smarter. + Improve my mathematical foundation

To do this I plan to go through math material and courses in Dyalog APL. In the beginning I will be reviewing basic math while learning APL, but eventually I will get to content where I am both learning APL and math at the same time. This is where I will document what I do.

Where to learn APL

Check out the fastai apl study group, accompanying videos, and anki decks if you want to learn APL .

2 Vector Name Conventions

For this (and other) posts I will follow the naming conventions in APL cart to keep things consistent with the APL community. For convenience here they are:

• X, Y, Z: any type array 
• M, N: numeric array 
• I, J: integer array 
• A, B: Boolean array 
• C, D: character array 
• f, g, h: function 
• ax: bracket axis 
• s: scalar 
• v: vector 
• m: matrix

If you find it hard to keep track of - I would recommend making a short anki deck to remember them.

3 Matrix Multiplication (matrix × vector)

Let’s start with a matrix and a vector and multiply them together. First, we define the our variables.

⎕ ← M ← 2 2 ⍴ 2 5 1 3
⎕ ← N ← 2 1 ⍴ 1 2

┌→──┐ ↓2 5│ │1 3│ └~──┘

┌→┐ ↓1│ │2│ └~┘

3.1 Manual Calculation

We take the rows of N times the columns of M to do a linear combination to do matrix multiplication.

\(1\begin{bmatrix}2\\1\end{bmatrix} + 2\begin{bmatrix}5\\3\end{bmatrix}\)

We can do this exactly in APL and see our answer.

⎕ ← (col1 ← M[;1] × N[1;]) + (col2 ← M[;2] × N[2;])

┌→───┐ │12 7│ └~───┘

3.2 APL Calculation

3.2.1 Dot

in APL we would normally not write is all out but would write it using the dot (.) function. Let’s figure out what that is and how it works.

The . applies the operators the surround it (⍺⍺ and ⍵⍵) in a specific way and order called an inner product.

In our matrix multiplication problem it looks like this. \(\begin{bmatrix}(1⍵⍵2)⍺⍺(2⍵⍵5)\\(1⍵⍵1)⍺⍺(2⍵⍵3)\end{bmatrix}\)

Let’s walk through this in our matrix multiplication example above one operator at a time to understand it a bit better

3.2.2 Apply the ⍵⍵ argument

I like to visualize the first step like this:

\(\begin{bmatrix}⍵⍵&&1\\&⍵⍵&2\\2&5&\end{bmatrix}\)

\(\begin{bmatrix}⍵⍵&&1\\&⍵⍵&2\\1&3&\end{bmatrix}\)

We first apply whatever the ⍵⍵ parameter is and combine elements. Just as we did above manually we need to do multiplication here so we know the ⍵⍵ parameter must be ×. These calculation are:

Matrix 1:

• 1 × 2 = 2
• 2 × 5 = 10

Matrix 2:

• 1 × 1 = 1
• 2 × 3 = 6

So far we have ⍺⍺.×. We can show the result of our calculations above in matrices.

\(\begin{bmatrix}2&⍺⍺\\&10\end{bmatrix}\)

\(\begin{bmatrix}1&⍺⍺\\&6\end{bmatrix}\)

3.2.3 Apply the ⍺⍺ argument

The next thing the . operator does is combine all the numbers in each of step 1 resulting matrices using ⍺⍺. To get the linear combination we did above we need to add the numbers in each matrix, so the ⍺⍺ operator must be +.

If we do that addition:

• Matrix 1: 2 + 10 = 12

• Matrix 2: 1 + 6 = 7

Leaving us with our answer of \(\begin{bmatrix}12\\7\end{bmatrix}\)

So to do matrix multiplication we simply need to use:

M +.× N

┌→────┐ ↓17 24│ │10 14│ └~────┘

3.2.4 Dot is flexible

This was just 1 example of using the . operator. We used + as ⍺⍺ and × as ⍵⍵ to fit what we needed for this problem.

Now that we understand that, we can flip our operators and look at ×.+ instead of +.×. We can also do any number of other operators to do lots of different matrix operations. Take a look at the examples below and try calculating them by hand to see what you get!

⍝ using addition.multiplication (normal matrix multiplication) 
M+.×N 

┌→─┐ ↓12│ │ 7│ └~─┘

⍝ using multiplication.addition
M×.+N 

┌→─┐ ↓21│ │10│ └~─┘

⍝ using max.min
M⌈.⌊N

┌→┐ ↓2│ │2│ └~┘

⍝ using addition.subtraction
M-.+N

┌→─┐ ↓¯4│ │¯3│ └~─┘

⍝ using exponent.division
M*.÷N

┌→──────────┐ ↓5.656854249│ │1 │ └~──────────┘

⍝ using factorial.natural_log
M!.⍟N

┌→───────────┐ ↓1 │ │0.6309297536│ └~───────────┘

4 Matrix Multiplication (matrix × matrix)

As you might imagine we can do a variety of shapes. Because we are simply applying our addition and multiplication operators, any shapes where those operators work works. Let’s go through another example, this time with two 2x2 matrices.

First we need to define our variables we want to multiply together.

⎕ ← M ← 2 2 ⍴ 2 5 1 3
⎕ ← N ← 2 2 ⍴ 1 2 3 4

┌→──┐ ↓2 5│ │1 3│ └~──┘

┌→──┐ ↓1 2│ │3 4│ └~──┘

4.0.1 Manual Calculation

We learned how to do matrix multiplication earlier which was just taking linear combinations of rows and columns. Let’s try to do that same thing here as well. In our previous example we were using all scalars because 1 row of N was a scalar, but let’s just follow our rule and put a vector in and see what happens.

\(\begin{bmatrix}([1,2]⍵⍵2)⍺⍺([3,4]⍵⍵5)\\([1,2]⍵⍵1)⍺⍺([3,4]⍵⍵3)\end{bmatrix}\)

As you can see this matrix multiplication is still just a linear combination taking the rows of N × the column of M. I believe it is most intuitive to think of matrix multiplication as a linear combination of the columns of M.

Note: There are many other ways to define functions to get the same number (such as using looping or recursion). The most intuitive way to think about something is not necessarily the most efficient way to program a function to tell a computer to calculate the number (which will depend on what programming language you are using).

⎕←ans ← ((1 2 × 2) + (3 4 × 5)) ((1 2 × 1) + (3 4 × 3))

┌→────────────────┐ │ ┌→────┐ ┌→────┐ │ │ │17 24│ │10 14│ │ │ └~────┘ └~────┘ │ └∊────────────────┘

To make it equivilant to the apl operator we need to reshape this array of arrays into a 2 × 2

⎕ ← ans ← 2 2 ⍴ ∊ ans

┌→────┐ ↓17 24│ │10 14│ └~────┘

Let’s use the matrix multiplication in APL to prove that it’s really getting the same answer. We can use ≡to test for equality

ans ≡ M +.× N

1

We can also see this in the same matrix format we did in our first example of a matrix and a vector, which is doing the same thing but I think illustrates it clearer. Let’s start with our problem again:

\(\begin{bmatrix}2&5\\1&3\end{bmatrix}\) \(\begin{bmatrix}1&2\\3&4\end{bmatrix}\)

4.0.2 Apply the ⍵⍵ argument

\(\begin{bmatrix}⍵⍵&&[1,2]\\&⍵⍵&[3,4]\\2&5&\end{bmatrix}\)

\(\begin{bmatrix}⍵⍵&&[1,2]\\&⍵⍵&[3,4]\\1&3&\end{bmatrix}\)

Since ⍵⍵ is × we do that first and get to our next step

4.0.3 Apply the ⍺⍺ argument

\(\begin{bmatrix}[2,4]&⍺⍺\\&[15,20]\end{bmatrix}\)

\(\begin{bmatrix}[1,2]&⍺⍺\\&[9,12]\end{bmatrix}\)

⍺⍺ is + so we can do the addition to end with

\(\begin{bmatrix}17&24\end{bmatrix}\)

\(\begin{bmatrix}10&14\end{bmatrix}\)

Which again matches the inner product that APL does.

M +.× N

┌→────┐ ↓17 24│ │10 14│ └~────┘