Neural Network Basics (Part 1)
Under the hood of a basic Deep Learning Image Classifier
- Intro
- Load the Data
- Defining our Linear Equation
- Loss Function
- Gradient Descent
- ReLu
- Train the Full Neural Network
- Recap of Tensor Shapes
- What More?
from fastai.vision.all import *
from fastai.data.external import *
from PIL import Image
import math
torch.manual_seed(100)
Intro
Today we will be working with a subset of the MNIST dataset. The goal is going to be to take an image of handwritten digits and automatically predict whether it is a 3 or a 7. We will be building a Neural Network to do this.
If you get through this and want more detail, I highly recommend checking out Deep Learning for Coders with fastai & Pytorch by Jeremy Howard and Sylvain Gugger. All of the material in this guide and more is covered in much greater detail in that book. They have some awesome courses on their fast.ai website as well.
https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527
path = untar_data(URLs.MNIST_TINY)
# This takes that path from above, and get the path for the threes and the sevens
threes = (path/'train'/'3').ls().sorted()
sevens = (path/'train'/'7').ls().sorted()
Let's take a look at an image. The first thing I reccomend doing for any dataset is to view something to verify you loaded it right. The second thing is to look at the size of it. This is not just for memory concerns, but you want to generally know some basics about whatever you are working with.
im3 = Image.open(threes[1])
im3
tensor(im3).shape
What I am going to do below is put the tensor into a dataframe, and color the pixels based on the value in each place. We can clearly see that this is a 3 just from the values in the tensor. This should give a good idea for the data we are working with and how an image can be worked with.
pd.DataFrame(tensor(im3)).loc[3:24,6:20].style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')
Setup
We will need a wieght matrix with 1 weight per pixel, meaning this will be a 784 row by 1 column matrix. In order to multiply this by our 784 pixels we need to frormat that in 1 row x 784 column matrix. Then we can do matrix multiplication. We are also going to add b, so let's initialize that as well. Since we haven't solved the problem yet, we don't know what good values for w and b are so we will make them random to start with.
def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()
w = init_params((28*28,1))
b = init_params(1)
max(w)
Now we just need x and y. A 784x1 matrix times a 1x784 matrix. We want all values to be between 0 and 1, so we divide by the max pixel value (255)
threes_t = [tensor(Image.open(i)) for i in threes]
sevens_t = [tensor(Image.open(i)) for i in sevens]
# Get my list of tensors and "stack" them. Also dividing by 255 so all values are between 0 and 1
threes_s = torch.stack(threes_t).float()/255
sevens_s = torch.stack(sevens_t).float()/255
#Verify max and min pixel values
torch.min(threes_s), torch.max(threes_s), torch.min(sevens_s), torch.max(sevens_s)
Let's do a simple average of all our threes together and see what we get. It's a nice sanity check to see that we did things ok. We can see that after averaging, we pretty much get a three!
show_image(threes_s.mean(0))
Perfect, lets finish definining our x. We want x to have both threes and sevens, but right now they are seperate. We will use torch.cat to combine them, and .view to change the format of the matrix to the right format. Y is being defined as a long matrix with 1 row per image (prediction) and 1 column.
x = torch.cat([threes_s, sevens_s]).view(-1, 28*28)
# Set my y, or dependent variable matrix. A three will be 1, and seven will be 0. So we will be prediction 0 or 1.
y = tensor([1]*len(threes) + [0]*len(sevens)).unsqueeze(1)
# Combine by xs and ys into 1 dataset for convenience.
dset = list(zip(x,y))
x_0,y_0 = dset[0]
x_0.shape,y_0
Perfect. We have exactly what we need and defined above. A 784 x 1 matrix times a 1 x 784 matrix + a constanct = our prediction. Let's take a look to verify things are the right shape, and if we actually multiply these things together we get 1 prediction per image.
w.shape,x_0.shape,b.shape,y_0.shape
print((x@w+b).shape)
(x@w+b)[1:10]
Great, we have the right number of predictions. Obviosly the predictions are no good at predictions 0 or 1. After all, our weights and biases are all random. Let's do something about that.
We will need to do all this on our validation set as well, so let's do that here.
valid_3_tens = torch.stack([tensor(Image.open(o)) for o in (path/'valid'/'3').ls()])
valid_3_tens = valid_3_tens.float()/255
valid_7_tens = torch.stack([tensor(Image.open(o)) for o in (path/'valid'/'7').ls()])
valid_7_tens = valid_7_tens.float()/255
valid_3_tens.shape,valid_7_tens.shape
valid_x = torch.cat([valid_3_tens, valid_7_tens]).view(-1, 28*28)
valid_y = tensor([1]*len(valid_3_tens) + [0]*len(valid_7_tens)).unsqueeze(1)
valid_dset = list(zip(valid_x,valid_y))
Loss Function
We need to improve our weights and biases (w and b) and we do that using gradient descent. I have a few posts on gradient descent, feel free to check those out if you want details on how it works. Here we will use the built-in pytorch functionality.
The first thing we need to use gradient descent is we need a loss function. Let's use something simple, how far off we were. If the correct answer was 1, and we predicted a 0.5 that would be a loss of 0.5.
The one addition is that we will add something called a sigmoid. All a sigmoid is doing is ensuring that all of our predictions land between 0 and 1. We never want to predict anything outside of these ranges as those are our 2 categories.
def mnist_loss(predictions, targets):
# make all prediction between 0 and 1
predictions = predictions.sigmoid()
# Difference between predictions and target
return torch.where(targets==1, 1-predictions, predictions).mean()
Gradient Descent
Background and Setup
- predict
- calculate loss
- calculate gradient
- subtract from weights and bias
Ok, now we have a function we need to optimize and a loss function to tell us our error. We are ready for gradient descent. Let's create a function to change our weights.
def linear1(xb): return xb@weights + bias
# Here is how we will initialize paramaters. This is just giving me random numbers.
def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()
First, we will make sure our datasets are in a DataLoader. This is convenience class that helps manage our data and get batches..
dl = DataLoader(dset, batch_size=256, shuffle=True)
valid_dl = DataLoader(valid_dset, batch_size=256)
We are now going to get the first batch out. We'll use a batch size of 256, but feel free to change that based on your memory. You can see that we can just simply call first dl, and it creates our shuffled batch for us.
xb,yb = first(dl)
xb.shape,yb.shape
Let's Initialize our paramaters we will need.
weights = init_params((28*28,1))
bias = init_params(1)
preds = linear1(xb)
preds.shape, preds[:5]
Now we calculate our loss to see how we did. Probably not well considering all our weights at this point are random.
loss = mnist_loss(preds, yb)
loss
Let's calculate our Gradients
loss.backward()
weights.grad.shape,weights.grad.mean(),bias.grad
Since we are going to want to calculate gradients every since step, let's create a function that we can call that does these three steps above. Let's put all that in a function. From here on out, we will use this function.
def calc_grad(xb, yb, model):
preds = model(xb)
loss = mnist_loss(preds, yb)
loss.backward()
calc_grad(xb, yb, linear1)
weights.grad.mean(),bias.grad
weights.grad.zero_()
bias.grad.zero_();
def train_epoch(model, lr, params):
for xb,yb in dl:
calc_grad(xb, yb, model)
for p in params:
p.data -= p.grad*lr
p.grad.zero_()
Naturally, we want to be able to measure accuracy. It's hard for us to gauge how well the model is doing from our loss function. We will create an accuracy metric to look at accuracy for that batch.
def batch_accuracy(xb, yb):
preds = xb.sigmoid()
correct = (preds>0.5) == yb
return correct.float().mean()
batch_accuracy(linear1(xb), yb)
Looking at accuracy of our batch is great, but we also want to look at our accuracy for the validation set. This is our way to do that using the accuracy funcion we just defined.
def validate_epoch(model):
accs = [batch_accuracy(model(xb), yb) for xb,yb in valid_dl]
return round(torch.stack(accs).mean().item(), 4)
validate_epoch(linear1)
Awesome! Let's throw this all in a loop and see what we get.
params = weights,bias
lr = 1.
for i in range(20):
train_epoch(linear1, lr, params)
print(validate_epoch(linear1), end=' ')
Linear Recap
Lots of stuff, but let's recap really what we did:
- Make a prediction
- Measure how we did
- Change our weights so we do slightly better next time
- Print out accuracy metrics along the way so we can see how we are doing
This is a huge start, but what we have is a linear model. Now we need to add non-linearities so that we can have a true Neural Network.
What is it?
First, a ReLu is our non-linearity in a neural network. A neural network is just alternating linear and nonlinear layers. We defined the Linear layer above, here we will talk about the non-linear ones. So what exactly does the non-linear layer do? It's actually much simpler than people like to believe, it's just a max.
For example, I am going to apply a ReLu to a matrix.
$\begin{bmatrix}-1&1\\-1&-1\\0&0\\0&1\\1&-1\\1&0\\-1&0\\-1&-1\\1&1\end{bmatrix}$ $=>$ $\begin{bmatrix}0&1\\0&0\\0&0\\0&1\\1&0\\1&0\\0&0\\0&0\\1&1\end{bmatrix}$
As you will see I just took max(x,0). Another way of saying that is I replaced any negative values with 0. That's all a ReLu is.
# The number 30 here can be adjusted for more a less model complexity. Future posts will talk more about what that is.
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)
# This can have more layers by duplicating the patten seen below, this is just the simplest model.
def simple_net(xb):
res = xb@w1 + b1 #Linear
res = res.max(tensor(0.0)) #ReLu
res = res@w2 + b2 #Linear
return res
So we have our new model with new weights. It's more than just the linear model, so how do we use gradient descent? We now have 4 weights (w1,w2,b1,b2).
Turns our it's exactly what we already did. Let's add the new paramaters to our paramsm, and change out the linear model with the simple_net we just defined. We end up with a pretty decent accuracy!
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)
params = w1,b1,w2,b2
lr = 1
for i in range(20):
train_epoch(simple_net, lr, params)
print(validate_epoch(simple_net), end=' ')
x.shape,w1.shape,b1.shape,w2.shape,b2.shape
What More?
This is a Neural Network. Now we can do tweaks to enhance performance. I will talk about those in future posts, but here's a few concepts.
- Instead of a Linear Layer, A ReLu, then a linear layer - Can we add more layers to have a deeper net?
- What if we average some of the pixels in our image together before dong matrix multiplication (ie a convolutions)?
- Can we randomly ignore pixels to prevent overfitting (ie dropout)?
There are many 'advanced' variations of Neural Networks, but the concepts are typically along the lines of above. If you boil them down to what they are really doing - they are pretty simple concepts.