Machine Learning for the Blind

I never understood machine learning. They show all of these diagrams of neural networks and I always took that at surface value. So I am going to do a deep dive and actually start understanding these things.

Yup, it's all floats. This diagram you see:

Is a LIE. All the circles are just straight up actual numbers, and the lines are also numbers.

What does this mean? Exactly what it means.

literally

float neuron = 0.5;

That's what those circles are. The same with the lines. They're literally randomly selected floating point numbers (that changes to be meaningful over time) with a tiny little bit of added complexity.

That's what gets you chat gpt and claude. Crazy stuff.

Anyhow, let's "build" a "project". Let's say we wanna have a model that detects whether or not some image is that of a cat. we have multiple inputs. So instead of making more variables by hand, let's use an array:

float inputs[3] = {0.9, 0.3, 0.7}
// very pointy ears, smallish animal, fluff

That's it. The comments is what the inputs represent. On a scale from 0.0 to 1.0, how pointy are the ears? Take a look at this:

Those are very pointy ears. In fact, I would say that they're about 9/10. But since we're talking about 0.0 to 1.0, I'll say 0.9/1.0. I'm assuming it's pretty small (0.3/1.0) and just cuz I drew it and i'm biased, it's 0.7/1.0 fluffy. idk how to draw fluff

Anyhow, now we have 3 inputs.

I'll explain the difference between inputs and neurons a little further down—don't worry about the difference just yet.

Now let's make some weights. A weight is just a number that indicates how relevant each input is.

float weights[3] = {0.8, 0.1, 1.0}

// pointy ears are important (weight[0] = 0.8)
// there are big cats and small cats so this doesn't matter that much (weight[1] = 0.1)
// and fluff is (OBVIOUSLY) very very important (weight[2] = 1.0)

Ok. So what exactly did we just make?? Well, technically we didn't actually make neurons.

float inputs[3] = {0.9, 0.3, 0.7}

These three aren't neurons, they're inputs. It's just cuz we put the numbers (0.9, 0.3, 0.7) in there ourselves. It was never computed by anything else before that, we just thought the ears were pointy and the cat, fluffy.

In order to actually create our first neuron we have to do some computations. Remember the weights?

float weights[3] = {0.8, 0.1, 1.0}

We can use the inputs with the weights and make some calculation that means something:

If the cat's ears are this (0.9) pointy, and pointiness matters this (0.8) much, doesn't it kinda make sense if we just multiply these two numbers?

And then we can do this for all the inputs and their corresponding weights. The cat is this (0.3) big, and size (of the cat) only matters this (0.1) much. The cat is this (0.7) fluffy, and fluffiness matters this(1.0) much.

Now we can add all the multiplied values together like this:

Result = (0.9 * 0.8) + (0.3 * 0.1) + (0.7 * 1.0) = 1.45

It doesn't have to be between 0 and 1 btw these are random values I made up and the result happens to be 1.45. But since this is hard to interpret (145% a cat???) we use something called a sigmoid function to squish the value to a percentage point (between 0 and 1).

This is how the sigmoid works:

// Big positive number -> close to 1
// Big negative number -> close to 0
// zero -> 0.5

float sigmoid(float x) {
    return 1.0 / (1.0 + exp(-x));
}

but that looks kinda scary so

let me put it into math format:

(the lil symbol thing is sigma 🗿)

this function, when graphed, looks like this:

Which means whatever x you input, it'll be between 0 and 1. Higher the x, closer f(x) is to 1. Lower the x into negative, the closer f(x) is to 0.

Anyway now if we put the number we got (1.45) into the sigmoid function we get:

The numbers were rounded but you get the point. Kids' stuff.

Now our model, based on

float inputs[3] = {0.9, 0.3, 0.7}

this and

float weights[3] = {0.8, 0.1, 1.0}

this, thinks that there's an 81% chance that we were in fact describing a cat.

Well, it's not actually a percentage. It's like a 0-1 confidence-ish value. But we're just gonna pretend it's a percentage for the rest of this section to make it more intuitive.

Congratulations! You've just done forward pass on a neural "network" model (it's not actually a network since we have like one neuron, but still)

Here's the code you can play with for your very first, bare bones, fully manual neural "network" model:

import math

inputs = [0.9, 0.3, 0.7]
# pointy ears, size of animal, fluff

weights = [0.8, 0.1, 1.0]
# pointy ears is important, size not so much, fluff very much so

result = 0.0
# init result var

for i in range(len(inputs)):
    result += inputs[i] * weights[i]
    # add (0.9 * 0.8) + (0.3 * 0.1) + (0.7 * 1.0) to results

sigma = (1/(1 + math.e ** (-1 * result)))
# sigmafy results to become percentage

print(sigma)
# print sigmafied result

But something is off... Imagine all the inputs are 0. The ears are not pointy at all, it has 0 size(..?) and it isn't fluffy at all.

(this is our reference image. It's a brick wall. not very "cat")

What would our model think? Let's do the math.

...huh??

When our inputs were all 0's, our model predicts that it's has a 50% chance of it being a cat. That's not right. If it has fully rounded ears, is infinitely small and is not fluffy at all, how could it possibly be a cat?? This means if I literally described a picture of a brick wall and gave our model the inputs, it would predict that it's 50% cat.

Well, it can't be that bad. Let's give it the most cat cat we could possibly find: maximum ear-pointiness, maximum size, and maximum fluffiness. Let's do the math.

(the most cat cat)

...wait what??? that can't be right. That's the most cat looking cat if i've ever seen one.

The maximum probability of something being a cat, according to our model, is 87%. The minimum is 50%.

It seems for any set of inputs, the absolute minimum is 50%, and the absolute maximum is 87%.

That's not good. In short, our model kinda sucks. It's super inaccurate. I guess I mentioned we're gonna make a model, but never really specified that it would be any good.

Let's try to duct tape a solution this problem.

Let's first try to account for the worst case brick-wall scenario.

If all the inputs are 0, we want the result to be (at least close to) 0%. Let's just, from the result of the sum, subtract like 10. So then we get:

Nice! 0.00005 is basically 0. So now, when all the inputs are 0, our model knows that it's almost 0% a cat!

...But what happens to the other side? What if we get the most cat cat? Then in this sequence:

we now subtract 10 like we did before, which becomes

...whoops. Now the most cat cat has a 0% chance of being a cat. That doesn't make sense.

So to prevent this let's add some more duct tape.

Let's just CRANK UP the weights so that the -10 just doesn't effect the final result. All this time, we've been multiplying 0 and 1 by the weights, which were 0.8, 0.1 and 1.0. Let's just crank all of em to 10 and see what happens. We'll try both the brick wall and the super cat.

Well, for the brick wall it doesn't change since it's all 0's. That's a feature. We want that to stay 0 so that the previous math of the brick wall is still 0%.

Let's try the super cat.

and we subtract 10:

let's continue:

Nice! Now the super cat is 99.999...% cat, and the brick wall is 0.00005% cat! What we just added is a bias. In a normal, many neurons example(this whole cat detector thing is like a single neuron and like 3 input values), every neuron would have their own bias that changes based on it's training. We'll talk about it later. This adds literally two lines to our intial code:

import math

inputs = [0.35, 0.35, 0.35]
# pointy ears, size of animal, fluff

weights = [10, 10, 10]
# pointy ears is important, size not so much, fluff very much so

# >>>>>>>>>>>>>>>>>>>> FIRST LINE ADDED HERE >>>>>>>>>>>>>>>>>>>>
bias = -10
# This is initializing what our bias is.
# >>>>>>>>>>>>>>>>>>>> FIRST LINE ENDS HERE >>>>>>>>>>>>>>>>>>>>>

result = 0.0
# init result var

for i in range(len(inputs)):
    result += inputs[i] * weights[i]
    # add (0.35 * 10) + (0.35 * 10) + (0.35 * 10) to results

# >>>>>>>>>>>>>>>>>>>> SECOND LINE ADDED HERE >>>>>>>>>>>>>>>>>>>>
result += bias
# Add the bias to our resulting value
# >>>>>>>>>>>>>>>>>>>> SECOND LINE ENDS HERE >>>>>>>>>>>>>>>>>>>>>

sigma = (1/(1 + math.e ** (-1 * result)))
# sigmafy results to become percentage

print(sigma)
# print sigmafied result

Anyhow, That's perfect. That's exactly what we want!

...right?

Well, we don't actually know, so let's test it a little. We'll do like 0.35 for each input case.

Ok, not bad, not good, idk.

Well, let's try on a 0.25 case. 0.35 was a weird number.

Wait what?? When all the inputs were 0.35, there was a 62.5% chance that this thing was a cat. When we get a thing that's slightly less a cat, 0.25, the probability that it's a cat becomes 8%. That is also a problem. And it's a problem that we can't solve with duct tape anymore. This is the reason it's called machine learning - we can no longer keep shifting values until something works. And we can't keep hand-picking inputs and biases randomly either. That leads us to our next section:

Now, we have to actually train the model. Right now all we've been doing is hand-picking some random numbers and doing some math to get some result-we've been learning about forward pass.

But now, we need more neurons, and more ways to properly calibrate these numbers into something meaningful so that we can make them as accurate as possible.

But how do we actually train the model? How do we know how to tweak what values to what, by how much, and why? To know this we first have to understand the idea of a label, and also loss. These are very simple ideas.

label is the correct answer.

this is 100% a cat, and thus the label should be 1.0 in our example. And IF our current model were to predict that this is 42% likely to be a cat, it would be 58% wrong. That is our loss. The lower the loss the better-since 0% wrong would be 100% right.

In short, our label is the objective answer to a question, and the loss is how wrong our model was in its guess.

label = 1.0
model_prediction = 0.42
loss = label - model_prediction

print(loss)
# 0.58

Yes, I know this cat example is getting boring... but hang on for a second-in my research I found a perfect excuse to move on to a different joke, but only after this one small idea you gotta understand.

Take this picture. We'll label it like 80% cat (the other 20% is cheetos)

label = 0.80

And then let's compute it using our model we made before. This will be the cranked up version.

inputs = [0.2, 1.0, 0.5]
# it's ears are like 20% pointy, it's 100% big, and it's 50% fluffy.

weights = [10, 10, 10]
# the cranked up weights

Ok. Now let's run it.

Well, I guess it did a pretty decent job. But we wanted 80%, not 99.91%! It forgot to account for the 20% cheetos.

Let's check the loss.

Wait a minute. The loss is negative. Then since the loss is -19.91%, are we 119.91% correct??

(no)

we need a way to interpret this negative number. Your immediate instinct might be to just take the absolute value of this number and move on. Which is a great instinct! That was my instinct, actually, which is why I said it was great.

But apparently not.

This is something I'll teach you in more detail a little later, but what you need to know is that we tend to square the loss, not only to get rid of the negative but to emphasize the bigger issues.

For example, if our loss is 0.9, and we square it, it becomes 0.81. If our loss is 0.1, and we square it, it becomes 0.01.

See the difference? The bigger losses get shrunk down less, and we can tend to the bigger issues which are a lot more "visible" compared to the others.

You don't really need to understand how everything works right now-just understand the vibe of what's going on. Since bigger numbers stay bigger, they're emphasized as bigger problems. Since smaller problems become even smaller, we can ignore them for now and tend to the bigger ones. ("problems" in this context is just referencing loss).

So let's square it.

Not the biggest problem it seems, but it's the only one we have right now, so let's try fix it. How?

First, let's try the duct tape method.

Let's have a "nudge" value:

nudge = 0.1

And then let's just either add or subtract it from our weights!

Let's do the second one, since that one has the highest input.

weights = [10, 10.1, 10]

Whoops! We accidently seem to have made loss a TINY bit bigger.

This was the previous number for comparison:

And this is the new number:

What we wanted was 0.80.

Our loss went up by a fraction of a decimal.

So instead of nudging it up into 10.1, let's nudge it downwards, to get 9.9.

WOW! YOU SEE THAT?!

Our loss went down. By a fraction of a decimal.

Here's the previous number for comparison.

Here's the new number:

We wanted 0.80. So our loss went down.

Bro I am NOT doing allat for every weight and like increasing and decreasing the nudge value and shi 💀🥀🪦💔

In fact, it's slow for us, and it's even slow for computers. Computers are working on literally hundreds of billions to trillions of weights and biases. They are NOT going to sit there manually testing and calculating the numbers for every individual neuron, like we did above.

Because imagine we have a neuron where the input is the output of another neuron. We would literally have to re-calculate every single neuron's value and see the final result and see if our loss went down. Even with computers, that's impossible at the scale humanity has achieved thus far.

So what do we do to make this drastically faster? Well, we use all mighty calculus.

Yeah ik I hate calculus too. But it's necessary. When you asked the school teacher when you'd ever use this stuff, this is an example of when.