path: root/notebook/ft_linear_regression_notebook.md

# ft_linear_regression

Project in which I need to build a linear regression model for a simple dataset.


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

data = np.genfromtxt('../data.csv', delimiter=',')[1:]
km = data[:, 0]
price = data[:, 1]
```


```python
def plot_data():
    plt.scatter(km, price)
    plt.xlabel("km")
    plt.ylabel("price")
plot_data()
plt.show()
```


![png](ft_linear_regression_notebook_files/ft_linear_regression_notebook_2_0.png)


This dataset contains information about cars, their cost and the the number of kilometers they've made.

We can see just by looking at the data that their is a **linear relationship**, when the distance made diminish, the price rize.

We can try to trace an approximate line which goes *through* the data.


```python
plot_data()
plt.plot([200_000, 20_000], [3_500, 8_500], color='r')
plt.show()
```


![png](ft_linear_regression_notebook_files/ft_linear_regression_notebook_4_0.png)


We can use this hypothetical line to predict new data, for example predict the price of a car knowing only it's distance traveled.

The equation of a line is in the form:

$$
\boxed {
    y = \theta_1 x + \theta_0
}
$$

Where $\theta_1$ is the slope of the line and $\theta_0$ the y-intercept. If we managed to tweak those values correctly we can find the best fitting line for our data (or at least very close to it).

If we switch to a function, we get $h(x) = \theta_1 x + \theta_0$, $h$ is our **hypothesis function**.


```python
def h(x, theta1, theta0):
    return x * theta1 + theta0

theta1 = -0.03
theta0 = 9000

line_xs = [km.min(), km.max()]
line_ys = [h(x, theta1, theta0) for x in line_xs]
plt.plot(line_xs, line_ys, color='r')

plot_data()
plt.show()
```


![png](ft_linear_regression_notebook_files/ft_linear_regression_notebook_6_0.png)


We can adjust our prediction by changing $\theta_1$ and $\theta_0$.

Now, we need a way to know if our line correctly predicts the data or not, so that if not, we can slightly change it to better fit the data.

Introducing the **error function** $J$ (also called cost function):

$$
\begin{align}
J(\theta_1, \theta_0) &= \frac{1}{2n} \sum_{i=1}^{n}(h(x_i) - y_i)^2 \\
                      &= \frac{1}{2n} \sum_{i=1}^{n}((\theta_1 x_i + \theta_0) - y_i)^2
\end{align}
$$

It's the sum of the square of the distance between what we predicted and the actual data.
$n$ is the number of entry we got, we divide by it to normalize the result.


```python
def error_function(xs, ys, theta1, theta0):
    return (1 / (2 * len(xs))) * sum(
        [(h(x, theta1, theta0) - y) ** 2
         for x, y in zip(xs, ys)])
```


```python
xs = km
ys = price

print(error_function(xs, ys, theta1, theta0))
```

    386217.6189791666


The error value is pretty high, this means that the approximated line is garbage.

The process of learning will be done by reducing this error function. To make that happen, we need to know *how* we are going to change $\theta_1$ and $\theta_0$.

Here comes partial derivatives:

$$
\begin{align}
    \theta_1 &:= \theta_1 - \alpha \frac{\partial}{\partial \theta_1} J(\theta_1, \theta_0) \\
    \theta_0 &:= \theta_0 - \alpha \frac{\partial}{\partial \theta_0} J(\theta_1, \theta_0)
\end{align}
$$

First of all $\alpha$ is the **learning rate** we multiply the change we want to make by it, it describes how *fast* we are changing the hypothesis component.

The partial derivative of $J$ by regard to $\theta_1$ for example is how the error function changes if we consider $\theta_0$ has a constant.

We compute the *next* $\theta_0$ by subtracting learning rate * partial derivative from it.

Lets expand theses partial derivatives:

$$
\newcommand{\partialthetazero}{\frac{\partial}{\partial \theta_0}}
\begin{align}
    &  \partialthetazero \left[ J(\theta_1, \theta_0) \right] \\
    =& \partialthetazero \left[ \frac{1}{2n} \sum_{i=1}^{n}((\theta_1 x_i + \theta_0) - y_i)^2 \right] \\
    =& \frac{1}{2n} \sum_{i=1}^{n} \partialthetazero \left[ ((\theta_1 x_i + \theta_0) - y_i)^2 \right] \\
    =& \frac{1}{2n} \sum_{i=1}^{n} 2((\theta_1 x_i + \theta_0) - y_i) \partialthetazero \left[ ((\theta_1 x_i + \theta_0) - y_i) \right] \\
    =& \boxed{ \frac{1}{n} \sum_{i=1}^{n} ((\theta_1 x_i + \theta_0) - y_i) }
\end{align}
$$

And for $\theta_1$:

$$
\newcommand{\partialthetaone}{\frac{\partial}{\partial \theta_1}}
\begin{align}
    &  \partialthetaone \left[ J(\theta_1, \theta_0) \right] \\
    =& \partialthetaone \left[ \frac{1}{2n} \sum_{i=1}^{n}((\theta_1 x_i + \theta_0) - y_i)^2 \right] \\
    =& \frac{1}{2n} \sum_{i=1}^{n} \partialthetaone \left[ ((\theta_1 x_i + \theta_0) - y_i)^2 \right] \\
    =& \frac{1}{2n} \sum_{i=1}^{n} 2((\theta_1 x_i + \theta_0) - y_i) \partialthetaone \left[ ((\theta_1 x_i + \theta_0) - y_i) \right] \\
    =& \boxed { \frac{1}{n} \sum_{i=1}^{n} ((\theta_1 x_i + \theta_0) - y_i) x_i }
\end{align}
$$


```python
def theta0_partial(xs, ys, theta1, theta0):
    return sum([h(x, theta1, theta0) - y
                for x, y in zip(xs, ys)]) / len(xs)

def theta1_partial(xs, ys, theta1, theta0):
    return sum([(h(x, theta1, theta0) - y) * x
                for x, y in zip(xs, ys)]) / len(xs)


def gradient_descent(xs, ys, theta1, theta0, alpha, iterations):
    for _ in range(iterations):
        next_theta1 = theta1 - alpha * theta1_partial(xs, ys, theta1, theta0)
        next_theta0 = theta0 - alpha * theta0_partial(xs, ys, theta1, theta0)
        theta1 = next_theta1
        theta0 = next_theta0
    return theta1, theta0

```

The gradient descent work by repeating the formula above a certain amount of times.

But before actually running the algorithm, the data need to be normalized. This means that they will be in a range like 0 < x < 1 but they will keep their relative distance to each other.

If we don't do this, our values will go to infinity or some do some sneaky floating point arithmetic stuff.


```python
from sklearn.preprocessing import normalize

X = normalize([xs])[0]
Y = normalize([ys])[0]
```


```python
theta1, theta0 = np.random.randn(2)

line_xs = [X.min(), X.max()]
line_ys = [h(x, theta1, theta0) for x in line_xs]
plt.plot(line_xs, line_ys, color='r')

plt.scatter(X, Y)
plt.show()
```


![png](ft_linear_regression_notebook_files/ft_linear_regression_notebook_14_0.png)



```python
print("error before: ", error_function(X, Y, theta1, theta0))
```

    error before:  0.20767778959463673



```python
alpha = 1
iterations = 1000

theta1, theta0 = gradient_descent(X, Y, theta1, theta0, alpha, iterations)

print(theta1, theta0)
print("error after: ", error_function(X, Y, theta1, theta0))
```

    -0.3766258389374899 0.2684839986208806
    error after:  0.00022231909239872055



```python
line_xs = [X.min(), X.max()]
line_ys = [h(x, theta1, theta0) for x in line_xs]
plt.plot(line_xs, line_ys, color='r')

plt.scatter(X, Y)
plt.show()
```


![png](ft_linear_regression_notebook_files/ft_linear_regression_notebook_17_0.png)