Best Line

the "Best Line" Fit to Data and the Error and Beta a continuation of Part II

We'll summarize what we found in Parts I and II:

We're fitting a straight line, y = Mx + K, to a collection of points (x_n, y_n) ... called the Regression Line.
We're using the notation:
Σx = x₁ + x₂ + ... + x_N
and
Σxy = x₁y₁ + x₂y₂ + ... + x_Ny_N
etc. etc.
M and K are chosen so as to minimize the Mean Squared Error:
Error² = (1/N)Σ{y_n - (M x_n + K) }²

That requirement gives:
M = { N Σxy - Σx Σy } / { N Σx² - ( Σx )² }
K = { Σx² Σy - Σx Σxy } / { N Σx² - ( Σx )² }

Figure 1
We saw that the slope of the "best fit" line can be written:
[1]
M = COVAR[x,y] / SD²[x]
where COVAR[x,y] = (1/N)Σxy - {(1/N)Σx} { (1/N)Σy} = Mean[xy] - Mean[x]Mean[y] is the Covariance of x and y
and SD²[x] = (1/N)Σx² - {(1/N)Σx}² = Mean[x²] - (Mean[x])² is the Variance or (Standard Deviation)² of the set of x's
(See stat-stuff.htm#3)
K = {Mean[x²]Mean[y] - Mean[x]Mean[xy]} / SD²[x]
= {(SD²[x] +(Mean[x])²)Mean[y] - Mean[x](COVAR[x,y] +Mean[x]Mean[y])} / SD²[x]

>Why are you doing this again?
Although we've identified the "best line" fit to the data, we failed to determine the minimum error.

>The minimum error?
Yes, the minimum of Error ... remember? The slope and intercept of the "best line", that's M and K, was chosen to minimize Error.

So we write:

Error² = (1/N)Σ{y - (M x+ K)}² where we're dropping the subscripts for sanitary reasons
= (1/N)Σ{y² - 2y(M x+ K) + (M x+ K)²}
= (1/N)Σy² - 2(M/N)Σxy - 2(K/N)Σy + (M²/N)Σx² + (2MK/N)Σx + (K²/N)Σ(1)

>Ugh.
Do you see all those Means?
>Ugh!
The Error can be expressed in terms of five Means.
In fact, Error can be expressed in terms of the statistical parameters of the x- and y-sets and their Covariance ... like so:
To simplify we'll let:

Mean[x] =X, SD[x] = A
Mean[y] =Y, SD[y] = B
COVAR[x,y] = C
then, using [1]:
Mean[xy] = COVAR[x,y] + Mean[x]Mean[y] = C + XY
Mean[x²] = SD²[x] + (Mean[x])² = A² + X²
Mean[y²] = SD²[y] + (Mean[y])² = B² + Y²
so we can write
M = C / A²
K = ((A²+X²)Y - X(C+XY) ) / A² = (A²Y - CX) / A²

so
Error² = (1/N)Σy² - 2(M/N)Σxy - 2(K/N)Σy + (M²/N)Σx² + (2MK/N)Σx + (K²/N)Σ(1)
= (B²+Y²) -2(C/A²)(C+XY) -2(A²Y - CX)Y/A² +(C/A²)²(A²+X²) +2(C/A²)(A²Y-CX)X/A²+(A²Y-CX)²/A⁴
where Σ(1) = 1+1+1+...+1 = N

>zzzZZZ
Patience! We just simplify. Lots of stuff cancels out and we get (finally!) Error² = B² - C² / A²:
We place this in a position of eminence:
[2]

Error² = SD²[y] - COVAR²[x,y] / SD²[x]
= SD²[y] (1 - COVAR²[x,y] / SD²[x]SD²[y])
= SD²[y] (1 - r²)

>Huh?
[3]
r = COVAR[x,y] / SD[x]SD[y] is the Pearson correlation

The square of r is called ...

>R-squared?
Yes. If the correlation r = 1 or -1, then the Error is zero. The points (x₁, y₁), (x₂, y₂) etc. lie right on that "best line".

r = +1 means perfect (linear) correlation
r = 0 means no correlation
r = -1 means perfect inverse correlation

>And for zero correlation then ... uh ...
Then the Error is just the Standard Deviation of the set of y's.
For example, stare at the charts here
The values of x₁, x₂, etc. are the same for both charts.
In fact, the Pearson correlation is also the same for both charts (namely r = 0.99).
The difference is in the volatility of the set of y's:
SD[y] is larger for the lower chart ... hence the Error is larger.
In fact, it's larger in proportion to the Standard Deviation.
(But that's just because r happens to be the same for both charts.)
In general, changing the Standard Deviation of the y's will change r as well so we can't conclude that the Error is smaller just because r is smaller.
>But it helps.
Yes. It helps.

Figure 2
>So that Error, SD²[y] (1 - r²) ... does it have a name?
Uh ... not that I know of. How about calling it Error?
>Very funny.
Besides, we're calling SD²[y] (1 - r²) the Error² because it's the Mean Squared Error.

One thing that's a little bothersome is that the Error isn't symmetrical in x and y.
>Huh?
Although the correlation r is unchanged if the x's and y's are switched, the Error does change. That seems strange, doesn't it? I mean, if you want to know the error in fitting a straight line to (x,y) data, why should the resultant error depend upon which variable you choose as x and which as y? Taking the vertical distance to that "best line" gives to the y's a special role.
We could introduce symmetry by calculating the average of the two Errors, when the x's and y's are switched:

symmetrical Error² = (1/2)(SD²[x] + SD²[y] )(1 - r²)
Or (and this one I like better), we could take as Error² the Mean Squared perpendicular distance of the points to that "best line". That'd give another symmetrical Error:
[4]
Error² = (1 - r²)SD²[x]SD²[y]/(SD²[x] + SD²[y] )

Figure 3

>So does that Error have a name?
Uh ... not that I know of. How about calling it another ...?
>Forget it. So ... how about the slope and intercept?
You mean M and K? So, what about them?
>Do they change when you interchange x and y?
Uh ... good point. They do change.

Since the covariance, C, doesn't change, M is either C/A² = COVAR[x,y]/SD²[x] or C/B² = COVAR[x,y]/SD²[y].

So the moral of this story is just this:
When doing "best line" fit to (x,y) data, be wise in choosing which variable is x and which is y.

There's this other thing called Beta, namely:
[5]
Beta[x,y] = COVAR[x,y] / SD²[x]

It's used to determine whether two time series (say the monthly S&P 500 returns and the returns for Microsoft) tend to move up or down together and ...

>Hey! That Beta is just the slope of that "best line" fit ... isn't it?
Yes. Beta[x,y] = M.
If the two sets, x and y, are daily (weekly? monthly?) Returns*, then a Beta[x,y] of 1.5 means that the increase in the y-Returns tend to be 1.5 times the increase in the x-Returns. That means that the y-Returns tend to change more than the x-Returns. That means ...

>But that Beta depends upon which set of returns you choose for x and y, right?
Yes. It could be COVAR[x,y] / SD²[x] or COVAR[x,y] / SD²[y] so one normally uses Beta to determine the relationship of stock returns to the Market ... in which case the x-Returns are, say, the TSE300 or the S&P500 or some other "benchmark" set of returns.

>So Beta = 1.5 means your stock is 1.5 times more volatile than the market, eh?
Uh ... not exactly. It's SD that measures volatility, not Beta.
>But I've read that "Beta measures the volatility of a stock compared to the volatility of the market".
Yeah, I've read that too. (Check google.)
In fact, Beta measures how changes in returns are related ... not the returns themselves.
Remember, the slope is (change in y) / (change in x).
For example, if Market returns increase from 8% to 10% (that's a 2% increase) then you might expect your stock returns to increase by 1.5(2) = 3%.
>Assuming Beta = 1.5, right?
Right. We could say that the stock will participate in Market moves, but more or less, depending upon Beta.
Figure 4

In fact, we have:
[6]
Beta[x,y] = r SD[y] / SD[x]
since COVAR[x,y] = r SD[x] SD[y].
So Beta is the ratio of volatilities multiplied by the Pearson correlation.

Here are some examples:

However, if the "best fit" line happens to pass through the origin, then the slope is y / x.

>Pass through the origin? That means the Intercept = 0, right?
Yes, and the intercept also has a name. It's called ...
>It's called K, right?
Well ... uh, investment gurus call it Alpha.

Notice an interesting thing: if we're measuring the Beta of a set of returns with itself (so y = x) then Beta[x,x] = COVAR[x,x] / SD²[x] = 1.
(That just says that the "best fit" line, namely y = x, has slope = 1.)
That means that Beta of the Market is 1 ... since we'd be comparing the Market with itself!

Bloomberg (and others), define Beta as the slope of the "best line" fit when you plot excess returns: the stock against the Market
>Excess?
Yes, the actual return less some risk-free return such as Money Market or maybe T-bills ... but we'll stick with the actual returns and forget using the excess.
>Does it make a difference?
Not really. Subtracting a constant risk-free rate, C, from returns will give the same value for
Beta = COVAR[x,y] / SD²[x] ... since neither COVAR[x,y] nor SD[x] will change when x and y are replaced by x-C and y-C.

>How about an example?
Consider Microsoft vs the S&P 500 (which is our Market).
We look at the daily returns over the last few weeks and get a "best fit" line like Figure 5.
>So Beta = 1.097 ... so MSFT is 1.097 times more volatile, right?
We'll check. The parameters turn out to be:

Correlation r = 0.4818
SD[MSFT] = 1.520% SD[S&P] = 0.668%
Ratio of Volatilities = SD[MSFT] / SD[S&P] = 1.520 / 0.668 = 2.275
... so MSFT was 2.275 times more volatile, over the past few weeks
Beta = 1.097 (the slope of the "best fit" line)
Note that Beta = r SD[MSFT] / SD[S&P] = (0.4818)(2.275) = 1.097

>So why do they say that Beta is a comparison of volatilities?
Probably because it provides a simple explanation ... even though it ain't true

Figure 5

>But if Beta is less than 1, then surely the stock is less volatile than the market, eh?
Think so?
Figure 6 shows the daily returns over the past few weeks for Eastman Kodak versus the S&P.
Beta is less than 1 (it's 0.81) so one might conclude that EK was less volatile than the S&P.
In fact, EK was twice as volatile! Look at that upper chart.
>The ratio of volatilities was 2?
Pretty close. SD[EK] = 1.377% and SD[S&P] = 0.668% and 1.377 / 0.668 = 2.06
In fact, the Correlation between EK and S&P is r = 0.393 which is pretty small (hence decreases the value of Beta). That's why Beta = (0.393) * (Ratio of Volatilites) is smaller than the ratio of volatilities.
>Beta is small ... even tho' the ratio of volatilies is large, eh?
Exactly.
>So why do they say that Beta is a comparison of volatilities?
I give up. Why?

Figure 6
* Beta is usually calculated using Monthly returns ... I think!