Question

Consider the simple linear regression model Yi=β0+β1xi+ϵi, where ϵi's are independent N(0,σ2) random variables. Therefore, Yi is a normal random variable with mean β0+β1xi and variance σ2. Moreover, Yi's are independent. As usual, we have the observed data pairs (x1,y1), (x2,y2), ⋯⋯, (xn,yn) from which we would like to estimate β0 and β1. In this chapter, we found the following estimators β1^=sxysxx,β0^=Y¯¯¯¯−β1^x¯¯¯. where sxx=∑i=1n(xi−x¯¯¯)2,sxy=∑i=1n(xi−x¯¯¯)(Yi−Y¯¯¯¯). Show that β1^ is a normal random variable. Show that β1^ is an unbiased estimator of β1, i.e., E[β1^]=β1. Show that Var(β1^)=σ2sxx.

Answer 1

Answer:

See proof below.

Step-by-step explanation:

If we assume the following linear model:

$y = \beta_o + \beta_1 X +\epsilon$

And if we have n sets of paired observations $(x_i, y_i) , i =1,2,...,n$ the model can be written like this:

$y_i = \beta_o +\beta_1 x_i + \epsilon_i , i =1,2,...,n$

And using the least squares procedure gives to us the following least squares estimates $b_o$ for $\beta_o$ and $b_1$ for $\beta_1$  :

$b_o = \bar y - b_1 \bar x$

$b_1 = \frac{s_{xy}}{s_xx}$

Where:

$s_{xy} =\sum_{i=1}^n (x_i -\bar x) (y-\bar y)$

$s_{xx} =\sum_{i=1}^n (x_i -\bar x)^2$

Then $\beta_1$ is a random variable and the estimated value is $b_1$. We can express this estimator like this:

$b_1 = \sum_{i=1}^n a_i y_i$

Where $a_i =\frac{(x_i -\bar x)}{s_{xx}}$ and if we see careful we notice that $\sum_{i=1}^n a_i =0$ and $\sum_{i=1}^n a_i x_i =1$

So then when we find the expected value we got:

$E(b_1) = \sum_{i=1}^n a_i E(y_i)$

$E(b_1) = \sum_{i=1}^n a_i (\beta_o +\beta_1 x_i)$

$E(b_1) = \sum_{i=1}^n a_i \beta_o + \beta_1 a_i x_i$

$E(b_1) = \beta_1 \sum_{i=1}^n a_i x_i = \beta_1$

And as we can see $b_1$ is an unbiased estimator for $\beta_1$

In order to find the variance for the estimator $b_1$ we have this:

$Var(b_1) = \sum_{i=1}^n a_i^2 Var(y_i) +\sum_i \sum_{j \neq i} a_i a_j Cov (y_i, y_j)$

And we can assume that $Cov(y_i,y_j) =0$ since the observations are assumed independent, then we have this:

$Var (b_1) =\sigma^2 \frac{\sum_{i=1}^n (x_i -\bar x)^2}{s^2_{xx}}$

And if we simplify we got:

$Var(b_1) = \frac{\sigma^2 s_{xx}}{s^2_{xx}} = \frac{\sigma^2}{s_{xx}}$

And with this we complete the proof required.