Updated 10-14-2002

The Normal distribution

Univariate Normal distribution, $X \sim N(\mu, \sigma^2)$

$$f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{- \frac{(x-\mu)^2}{2 \sigma^2}}$$

Multivariate Normal distribution $\underline{X} \sim N_p(\underline{\mu}, \underline{\Sigma})$

$$f(x) = \frac{1}{(2 \pi)^{\frac{p}{2}} \vert\underline{\Sigma}... ...ime \underline{\Sigma}^{-1} (\underline{x} - \underline{\mu})}$$

Normal distribution and the likelihood function


Recall that maximum likelihood estimation asks us to find the $\mu$and $\Sigma$ which maximize the likelihood L (or the log likelihood).

Given n i.i.d. random vectors $\underline{X}_i \sim N(\underline{\mu}, \underline{\Sigma})$ the log likelihood is

$$log L = \frac{pn}{2} \log {2\pi} - \frac{n}{2} \log \vert\und... ...me \underline{\Sigma}^{-1} (\underline{x}_i - \underline{\mu})$$

After a bit of matrix algebra, this can be transformed into

$$log L = constant + \frac{n}{2} \left[ \log \vert\underline{\S... ...nderline{\Sigma}^{-1} (\bar{\underline{x}} - \underline{\mu})$$

Since $\underline{\mu}$ only appears in the last term, we can maximize the likelihood with respect to $\underline{\mu}$ by minimizing the last term with respect to $\underline{\mu}$. This is clearly done when $\underline{\hat{\mu}} = \bar{\underline{x}}$

So what we are left with doing is maximizing the remaining part with respect to the parameters in $\underline{\Sigma}$. Recall that in the exploratory factor analysis $\underline{\Sigma} = \underline{\Lambda} \underline{\Lambda}^{\prime} + \underline{\Psi}$. So we want to maximize the following with respect to $\Lambda$ and $\Psi$.

$$log L = \frac{n}{2} \left[ \log \vert\underline{\Sigma}^{-1}\vert - trace({\bf S} \underline{\Sigma}^{-1})\right]$$

Maximizing the Likelihood

• The maximization is done by iteration.
• Details of it are probably not that interesting for most in the class.
• Needs starting values the same way that Principal Factor Analysis did but uses these starting values differently
• For those who are interested: The iteration starts by taking the eigenvalues of the following: $(\hat{\Psi}^s_{(0)})^{-\frac{1}{2}} {\bf R} (\hat{\Psi}^s_{(0)})^{-\frac{1}{2}} - {\bf I_{(p \times p)}}$This is implicit in the discussion of p. 47 of BK. It is the eigenvalues of this that appear on the first page of the PROC FACTOR output.
• I don't think I can provide any simple intuition about this iterative technique. Just trust that it converges to the Maximum Likelihood Estimators. Call them $\hat{\Lambda}$ and $\hat{\Psi}$, so $\hat{\underline{\Sigma}} = \hat{\underline{\Lambda}} \hat{\underline{\Lambda}}^{\prime} + \hat{\underline{\Psi}}$.

GOODNESS of FIT TEST
BESIDES GIVING ESTIMATES for $\Lambda$ and $\Psi$, Maximum Likelihood Provides a GOODNESS of FIT TEST.

• Use Likelihood Ratio Test, i.e.,

  
  

  $$\begin{eqnarray*}& & \hspace{-4 cm} -2 \log \frac{\mbox{L(MLE of restricted mode... ...} - \log \vert\hat{\underline{\Sigma}}^{-1} {\bf S}\vert - p \} \end{eqnarray*}$$
  
  

• IF the model fits the data well, this statistic should be small!

• Its distribution is asymptotically distributed $\chi^2$ with degrees of freedom = ( $\frac{p(p+1)}{2}$ - number of free parameters in model).

• We can determine if the the statistic is ``small'' enough by comparing to the $\chi^2$ distribution and obtaining a p-value.

• The Hypothesis being tested

  H₀: The model is correct (i.e. q factors sufficiently describe the p dimensional vector)

  H_A: More factors are needed

• Thus we are looking for the model where we DO NOT REJECT the H₀ (i.e. find a big p-value)

ROTATION

No matter what estimation procedure, for exploratory factor analysis we get estimates that look like:

$$\hat{\underline{\Sigma}} = \hat{\underline{\Lambda}} \hat{\underline{\Lambda}}^{\prime} + \hat{\underline{\Psi}}$$

We can get the exact same $\hat{\underline{\Sigma}}$ by taking

$$\hat{\underline{\Sigma}} = \hat{\underline{\Lambda}} T T^\prime \hat{\underline{\Lambda}}^{\prime} + \hat{\underline{\Psi}}$$

where T is a $q \times q$ matrix such that $T T^\prime$ = Identity matrix.

Thus

$$\hat{\underline{\Sigma}} = \hat{\underline{\Lambda}}^* {\hat{\underline{\Lambda}}}^{* \; \prime} + \hat{\underline{\Psi}}$$

where $\hat{\underline{\Lambda}}^*$ is the orthogonally rotated factor loading matrix, i.e. $\hat{\underline{\Lambda}}^* = \hat{\underline{\Lambda}} T$ . BUT, There are an infinite number of matrices T that satisfy $T T^\prime$ = I.

• Why do we bother to rotate? - We want to find out which variables ``stick together''
• How to choose a T? - Find one that gives the ``simplest'' structure to the factor loadings

ROTATION

• What about if simple structure is not found by orthogonal rotation?

• NOTE: implicitly orthogonal rotation keeps the assumption that $Var(\underline{f}) = I$, i.e. the factors are uncorrelated.

• Maybe the assumption that $Var(\underline{f}) = I$ should be dropped. This leads to oblique rotations of the factor loadings.

• We can search for
  
  $$\hat{\underline{\Sigma}} = \hat{\underline{\Lambda}^{**}} \Ph... ...t{\underline{\Lambda}}^{** \; \prime} + \hat{\underline{\Psi}}$$
  
  

  so that $\hat{\underline{\Lambda}^{**}}$ has ``simple structure'' and $\Phi \ne I$

Determining which variables measure which factors

• Examine the rotated factor loadings

• Rule of thumb, if the absolute value of the standardized loading is >.3, the variable is relevant for the particular factor.

• Do the variables that load on a given factor share some conceptual meaning?

• Examine the communality. Rule of thumb, if the communality is less than .10 then the observed variable can be deleted. A communality < .10 means that less than 10% of the variability in the observed variable is expained by all of the common factors.

• Cudeck, R. and O'Dell, L. (1994) "Applications of standard error estimates in unrestricted factor analysis: Significance Tests for factor loadings and correlations" Psychological Bulletin This paper argues that the >.3 cutoff is too simplistic and provides a method for significance testing of rotated loadings.

Degrees of freedom in the exploratory factor analysis model

• $\Lambda$ has p*q parameters

• $\Psi$ has p parameters

• Recall that the degrees of freedom equal p*(p+1)/2 minus the number of parameters that are being estimated freely.

• Because of rotation, a set of restrictions needs to placed on the parameters so that the software knows which version of $\Lambda$ to estimate. SAS Proc Factor assumes that $\Lambda^\prime \Psi^{-1} \Lambda$ is diagonal. (i.e. it places q*(q-1)/2 restrictions on the parameters in $\Sigma$.

• So d.f. = p*(p+1)/2 - (p*q + p) + q*(q-1)/2

SAS Proc Factor Example using the visual perception/ reading skills data - Handouts in class.

1. Output from Proc Factor (handout)

2. Picturing rotation (handout)

Degrees of freedom for the visual perception/ reading skills data using exploratory factor analysis with 2 factors:

p = 6 (observed variables)

q = 2 (latent factors)

d.f. = 6*7/2 - (6*2+6) + 2*1/2 = 21 - 18 + 1 = 4