Updated 9-24-02

FACTOR ANALYSIS MODEL
A bunch of Factor Analysis exmples can be found at http://www.psych.yorku.ca/lab/psy6140/ex/factor.htm

$$\underline{X} = \underline{\mu} + \underline{\Lambda} \underline{f} + \underline{\epsilon}$$

$\underline{X}$:

p - dimensional observed vector

$\underline{f}$:

q - dimensional underlying factors. $\underline{f}$ are often called ``common factors''. Assume for now $E (\underline{f}) = \underline{0}$ and $Var (\underline{f}) = {\bf I}_{q \times q}$

$\underline{\epsilon}$:

p - dimensional random error. $\underline{\epsilon}$ are often called ``unique factors'' or ``specific factors''. Assume $E (\underline{\epsilon}) = \underline{0}$ and $Var (\underline{\epsilon}) = \underline{\Psi}$where $\underline{\Psi}$ is a diagonal matrix.

$\underline{\Lambda}$:

$p \times q$ matrix of scalars called ``factor loadings''

$\underline{\mu}$:

$p \times 1$ vector of scalar intercepts. Often ignored if we are only interested in interrelations. Most software assume $\underline{\mu}= \underline{0}$ by default and center the $\underline{X}$ variables, i.e. analyze $\underline{X} - \underline{\bar{X}}$

KEY ASSUMPTIONS

1.

$COV(\underline{f},\underline{\epsilon}) = 0_{q \times p}$.

• factors are uncorrelated with the error terms

2.

$\underline{\Psi}$ is a diagonal matrix.

• Recall $Var (\underline{\epsilon}) = \underline{\Psi}$
• Each individual error term is uncorrelated with all the others.

• Thus, given that we know $\underline{f}$, the observed variables are uncorrelated, i.e. the partial correlations of any of the observed variables on the others is zero once we have adjusted for the factors.
• Basic idea underlying most latent variable models called the AXIOM of CONDITIONAL INDEPENDENCE

Focus on Covariance structure

$$Var(\underline{X}) = \underline{\Sigma} = \underline{\Lambda} \underline{\Lambda}^\prime + \underline{\Psi}$$

• $\underline{\Sigma}$ is called the model covariance matrix
• $\underline{\Sigma}$ is a function of all the model parameter that make up the covariance structure
• We will estimate $\underline{\Sigma}$ using ${\bf S}$, the sample covariance matrix

OR Standardize $\underline{X}$ to get $\underline{Z}$, i.e. $\underline{Z} = \left( \begin{array}{c} \frac{x_1 - \bar{x}_1}{s_1} \\ \frac{x_2 - \bar{x}_2}{s_2} \\ . \\ . \\ \frac{x_p - \bar{x}_p}{s_p} \end{array} \right)$

$$Var(\underline{Z}) = \underline{\rho} = \underline{\Lambda}^s {\underline{\Lambda}^s}^\prime + \underline{\Psi}^s$$

• $\underline{\Lambda}^s$ are the ``standardized factor loadings''
• We will estimate $\underline{\rho}$ using ${\bf R}$, the sample correlation matrix

NOTE: There are several different estimation procedures (e.g. 1. principal factor method, 2. normal theory maximum likelihood method, and others). For the maximum likelihood method, $\underline{\Lambda}^s$ obtained by analyzing the correlation matrix is the same as rescaling the $\underline{\Lambda}$ obtained by analyzing the covariance matrix by the observed standard deviations, i.e. $\underline{\Lambda}^s = (diag({\bf S}))^{-\frac{1}{2}} \underline{\Lambda}$. (For a discussion, see section 3.17 in Bartholomew and Knott 1998) THIS IS NOT TRUE WHEN THE PRINCIPAL FACTOR METHOD IS USED.

COMMUNALITIES

• Communalities are the diagonal elements of $\underline{\Lambda} {\underline{\Lambda}}^\prime$ or $\underline{\Lambda}^s {\underline{\Lambda}^s}^\prime$
• Communalities are the part of the variance of each observed variable which is due to the q underlying factors

$$\begin{eqnarray*}x_1 &=& \mu_1 + \lambda_{11} f_1 + \lambda_{12} f_2 + \epsilon_... ...&=& \mu_5 + \lambda_{51} f_1 + \lambda_{52} f_2 + \epsilon_5 \\ \end{eqnarray*}$$

$$\begin{eqnarray*}Var(x_1) &=& \lambda_{11}^2 + \lambda_{12}^2 + \psi_1 \\ Var(x... ..._4 \\ Var(x_5) &=& \lambda_{51}^2 + \lambda_{52}^2 + \psi_5 \\ \end{eqnarray*}$$

EXAMPLE: The communality for x₂ is $\lambda_{21}^2 + \lambda_{22}^2$

• Discuss difference between PCA and FA from the Hatcher Handout.

PARAMETER ESTIMATION

We want to analyze the covariance structure of the factors, i.e. $\underline{\Lambda} {\underline{\Lambda}}^\prime$. We want to estimate $\underline{\Lambda}$.

TWO estimation procedures will be discussed (Only one will be discussed in 2002 school year)

1.

Principal factor method (won't discuss details of this method)

• Default method in SAS Proc FACTOR (probably SPSS too)
• Computationally simpler, historically this method came first
• Discussed in sections 3.10-3.15 of BK

2.

Normal theory Maximum likelihood method

• Discussed in sections 3.5-3.9 of BK
• Prefered method in general

SKIP in 2002.....PARAMETER ESTIMATION - VIA Principal factor method

Some NOTES

• The rank of $\underline{\Lambda} {\underline{\Lambda}}^\prime$is equal to q (i.e. equal to the number of underlying factors). This implies that (p-q) eigenvalues of $\underline{\Lambda} {\underline{\Lambda}}^\prime$ are zero.
• PCA (or eigenvalue/eigenvector decomposition) allowed us to take a matrix and factor it into $UDU^\prime$, well this can be written as $UDU^\prime = UD^{\frac{1}{2}}D^{\frac{1}{2}}U^\prime = AA^\prime$
• Thus we will use this idea to find $\underline{\Lambda} {\underline{\Lambda}}^\prime$ by using PCA on ${\bf S} - \hat{\underline{\Psi}}$ or ${\bf R} - \hat{\underline{\Psi}}^s$

Since we don't know $\Psi$, we will start off with a guess and then iterate.....

SKIP in 2002......Need a first guess for $\Psi$

• call it $\hat{\Psi}_{(0)}$
• Recall that there are p nonzero elements in $\Psi$, i.e. a uniqueness for each error term
• Basically can use anything bigger than 0 and less than $diag({\bf S})$ for our first guess.
• By default SAS PROC FACTOR analyzes the correlation matrix so actually we are finding a first guess for $\hat{\Psi}^s$
• A common choice is to takes $diag({\bf I} - diag(SMC))$ as $\hat{\Psi}_{(0)}^s$. If you are analyzing the covariance matrix,
  use $diag({\bf S})({\bf I} - diag(SMC))$ as $\hat{\Psi}_{(0)}$.
  • SMC's are the squared multiple correlations. That is the R² resulting from a regression of each variable on all the rest.
  • SAS calls these the prior communalities

SKIP in 2002..... ITERATIVE PROCEDURE

1.

Extract q eigenvectors from ${\bf R} - \hat{\underline{\Psi}}_{(t)}^s$

2.

Call these q eigenvectors $\hat{\underline{\Lambda}}^s_t$

3.

Re estimate $\hat{\underline{\Psi}}^s$ by taking $diag({\bf I} - diag({\hat{\underline{\Lambda}}^s_t \hat{\underline{\Lambda}}^s_t}^\prime))$

4.

Label this new estimate of $\hat{\underline{\Psi}}^s$ as $\hat{\underline{\Psi}}_{(t+1)}^s$

5.

Go back to step 1 and continue iterating until convergence

SKIP in 2002......INTERESTING NOTE about Principal factor method

BK (sections 3.10, 3.11) show that the principal factor method is identical to solving for $\Lambda$ and $\Psi$ such that

$$\sum_{i=1}^{p} \sum_{u=1}^{p} (s_{iu} - \sigma_{iu})^2$$

is as small as possible.

This is the same thing as solving for $\Lambda$ and $\Psi$ such that

$$trace ({\bf S} - \underline{\Sigma})^2$$

is as small as possible. This is the Ordinary Least Squares discrepancy function.

Note that this minimization criterion does not assume anything about the correlations between the elements of ${\bf S}$. Hence we are ignoring information.