Draft:Invariant coordinate selection

Invariant coordinate selection (ICS) is a multivariate statistical technique used to identify interesting structures in high-dimensional data. It is commonly applied in outlier detection, independent component analysis (ICA), and robust dimension reduction. ICS generalizes principal component analysis (PCA) by simultaneously diagonalizing two scatter matrices, typically the sample covariance matrix and a higher-order or robust scatter matrix.

Overview

Unlike PCA, which depends on the scale of the variables and captures only second-order structure, ICS is invariant under full-rank affine transformations of the data and can reveal non-Gaussian features of the underlying distribution. The core idea is to find a linear transformation that simultaneously diagonalizes two scatter matrices, yielding coordinates whose ordering reflects the disagreement between the two scatter measures. The method draws heavily on the robust statistics literature for its repertoire of scatter functionals.

History

The simultaneous use of two scatter matrices for data transformation predates ICS and originated in the clustering literature. Art, Gnanadesikan and Kettenring (1982) introduced a data-based metric for cluster analysis in which an estimate of the within-cluster covariance matrix is computed iteratively without knowledge of the cluster labels, and the data are then rescaled by its inverse square root. This procedure, implemented as PROC ACECLUS in SAS, is generally regarded as the earliest instance of the simultaneous-diagonalization idea later generalized by ICS. Caussinus and Ruiz-Gazen (1990) developed a closely related generalized principal component analysis for projection pursuit and cluster identification.

In parallel, the same algebraic structure was being investigated in signal processing under the headings of independent component analysis and blind source separation. Cardoso's Fourth-Order Blind Identification (FOBI) jointly diagonalizes the covariance matrix and a fourth-moment scatter matrix to recover independent sources. Oja and collaborators subsequently showed that any pair of affine-equivariant scatter matrices possessing the so-called independence property can be used to recover an independent component model, placing FOBI and related procedures in a common statistical framework.

A third influence comes from the robust statistics community, where most of the alternative scatter functionals later adopted by ICS were originally introduced—including the M-estimators of multivariate scatter studied by Maronna, the distribution-free M-estimator of Tyler, the symmetrized M-estimator of Dümbgen, and the minimum covariance determinant (MCD) estimator of Rousseeuw.

These threads were unified by Tyler, Critchley, Dümbgen and Oja (2009), who introduced the name invariant coordinate selection, characterized its affine-invariance properties, and established it as a general-purpose exploratory method that subsumes the earlier proposals as special cases.

Definition

Let $X=(x_{1},\dots ,x_{n})^{T}\in \mathbb {R} ^{n\times p}$ be a sample of $n$ observations in $\mathbb {R} ^{p}$, with sample mean ${\bar {x}}$.

A scatter matrix is a $p\times p$ symmetric positive-definite matrix-valued functional $S(X)$ of the data that is affine equivariant, meaning that for any non-singular matrix $A$ and vector $b$,

$S(XA+\mathbf {1} _{n}b^{T})=A^{T}\,S(X)\,A.$

The sample covariance matrix is the canonical example, but many alternatives drawn from the robust statistics literature are available (see below).

Given two scatter matrices $S_{1}(X)$ and $S_{2}(X)$, ICS solves the generalized eigenvalue problem

$S_{2}(X)\,\mathbf {v} =\lambda \,S_{1}(X)\,\mathbf {v} ,$

producing eigenvalues $\lambda _{1}\geq \dots \geq \lambda _{p}>0$ and a matrix $V=(\mathbf {v} _{1},\dots ,\mathbf {v} _{p})$ of corresponding eigenvectors normalized so that $V^{T}S_{1}(X)V=I_{p}$. The invariant coordinates are the columns of

$Z=(X-\mathbf {1} _{n}{\bar {x}}^{T})\,V.$

The eigenvalues $\lambda _{k}$ quantify how strongly the two scatter matrices disagree along each direction. For any unit vector $\mathbf {u}$, the ratio $\mathbf {u} ^{T}S_{2}(X)\mathbf {u} \,/\,\mathbf {u} ^{T}S_{1}(X)\mathbf {u}$ can be interpreted as a generalized kurtosis measure along that direction, so the invariant coordinates are ordered by a notion of multivariate kurtosis. Coordinates associated with extreme eigenvalues are those typically of greatest interest for outlier detection or clustering.

When $S_{1}$ is taken to be the identity matrix and $S_{2}$ the sample covariance matrix, ICS reduces to ordinary principal component analysis; PCA is therefore a special case in which only second-order structure is exploited.

Common scatter matrices

The choice of the pair $(S_{1},S_{2})$ determines which features ICS will emphasize. Common scatter functionals used in practice include:

• the sample covariance matrix;
• the family of M-estimators of multivariate scatter introduced by Maronna;
• the minimum covariance determinant (MCD) and minimum volume ellipsoid estimators of Rousseeuw;
• the distribution-free M-functional of Tyler;
• the symmetrized M-functional of Dümbgen;
• the scatter matrix of fourth moments (sometimes denoted COV₄), which corresponds to the second-stage matrix of Cardoso's FOBI;
• spatial sign and rank covariance matrices in their symmetrized, affine-equivariant forms.

Pairing two functionals with contrasting sensitivities—for instance the covariance matrix with a robust scatter, or with the fourth-moment scatter—produces invariant coordinates that highlight different features of the data, such as outliers, clusters, or non-Gaussian sources.

Intuition

A scatter matrix generalizes the covariance matrix as a summary of multivariate spread, and different scatter matrices emphasize different aspects of the data distribution. ICS typically pairs the ordinary covariance matrix with a second scatter matrix that is either robust to outliers or sensitive to higher-order moments such as kurtosis.

For elliptically symmetric distributions, such as the multivariate normal distribution, any two affine-equivariant scatter matrices are proportional, so all generalized eigenvalues coincide and no direction is preferred. Departures from this equality therefore signal departures from ellipticity—such as the presence of clusters, skewness, or outliers—and the corresponding invariant coordinates highlight the directions in which these features are most pronounced.

Properties

• Affine invariance. If $X$ is replaced by $XA+\mathbf {1} _{n}b^{T}$ for any non-singular $A$ and vector $b$, the invariant coordinates $Z$ are unchanged up to sign and permutation.
• Behaviour under ellipticity. Under an elliptical distribution, the two scatter matrices are proportional and all eigenvalues are equal in the population.
• Connection to ICA. Under the independent component model, ICS recovers the independent components up to sign and order, provided the source kurtoses are pairwise distinct.
• Reduction to PCA. Taking $S_{1}=I_{p}$ and $S_{2}$ equal to the sample covariance matrix recovers principal component analysis.

Applications

• Outlier detection: ICS can identify multivariate outliers that are not apparent in low-order projections such as those produced by PCA.
• Independent component analysis: ICS can serve as an alternative to, or pre-processing step for, ICA.
• Clustering: ICS is used as a projection step in tandem clustering pipelines, where it is combined with a clustering algorithm in the reduced space.
• Other reported applications include financial data analysis, bioinformatics, and statistical quality control.

Advantages and limitations

Advantages

• Invariant under affine transformations of the data.
• Captures higher-order structure rather than second-order variance alone.
• Robust versions perform well under contamination or heavy-tailed distributions.

Limitations

• Results depend on the choice of the two scatter matrices.
• Invariant components may be less directly interpretable than principal components.
• Robust scatter matrices can be computationally demanding compared with PCA.

See also

• Principal component analysis
• Independent component analysis
• Linear discriminant analysis
• Sliced inverse regression
• Factor analysis
• Whitening transformation
• Mahalanobis distance
• Robust statistics
• Multivariate statistics
• Projection pursuit
• Blind source separation

External links

• ICS package on CRAN
• ICSOutlier package on CRAN
• ICSClust package on CRAN

Category:Multivariate statistics Category:Matrix decompositions Category:Dimension reduction Category:Robust statistics Category:Exploratory data analysis Category:Cluster analysis Category:Statistical methods