Draft:Feature Importance

Feature Importance (or Variable Importance or Feature Attribution) refers to a set of techniques and mathematical frameworks used in machine learning, statistics, and their applications to quantify the contribution of input variables (features) to a model's output or to the underlying data-generating process. It is a component of Explainable Artificial Intelligence (XAI) and Interpretable Machine Learning (IML) and is used for model improvement, scientific inference, and feature selection.

Feature importance is a context-dependent score that varies based on the scope of the explanation, the model (or data) of interest, and the ultimate purpose of the analysis.

History

The earliest forms of feature importance assessed the strength of the relationships between pairs of variables within animal biology or human psychology using methods such as Francis Galton's correlation coefficient. Sewall Wright further developed variable importance, by introducing path analysis in 1921, to understand causal influences in complex systems. Path analysis determines correlative influences along direct paths by decomposing correlation and partial correlation coefficients into their path-based components.

For decades, the inspection of standardized regression coefficients (or regression weights) were the dominant approach to variable importance. These weights, however, were unstable in the presence of multicollinearity (i.e., when there is high correlation between predictor variables). This led to a problem where the importance assigned to a variable depended on the order in which it was entered into a sequential regression model. In the 1980s, William Kruskal proposed that relative importance should be averaged over all possible orderings of the independent variables to ensure that no single variable was unfairly penalized or elevated by its position in the model. This approach was formalized as the LMG method, named after Lindeman, Merenda, and Gold.

The Random Forest and XAI Era (2001–Present)

With Leo Breiman’s introduction of Random Forests, feature importance expanded to more nonlinear models and explanations. Instead of model parameters (coefficients) and linear importance, he introduced "permutation importance" (Mean Decrease Accuracy or GINI), which assessed nonlinear importance by measuring the drop in model performance when a feature's values were randomly shuffled. Simultaneously in 2001, Lipovetsky and Conklin applied the Shapley value from cooperative game theory to regression, providing a consistent method for variance attribution in the presence of multicollinearity.

Gradient-based feature importance methods emerged from early sensitivity analysis in neural networks and have become a popular family of feature importance following the deep learning revolution. Gradient-based feature importance relies on the computation of the partial derivatives of a model’s output with respect to its inputs in order to quantify how small input changes affect predictions. Simple input gradients and gradient/input methods were among the earliest and most widely used due to their computational efficiency. Integrated Gradients were later introduced to address instability and noise, which was especially observed in deep, non-linear models . These methods average gradients along a path from a baseline input to the actual input. Closely related methods include DeepLIFT, which propagates contribution scores relative to a reference activation, and Layer-wise Relevance Propagation (LRP), which redistributes prediction scores backward through the network. Variants such as SmoothGrad further improve robustness by averaging gradients over noisy perturbations.

In 2017, Scott Lundberg and Su-In Lee introduced SHAP (SHapley Additive exPlanations). SHAP unified various local attribution methods (like LIME and DeepLIFT) under the umbrella of Shapley values, providing the first mathematically consistent framework for local feature importance in any machine learning model.

While the majority of modern feature importance methods are directly derived from machine learning models, and are often used for feature selection and model optimization, machine learning-based methods have also been developed specifically for the task of interpreting relationships in the data.

Axioms for Feature Importance

Formal feature importance frameworks use mathematical axioms to define consistent scoring criteria and validate the theoretical properties of different estimation methods.

The Shapley Axioms (S1–S4)

Shapley values are the unique solution that satisfies four core game-theoretic axioms, which describe how to fairly distribute the "total gain" of a model's prediction among the participating features.

1. Efficiency (Local Accuracy): The sum of the importance scores $\phi _{j}$ for all features must equal the difference between the model's prediction for an instance $f(x)$ and the expected prediction $E[f(X)]$.
2. Symmetry: If two features $i$ and $j$ contribute exactly the same value to every possible subset of other features, they must receive the same importance score: $\phi _{i}=\phi _{j}$.
3. Dummy (Null Player): If a feature contributes nothing to the value function for any subset of features, its importance score must be zero. This is crucial for identifying irrelevant features.
4. Additivity (Linearity): If the value function is the sum of two functions $v=v_{1}+v_{2}$, then the importance scores must be the sum of the scores calculated for each function: $\phi _{j}(v)=\phi _{j}(v_{1})+\phi _{j}(v_{2})$.

Axioms for Explaining the Data

While the Dummy (Null Player) axiom is desirable for obtaining feature importance scores that can be used to optimize a model, this is not the case for explaining the data, since variables may be highly correlated (and hence offer little value to the existing model), yet remain causally linked to the response variable. Alternative axioms have been proposed for feature importance methods that intend to explain the data.

• Blood Relation (Causal Graph Association): A feature should have non-zero importance if and only if the feature is blood related (associated) with the response in the ground-truth causal graph.
• Marginal Contribution: The importance of a feature must be at least as high as the gain it provides when added to the set of all other features.
• Elimination: Removing other features from the feature set can only decrease (or leave unchanged) the importance of a remaining feature. It cannot increase it.
• Invariance under Redundant Information and Symmetry under Duplication: Adding a redundant feature should not change the importance of preexisting features, and identical features should receive equal importance.

Classification of Methods

By Purpose

The choice of method is categorized based on the user's objective.

• Model Explanation: The goal is to understand the "logic" of a black-box model to ensure safety, fairness, and reliability. Methods like SHAP, LIME, and Accumulated local effects are standard here.
• Data Explanation (Scientific Inference): The goal is to learn about the real world. Researchers prioritize methods that handle redundancy and correlation in a way that reflects the true underlying relationships (e.g., MCI, UMFI).
• Model Optimization (Feature Selection): The goal is to improve the model's performance by removing irrelevant or redundant features. Techniques like Recursive Feature Elimination (RFE) use importance scores as a selection criterion.

By Scope

Feature importance methods have been classified into four distinct settings based on two axes: Global vs. Local and Data vs. Model.

	Reference: Model	Reference: Data
Global	Global-Model Importance: Explains how a trained model behaves across the entire dataset. It identifies which features the model generally relies on for its predictions.	Global-Data Importance: Explains the true relationships in the underlying phenomenon. It seeks to identify the intrinsic predictive power of features within the population, regardless of a specific model's choices.
Local	Local-Model Importance: Explains why a specific prediction was made for a single instance. It quantifies the influence of each feature on that a specific model's specific outcome.	Local-Data Importance: Explains the role of a feature for a specific individual in the real world (e.g., why a specific patient developed a disease), focusing on the causal or statistical dependencies for that point.

Methods like SHAP and LIME are local-model. Global-model importance metrics include permutation importance and SAGE. Global-data methods include MCI and UMFI.

By Treatment of Correlated Features

Feature importance methods often differ in how they treat the correlation between features. In the presence of correlated features, predictive credit can be distributed in a variety of ways. Methods can therefore be classified based on how they assign credit to correlated features, where one end of the spectrum is given by marginal feature importance, and the other is given by conditional feature importance.

• Conditional Feature Importance: Evaluates a feature by conditioning on the values of all other features. This approach respects the dependence structure of the data and measures the unique information a feature provides that is not already captured by other variables. Methods in this category include conditional permutation importance, Leave-One-Covariate-Out, and partial correlation.

• Marginal Feature Importance: Relies on associations between the response and predictors, regardless of multicollinearity. Purely marginal methods assign the same high importance to all features. Examples include correlation, marginal contribution feature importance, and ultra-marginal feature importance.

Methods such as SHAP and permutation importance are somewhere in between the two extremes as importance is shared among correlated features.

Gradient vs. Non-Gradient

The technical implementation of the importance measure often dictates its applicability to different model architectures.

Mechanism	Description	Examples
Gradient-Based	Utilizes the derivatives (gradients) of the model's output with respect to its input features. These are typically model-specific and used for differentiable models like neural networks.	Saliency Maps, Integrated Gradients, Grad-CAM, DeepLIFT.
Non-Gradient Based	Treats the model as a "black box" and relies on perturbations, shuffling, or submodel training. These are typically model-agnostic and applicable to any algorithm.	Permutation Importance, KernelSHAP, LOCO, MCI.

Other Classifications

Achen (1982) introduced a classification of linear regression-based feature importance methods: "dispersion importance" (explained variance), "level importance" (impact on the mean), and "theoretical importance" (the change in response for a given change in regressor).

Applications

• Medicine
• Earth Science
• Computer Science

See also

• Feature selection
• Explainable Artificial Intelligence
• Shapley value