In this post, let us explore:
- What is feature selection?
- Why we need to perform feature selection?
- Methods
What is Feature Selection?
Feature selection means selecting and retaining only the most important features in the model. Feature selection is different from feature extraction. In feature selection, we subset the features whereas in feature extraction, we create a new feature from the existing features.
Why Feature Selection is important?
- It simplifies the model: data reduction, less storage, Occam’s razor and better visualization
- Reduces training time
- Avoids over-fitting
- Improves accuracy of the model
- Avoids curse of dimensionality.
Methods
Feature selection methods can be grouped into three categories: filter method, wrapper method and embedded method.
- Filter method
In this method, features are filtered based on general characteristics (some metric such as correlation) of the dataset such correlation with the dependent variable. Filter method is performed without any predictive model. It is faster and usually the better approach when the number of features are huge. Avoids overfitting but sometimes may fail to select best features.
- Wrapper method
In wrapper method, the feature selection algorithm exits as a wrapper around the predictive model algorithm and uses the same model to select best features (more on this from this excellent research paper). Though computationally expensive and prone to overfitting, gives better performance.
- Embedded method
In embedded method, feature selection process is embedded in the learning or the model building phase. It is less computationally expensive than wrapper method and less prone to overfitting.
Summary
Filter method is faster and useful when there are more number of features. Wrapper method gives better performance while the embedded method lies in between the other two methods.
