scikit-learn has several feature selection tools. When should I use each one?
The three families of methods
Feature selection methods fall into three broad groups, and the scikit-learn tools map onto them neatly.
Filter methods score each feature using a statistic, without training a model. SelectKBest, SelectPercentile, and VarianceThreshold belong here. They are very fast and work on any dataset size. Their weakness is that they look at one feature at a time. They miss features that are only useful in combination with others, and they don't notice when two features carry the same information.
Wrapper methods train a model repeatedly and use its performance or weights to choose features. RFE, RFECV, and SequentialFeatureSelector belong here. Because they use a real model, they account for interactions between features. The cost is speed: they need many model fits.
Embedded methods select features as part of training a single model. SelectFromModel with Lasso, L1-regularized logistic regression, or a tree ensemble belongs here. It gives model-aware selection at roughly the cost of one fit.
Answer 1: A practical decision guide
Use SelectKBest when you have a very large number of features (text, genomics), need a quick baseline, or want a first-pass filter before a slower method.
Use RFE or RFECV when you have a moderate number of features and a model with coef_
or feature_importances_, and you want a careful selection. Use RFECV if you don't know how many features to keep.
Use SelectFromModel when you want model-aware selection but RFE is too slow. It pairs especially well with L1 regularization or random forests.
Use SequentialFeatureSelector when your model doesn't provide weights (KNN, RBF SVM, neural networks) and you can afford the computation.
