I ran SelectKBest on the full dataset, then did train_test_split and cross_val_score. The scores look suspiciously good, and the model performs much worse on new data.
Why this happens
This is one of the most common mistakes in feature selection. When you fit the selector on the full dataset, it uses the labels of the rows that later become your test set. The selected features are therefore partly chosen because they fit the test data. Your evaluation is no longer honest.
The effect can be dramatic. With many features and few samples, you can get high accuracy even on pure random noise. The selector simply finds features that happen to match the labels by chance.
Answer 1: Put the selector in a Pipeline
A Pipeline makes sure that, in each cross-validation fold, the selector is fit only on that fold's training data. The validation data is never seen during selection.
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
pipe = Pipeline([
("select", SelectKBest(f_classif, k=10)),
("clf", LogisticRegression(max_iter=1000)),
])
scores = cross_val_score(pipe, X, y, cv=5)
print(scores.mean())
This is the recommended approach. It also makes deployment easier, because the whole pipeline can be saved and applied to new data in one step.
