I picked k=10 at random. Is there a principled way to choose it?
Why this matters
There is no universally correct k. It depends on your data and your model. The reliable approach is to treat k as a hyperparameter and pick the value that gives the best validation score.
Answer 1: Tune k with GridSearchCV inside a Pipeline
This is the most common and most reliable approach. The selector is re-fit inside each fold, so the score for each k is honest (see Q6 on data leakage).
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
pipe = Pipeline([
("select", SelectKBest(f_classif)),
("clf", LogisticRegression(max_iter=1000)),
])
param_grid = {"select__k": [5, 10, 15, 20, 30, "all"]}
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy")
grid.fit(X, y)
print("Best k:", grid.best_params_["select__k"])
print("Best CV score:", grid.best_score_)
Note the double underscore in select__k. That's how you reach a parameter of a step inside a Pipeline. Including "all" lets you check whether feature selection helps at all.
You can also tune the model's hyperparameters in the same grid (e.g. "clf__C": [0.1,
1, 10]), so the best k and the best model settings are found together.
