I have millions of rows (or a large TF-IDF matrix) and PCA crashes with a MemoryError or a TypeError about sparse input.
Why this happens
Standard PCA needs the whole dataset in memory and centers it by subtracting the mean of each column. Centering a sparse matrix turns all its zeros into non-zero values, producing a huge dense matrix. Older versions of scikit-learn refuse sparse input for this reason. Newer versions support it with some solvers, but memory can still be a problem.
Answer 1: Use the randomized solver for faster PCA
If you only need a small number of components from a large dataset, the randomized solver approximates them much faster than a full decomposition, with very little loss of accuracy:
pca = PCA(n_components=50, svd_solver="randomized",
random_state=0)
X_reduced = pca.fit_transform(X_scaled)
