I used Hugging Face AutoModel to create embeddings. The output has an extra dimension, and every similarity function rejects it.
Why this happens
A transformer model returns one vector per token, not one per sentence. outputs.last_hidden_state has shape (batch, number_of_tokens, hidden_size). To get a single sentence embedding, you have to combine the token vectors into one, which is called pooling. Different sentences have different token counts, so skipping pooling also causes shape mismatches between texts.
Answer 1: Mean pooling with the attention mask
Average the token vectors, ignoring padding tokens. This is the pooling used by most sentence-transformer models:
import torch
from transformers import AutoTokenizer, AutoModel
name = "sentence-transformers/all-MiniLM-L6-v2"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModel.from_pretrained(name)
batch = tokenizer(texts, padding=True, truncation=True,
return_tensors="pt")
with torch.no_grad():
out = model(**batch)
mask = batch["attention_mask"].unsqueeze(-1).float() # (batch, seq, 1)
emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
emb = torch.nn.functional.normalize(emb, p=2, dim=1) # (batch, 384)
Without the mask, padding tokens are included in the average, so the same sentence gets a different embedding depending on what else is in the batch.
