MIS 752 · Lab 5 · richardyoung
Speak the Patient's Language: semantic search vs keyword search
A 20-sentence clinical corpus embedded with all-MiniLM-L6-v2, the full cosine similarity matrix, and a scored comparison of semantic search against a keyword baseline on a written gold standard. Runs locally on a free CPU: no API key, no PHI, $0.
The 30-second read
THE MODEL
encoder : sentence-transformers/all-MiniLM-L6-v2
dimensions : 384 numbers per sentence
corpus : 20 sentences, 4 topics, 2 voices
embed time : 412 ms for the whole corpus
API cost : $0.00 (runs locally, no key, no network after load)
DOES IT CAPTURE MEANING?
mean cosine within a topic : 0.235
mean cosine across topics : 0.129
separation : +0.106
RETRIEVAL vs THE GOLD STANDARD (12 queries, top-2)
keyword (word overlap) precision 0.38 · recall 0.28 · MRR 0.50
semantic (embeddings) precision 0.58 · recall 0.51 · MRR 0.92
overall, semantic beats keyword on MRR by +0.42
THE FINDING - MRR SPLIT BY WHETHER THE PATIENT GUESSED OUR WORDS
patient guessed our words keyword 0.86 · semantic 1.00
patient's own words keyword 0.00 · semantic 0.80
EXAMPLE QUERY "I eat licorice, and my blood sugar goes crazy."
1. dia1 [diabetes/patient] cosine 0.603
2. car4 [cardiac/patient] cosine 0.360
Do embeddings capture meaning?
The 20×20 cosine similarity matrix, rows and columns sorted by topic and boxed into the four topic blocks. Each label shows the sentence id and its voice (pat = patient, cli = clinical). The diagonal is blanked and the colour scale spans the off-diagonal range: every sentence scores 1.00 against itself, and leaving that in flattens everything else into one shade. A dark cell spanning both voices is the whole point of the week: same meaning, no shared words. A pale row inside a box is a sentence this model cannot place.
Meaning beats spelling only where spelling runs out
Scored against the written-down gold standard, not by eyeball. MRR asks how high the first right answer was, and it is the metric a patient on a phone actually feels: 1.0 means rank one. The two groups differ by whether the patient happened to guess the corpus's own vocabulary — a split computed from the text, not labelled by hand. In the left group keyword search is fine, because there are words to match. In the right group there are none and it collapses. Semantic search never needed the words to match in the first place.
Per-query detail
Where each method wins and loses, query by query. The keyword baseline is deliberately naive word overlap, so this is not the strongest possible keyword system — BM25 would do better, and a reranker better still. It is enough to show that the vocabulary gap, not the algorithm, is what defeats word matching.
query_type query method retrieved p_at_k recall_at_k rr first_rank
patient guessed our words chest hurts when i climb stairs keyword (word overlap) car1,car2 1.0 0.667 1.0 1.0
patient guessed our words chest hurts when i climb stairs semantic (embeddings) car1,car2 1.0 0.667 1.0 1.0
patient guessed our words my refill is not ready at the pharmacy keyword (word overlap) med5 1.0 0.500 1.0 1.0
patient guessed our words my refill is not ready at the pharmacy semantic (embeddings) med5,dia4 0.5 0.500 1.0 1.0
patient guessed our words is it safe to take ibuprofen with warfarin keyword (word overlap) med1,dia1 0.5 0.500 1.0 1.0
patient guessed our words is it safe to take ibuprofen with warfarin semantic (embeddings) med1,med2 1.0 1.000 1.0 1.0
patient guessed our words i feel low and cannot sleep keyword (word overlap) men1,men3 0.5 0.333 1.0 1.0
patient guessed our words i feel low and cannot sleep semantic (embeddings) men1,dia3 0.5 0.333 1.0 1.0
patient's own words it aches to climb to the second floor keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words it aches to climb to the second floor semantic (embeddings) car1,dia5 0.5 0.333 1.0 1.0
patient's own words the reading on my meter jumped since supper keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words the reading on my meter jumped since supper semantic (embeddings) car5,dia5 0.0 0.000 0.0 NaN
patient guessed our words would the anti-inflammatory clash with what thins my blood keyword (word overlap) car5,med1 0.5 0.500 0.5 2.0
patient guessed our words would the anti-inflammatory clash with what thins my blood semantic (embeddings) med1,med2 1.0 1.000 1.0 1.0
patient guessed our words i have been down and wide awake at 3am for weeks keyword (word overlap) med5,men1 0.5 0.333 0.5 2.0
patient guessed our words i have been down and wide awake at 3am for weeks semantic (embeddings) men1,dia3 0.5 0.333 1.0 1.0
patient's own words i cannot pay for the name brand any more keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words i cannot pay for the name brand any more semantic (embeddings) med3,med5 0.5 0.500 1.0 1.0
patient guessed our words my feet feel like pins and needles keyword (word overlap) dia5,men3 0.5 0.500 1.0 1.0
patient guessed our words my feet feel like pins and needles semantic (embeddings) dia5,car1 0.5 0.500 1.0 1.0
patient's own words the drug store says come back next week keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words the drug store says come back next week semantic (embeddings) med5,men5 0.5 0.500 1.0 1.0
patient's own words i get winded carrying the groceries in keyword (word overlap) (nothing) 0.0 0.000 0.0 NaN
patient's own words i get winded carrying the groceries in semantic (embeddings) car4,car1 0.5 0.500 1.0 1.0
How this was measured
Sentences were embedded with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions, ungated, run locally on a free Colab CPU -- no API key and no network traffic after the model loads). Cosine similarity is computed directly as E @ E.T on L2-normalised rows, not via a library convenience method, so every number is reproducible from the saved embeddings.npy.
Retrieval is scored against 12 queries carrying 29 written relevance judgements (GOLD in the notebook). The keyword baseline counts shared content words (stopwords dropped, length > 2). Metrics are precision@k, recall@k and mean reciprocal rank at k=2.
This is a 20-sentence teaching corpus with a 12-query gold set -- a demonstration of method, not a clinical validation. The corpus is teaching text I wrote: no PHI, no real records, nothing to act on clinically.