Recently, I wanted to find the origin of the idiom “My word is my bond.” Unfortunately, the legendary Quote Investigator® had nothing to say. Some articles also suggested that the saying had its basis in the Hip Hop movement, but I had a feeling it existed earlier.
After some Googling, I found an article that indicated the origin was Shakespeare’s play, Merchant of Venice. I didn’t want to read through the whole play to determine this, so it seemed like a prime opportunity to test out an LLM’s needle-in-a-haystack ability.
I recently subscribed to Claude Pro after trying Gemini for two months. Opus, Anthropic’s frontier model, boasts an impressive 200K context window. So, I decided to see if Claude was up to the task.
First, I retrieved the full plain text of Shakespeare’s play. MIT generously provides this.
Next, I wanted to verify that the play would fit into the context window. I used OpenAI’s tiktoken library to approximate the number of tokens in the play.
Armed with the knowledge that the whole play would fit in the context window, I loaded the file into Claude Opus with the following prompt:
Here is Claude’s response:
The first response gives an accurate location (Act I Scene 3). Unfortunately, Claude incorrectly believes the second passage is in Act 4, when in fact it is in Act III, Scene 3. Interestingly, it made the same mistake on the same passage on an earlier try. However, that time it thought it was Act I.
Going back to the former response, Claude’s explanation of the first passage is correct. Antonio is taking out a loan, and he’d better repay it, lest he offer up his flesh instead.
There are a couple issues with the second passage. As I mentioned, the quote is from Act III, Scene 3, not Act IV. However, when Shylock receives an offer of twice what he is owed (not thrice, Claude’s mistake), that does occur in Act IV Scene 1.
Finally, Claude wraps up by saying that the exact phrase, “My word is my bond” is not used, but the theme of the saying is a central part of the play. I agree with this.
Overall, I’m quite satisfied with Claude’s performance, and I am excited for a future where we can use LLMs to “talk” to large documents.
Featured image: Maurycy Gottlieb’s “Shylock and Jessica” (1876) from Wikipedia
Leave a Reply