I tried Vicuna and it made me laugh
Two weeks ago, I found Vicuna, “an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality”. I knew to take that 90% stat with a pinch of salt, regarless of the asterisk, but wanted to try it. (tweet for proof)
For my prompt, I tried something I’ve been using on GPT3.5:
And the output:
Sounds believable… but it’s not all true. There isn’t a flying eagle nor a rocky shore on the back. The rest is generic and subjective. It certainly wasn’t in the style of the redacted person I used. But I didn’t expect perfection or 100% truth (and neither should you!)
Now to test the chatbot element. I then prompted Vicuna to reword the previous output “in the style of Don King”. I wanted something ostentatious and my mind went to Don King as I’ve used boxing related prompts in previous examples. Here’s the result:
It’s a very bad caricature of Don King but it made me laugh because it was so bad. I could see a crap (probably white, almost certainly cis male) comedian try this and flop. If there’s one profession I’d like to see put out of business because of AI, it’s that (I jest; AI is still as prone to racist as white male comedians given they both get their ‘humour’ from Reddit anyway, but I digress).
More on Vicuna
Vicuna published a full breakdown of the model, it’s advantages, and disadvantages. The tl;dr: it’s a good open-source alternative but it could never be as good as Bard/any GPT model. Unfortunately, the dataset isn’t available and there are no plans to release it which is a shame and undermines the open-sourceness of it a little, but it might also be a legal thing; I dunno.
A different test
I also used it to try and find a link between two randomly selected tags from my blogs. I got ‘bell hooks’ and ‘Tony Hawk’ and this is what it said:
Coherent and fairly true. I wouldn’t have expected an open-source model to find a deeper connection than I would have found besides them both being American.
I also tried Teenage Mutant Ninja Turtles and jewelry:
Again, plausible. The last part in particular inspired me to find this piece of TMNT-themed jewelry which I blogged about. And that’s the beauty of this tech. To paraphrase Joe MacMillan from Halt and Catch Fire, “AI isn’t the thing; it’s the thing that gets you to the thing.” AI can’t give me a finished product but it can certainly help me get there, alongside my ingenuity.
As we saw in the very first prompt + output, hallucinations are still a big flaw and it gets bigger with open source models created from better closed-source models. The idea is to take what’s there and refine and pre-train it on high quality training data to perform specific tasks. That’s where this tech excels. Where you’ll find outliers and inconsistent performance is trying to turn this proverbial black box into a Tower of Babel but that isn’t stopping every AI startup from trying.
I tried another pair of entities to see how they were linked (Martin Luther King and Cuba) and the model’s output said MLK had visited Cuba in 1960. That didn’t happen. These things aren’t great for factual information, even when you hook them up to the internet—you know, that completely factual and never wrong platform of knowledge.