AI for Everyone: How Do We Know If an AI Really Works? (Part 6)
“So, what did you think of the restaurant?” was the natural question at the end of our long lunch. As we all started handing out our scores and opinions (“the pasta was outstanding”, “the main course not so much”, “the service could have been better”), it occurred to me that we evaluate AIs exactly the way we evaluate a restaurant: with specific criteria, personal feedback and the occasional unexpected surprise.
Because you see, just as it’s not enough to say “the restaurant is nice” or “the food is good”, it’s not enough to say “the AI works” or “the AI is smart”. We need specific criteria, metrics, ways to figure out whether what we’re using really lives up to our expectations. And believe me, some of these criteria are surprisingly similar to the ones we use to judge a restaurant experience.
Hallucination: When AI Makes Things Up
Hallucinations in AI are like when a waiter makes up the ingredients of a dish because he doesn’t want to admit he doesn’t know them. “Oh yes, our soufflé is made with eggs from quails raised on the moon” - sound familiar?
The other day I was using ChatGPT to look up information about a book, and it started quoting chapters that didn’t exist. It’s as if our imaginary waiter, instead of admitting he hadn’t read the menu, started inventing fanciful dishes. Hallucinations are one of the main problems of modern AI, and learning to recognize them is essential.
Perplexity: Measuring Confusion
Perplexity is like measuring how confused a customer is while looking at the menu. The more perplexed the customer (or the more uncertain the model), the higher the perplexity. It’s a technical way of saying “how sure is the AI about what it’s saying?”
It’s like when the waiter describes a dish to you and you can immediately tell whether he really knows it or is improvising. Perplexity measures exactly that: how “fluent” and natural the AI’s answer is.
RLHF: The Feedback That Counts
Reinforcement Learning from Human Feedback (RLHF) is exactly what it sounds like: learning from human feedback. It’s like when a restaurant takes customer reviews seriously and improves its service based on the comments it receives.
“It’s like TripAdvisor for AIs!” exclaimed Laura, who works in marketing. She couldn’t have come up with a better example. RLHF is a systematic way of collecting human feedback and using it to improve the AI’s performance.
Alignment: Being on the Same Wavelength
Alignment is like making sure the kitchen understands exactly what the customers want. It’s not enough for the food to be technically good; it also has to match the customers’ expectations and values.
In the world of AI, alignment means making sure the system doesn’t just work well, but does so in an ethical way, aligned with human values. It’s like having a restaurant that not only serves great food, but does it in a sustainable, ethical and respectful way.
Bias: The Hidden Prejudices
Biases are like those unconscious prejudices that might lead staff to treat customers differently based on how they look. In the world of AI, biases are prejudices baked into the training data that can lead to discriminatory results.
“Like when the waiter automatically brings the check to the man at the table?” asked Claudia. Exactly! Bias in AI works in a similar way: they’re hidden assumptions and prejudices that can influence the system’s answers.
The Modern Turing Test
The old Turing test (figuring out whether you’re talking to an AI or a human) is no longer enough. Today we need more sophisticated evaluations:
- Is the AI useful for the specific task?
- Are its answers reliable?
- Is it transparent about its limits?
- Is it ethically aligned?
It’s like judging a restaurant not just on the food, but also on the atmosphere, the service, sustainability and value for money.
Practical Metrics for the Real World
In real life, we evaluate AIs on very practical criteria:
- How often it says sensible things vs. how often it “hallucinates”
- How consistent its answers are
- How useful it is for real tasks
- How safe and reliable it is
It’s like judging a restaurant not on a single visit, but on its consistency over time.
Conclusion: Toward a More Mindful Future
As we were paying the bill for our lunch (excellent, for the record), I thought about how evaluating AIs is a bit like being a food critic in the world of technology. It’s not enough to say “it works” or “it doesn’t work”; we need to be more sophisticated and mindful in our evaluations.
And so our series on AI comes to an end. From the basic concepts to the advanced techniques, from architectures to optimization, all the way to evaluation, we’ve taken a complete journey through the world of artificial intelligence. And it all started from a simple chat over lunch with friends.
P.S. If you’re wondering what score we gave the restaurant, let’s just say it passed its own personal “gastronomic Turing test” with a solid 9/10. Ok.. I already know.. you’d like to see photos of the dishes. Well, I’m working on a “PhotoGallery” feature for POSTS. As soon as I’m done, I promise they’ll be added to the guide!
P.P.S. More than once I had the feeling the waiter’s recommendations had been generated by ChatGPT! 😉