How can you tell when your favourite neural network has suddenly “broken” and become ten times dumber? Why did one innocent query make it consume 25 times more tokens? And how does an ML-product tester become almost an ML-engineer?
Postgres Pro recently has introduced an ML-bot for working with databases—and we put it through rigorous testing!
In this talk, we’ll cover:
-
The specifics of testing LLMs.
-
Peering inside the “black box” with a Langfuse.
-
Detecting when a new model version was getting worse.
-
Testing one neural network with another.
-
Why negative test cases can be more important than positive ones.
This talk will be valuable for everyone—from beginners to mature professionals!