We've launched! After months of work, MLCommons has released our v1.0 benchmark that measures LLM (aka "AI") propensity for giving hazardous responses.

Here's the results for 15 common models: https://ailuminate.mlcommons.org/benchmarks/

And here's the overview: https://mlcommons.org/ailuminate/

I was the tech lead for the software and want to give a shout out to my excellent team of developers and the many experts we worked closely with to make this happen.

If you see somebody described as "he's just like that" or "he can't help himself", ask if he's like that when he might be held accountable. E.g., in front of his boss, his mom, at church, when a video camera is recording.

Occasionally, he really can't help himself. But it usually turns out that he can, and that there's a pattern to when he can't. E.g., he's a jerk to juniors and peer women, but perfectly controlled in front of his male boss. That's when you know it's not illness, it's abuse.

It is very telling that (1) it’s very clear that the only people a decade ago to correctly predict the current moment were, at the time, derided as “radical” and (2) not a single media outlet has learned a single lesson from (1)