I tested Google’s new SynthID detector on AI slop. There’s a big catch | usagoldmines.com

Aw, have you seen that viral video of the little puppy nuzzling with a baby? It’s absolutely adorable — so adorable that it’s probably AI, although not even Reddit seems to know for sure.

Now Google has unveiled a site featuring a tool designed to spot AI-generated images, videos, and audio files, using a technology called SynthID to pinpoint invisible watermarks embedded in AI-generated content. Surely the SynthID tool can tell us whether that suspicious puppy video was created by AI, right?

I plopped the ultra-cute video into the tool, waited a few seconds, and got the result: no AI watermarks detected. So, the video must be real, right?

Well, not exactly.

Developed by Google as a method of tagging AI-generated content, SynthID allows AI models to embed invisible watermarks in the images, videos, audio files, and other content that they create. There’s even a version of SynthID for watermarking text, but this particular SynthID detection tool (which was previously available to researchers, journalists, and other media professionals) is limited to image, video, and audio formats.

Of course, not every AI model supports SynthID watermarking, and for now, Google’s SynthID tool can only detect AI-generated media created by models from Nvidia, Google, OpenAI, and South Korea-based Kakao.

Put all that together, and it means Google’s SynthID detection tool can tell you one of two things: that it detected a SynthID watermark in a given piece of content, or that it didn’t detect one. That’s all.

To put the tool to the test, I asked Gemini and ChatGPT to create some AI slop for me — namely, images and videos of “a robot visiting Disneyland.”

I plugged those images and clips into the Google tool, and got a result within seconds: “SynthID was detected,” with the site correctly telling me that one image was created by OpenAI (ChatGPT) while the other was from Google AI (Gemini).

But the baby video I mentioned earlier? The SynthID tool chewed on it for a moment before returning its verdict — not that the video was real, mind you, but merely that it hadn’t detected an embedded SynthID watermark.

Ben Patterson/Foundry

I got the same results from other and more obvious AI slop videos I found on Reddit and elsewhere, including one of a man gulping soda from an overturned Coca-Cola truck and another of a bunch of cute kittens lining up to sip milk from a row of baby bottles. Were those videos generated by AI? Sure. Were they watermarked? Nope.

Another look at the SynthID’s report spells it out: “It is unlikely that this media was made with AI from a SynthID partner,” the tool says, while adding that “this may have been generated by a non-partner organization, or at a time prior to a partner adopting SynthID.”

In other words, Google’s SynthID tool can’t prove that an image, video, or other content was definitely created by a human; instead, all that it can tell you is whether it found a SynthID watermark or not. That doesn’t mean Google’s SynthID detector is broken; instead, it’s simply a tool with a narrowly defined job.

So when will we get a foolproof AI detector, including one that works with watermarking tools from a wide range of companies? As Google notes in its SynthID detector FAQ, developing such a tool “takes time,” and would require a “layered ecosystem where watermarking, metadata standards, and verification tools work together across the industry.”

In the meantime, is that video of the puppy nuzzling the baby real or AI? Your guess is as good as mine.