Controversy surrounding preventative measures for inappropriate AI image generation

By Kate Northrop

The Washington's Lottery AI-powered website was discontinued following a user complaint that it had generated a topless photo using her face.

What began as an innocent way to help lottery players envision their lives after a big win turned into a disturbing controversy after the Washington Lottery's AI-powered website essentially generated softcore pornography of one of its users.

"Test Drive a Win" is a website that allows users to upload a photo of themselves to have the AI superimpose their likeness over a neat activity or vacation spot, alluding to the fun possibilities that come with a big lottery win.

50-year-old mother Megan of Tumwater thought she would give it a try. She landed on a "swim with the sharks" choice, and used the site's feature that allowed her to upload a photo of her face to find out what she would look like doing the activity herself.

Except, the AI-generated photo looked nothing like swimming with the sharks.

The photo, which she shared with "The Jason Rantz Show" on KTTH on Tuesday, showed her smiling face superimposed on her body posing on a bed in only a bikini bottom. With Megan surrounded by various aquarium fish, there were no sharks to be found, either. The AI-generated image is watermarked by the Washington's Lottery logo in the bottom right corner.

"Our tax dollars are paying for that!" Megan told the Jason Rantz Show. "I was completely shocked. It's disturbing to say the least."

Megan explained that she feared the uncensored image would somehow spread around or that the same thing would happen to another user. She told a friend, who knows an employee at the Lottery, what happened when she used the website.

"I also think whoever was responsible for it should be fired," she affirmed.

The Jason Rantz Show submitted an inquiry to the Lottery. A response from a Lottery spokesperson did not positively confirm that the image was generated by the website per se, but they did say that the Lottery received a report that the image was created by the AI system.

Three hours later, the site was taken down.

"We were made aware that a single user of the AI platform was purportedly provided an image that did not adhere to the built-in parameters set by the developers," the Lottery said in a statement to the Jason Rantz Show. "Prior to launch, we agreed to a comprehensive set of rules to govern image creation, including that people in images be fully clothed. We have not seen the image that the AI provided to this user, but out of an abundance of caution we took the website down."

Lottery Post reached out to the developers of Washington's Lottery's "Test Drive a Win" website to find out more about the technical root cause of the issue. They replied that they have nothing else to add beyond the general statements issued by the Lottery.

To gain more insight on how the problem may have manifested, Lottery Post spoke to AI expert Theodor Marcu, who founded San Francisco-based Superdrop Labs Inc. and has built several consumer-facing AI products.

One of the most important factors in the generation of the topless image is whether the AI platform on the Lottery's website only allowed users to input their own "prompts," such as strings of words or images that are meant to elicit a response from an AI model, or if the AI only churned out content using canned answers pre-selected by the Lottery. In this case, the prompt was "swim with the sharks."

At this time, there is no confirmation from the Lottery about whether that prompt was canned or whether she was allowed to insert her own input.

"Whether the prompt was canned or not matters a lot because it tells us where the system could be engineered or influenced by the user to do things it was not supposed to do," Marcu elaborated.

Marcu also pointed out that we should pay attention to the fact that the user submitted a photo of herself to the AI platform. That photo may have had the potential to sway the AI toward producing the image it did, but we will never know unless we can delve into the back-end of the AI model itself or see the uploaded photo.

 "Even with the most advanced large language models today, users have been able to force them to do inappropriate things. So, it matters if the user was able to guide the system in any way, especially if they had access to either a public facing text input, submit photos, or even a public API that they could influence. That could have a drastic impact. It's hard to know without having access to the app or seeing the photo."

"If she submitted a picture of herself in a bikini, it may have impacted the generated photo," he added. "She very well may not have, but without that information, as well as confirmation from the Lottery as to how much freedom users had in influencing the prompts, it's near impossible to know."

There are AI models that cater to producing both "safe for work" (SFW) and "not safe for work" (NSFW) types of content. It's hard to imagine that the developers would employ a model geared toward explicit content, and most well-known models, such as GPT4, are trained to avoid generating it. However, Marcu mentioned, a user can still trick the model into "doing something inappropriate if they're allowed input."

But Marcu believes that the developers must have been cognizant enough to code in restrictions against inappropriate material. Even if a developer technically were to do everything right, there is always a chance, even despite hard restrictions and rules in place, that the content generated by AI is undesirable.

"There's always the potential for a slip up, since these systems are non-deterministic and can always produce something that was unexpected," he explained. "Even big companies like Google face this problem. However, it's possible to minimize risks too. It seems that the [Lottery's developers] are adamant that they did the right thing, and I believe them, but it's hard to assess without having access to the app and knowing what those guidelines and regulations were."

"Prior to launch we worked closely with the developers of the AI platform to establish strict parameters to govern image creation," a statement from the Lottery said. "We were made aware that a single user of the AI platform was purportedly provided an image that did not adhere to those guidelines. This campaign was launched more than a month ago and has had thousands of images created that all fall within the prescribed guidelines. Regardless, one purported user is too many and as a result we have shut down the site."

Marcu did agree that this slip up was enough to warrant shutting down the site, but he doesn't believe that this is the nail in the coffin for big companies to employ similar AI platforms geared toward consumers in the future. After all, just as the Lottery alluded to in their statement, this was one single instance out of over a month's worth of successful content. Since no AI model is perfect, companies should be ready to deal with that rare instance an AI platform or app produces undesirable content, like in Megan's case.

"If companies want to adopt AI models in their products for consumers to freely use, the best thing to do is to add disclaimers so that users cannot abuse the AI model to produce inappropriate content, either on purpose or accidentally," Marcu recommended. "Your disclaimer should make it absolutely clear that some generated content may be out of the company's control. You want to remove the association between yourself and potentially damaging content."

Megan was happy to see that the website was discontinued and expressed hopes that, should the site become available again, any back-end issues would be corrected by developers.

"We heard about this earlier this week," the Lottery said in an additional statement. "As soon as we did, we had the developers check all the parameters for the platform and were comfortable with the settings, but after further internal discussions today, we chose to take down the site out of an abundance of caution, as we don't want something like this purported event to happen again."

(Click to display full-size in gallery)A Washington Lottery player who wanted the lottery's AI-powered website to show her "swimming with the sharks" after winning a jackpot instead ended up with a NSFW photo of her sitting in bed topless with some goldfish.