PizzaGAN database

MIT Teaches AI to Configure Recipes from Pictures of Pizza

Researchers at Massachusetts Institute of Technology (MIT) have created an AI network that reverse-engineers pizza by deconstructing a scanned picture of the pie to its basic ingredients.

The scientists at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Qatar Computer Research Institute (QCRI) created a neural network which they aptly named PizzaGAN. This network can scan and then create images of pizza without toppings, with more toppings, before baking, or after it’s fresh out of the oven.

This may seem similar to Domino’s newest AI, DOM Pizza Scanner, but PizzaGAN differs by its ability to not only scan but to completely break the pizza down to basic ingredients thus configuring its recipe.

This process began with synthetic images of pizza, scanning around 5,500 clip art pizza images which it edited by removing toppings layer by layer.

Social media played an important role in the learning process. According to the research paper, the scientists used the #pizza hashtag on Instagram to scroll through “over 38 million posts” of actual pizza which yielded 9,213 pictures after filtering out “undesired images” using a CNN-based classifier trained on a small set of manually-labeled pizza and non-pizza images.

Related: Domino’s Introduces an AI Tool to Review Pizza Before Delivery

After compiling a database of real and synthetic pizzas, the AI then learned how to add and remove ingredients such as pepperoni and then predict the order of the toppings. For example, when PizzaGAN identifies three toppings (bell peppers, onions, and mushrooms) it can then deduce that the bell peppers were added last. It can also edit pictures to show what it would look like pre- and post-cooking.

You can play around with removing and adding ingredients, as well as cooking and uncooking the pizza on the PizzaGAN site.

In the research paper, the researchers wrote that they believe “a similar approach is promising for other types of foods that are naturally layered such as burgers, sandwiches, and salads. Beyond food, it will be interesting to see how our model performs on domains such as digital fashion shopping assistants, where a key operation is the virtual combination of different layers of clothes.”

In other words, it might be possible to upload a picture of an appetizing pie, then figure out and customize the recipe to your liking. We have yet to reach an end-to-end AI robot independently crafting a customized pizza from scratch with memory, but it’s a step.