How Art Can Be Helped By (Data) Science

Prior to class this week I was listening to a presentation about how art collectors are now using AI to spot forgeries of artwork. Obviously, some pieces of art are very valuable because of the uniqueness and rarity of them. The ability to make obscene amounts of money from an “original” work of art breeds an abundance of copycats.

So how does AI actually do this? They look at paintings that have been validated to be done by a specific artist, like Picasso, and study all of the work that he has done over the years using high resolution pictures. All of the paintings have been traced back to the original artist through a verification process that analyzes the painting. The analysis of previous work is the most critical aspect of the whole process. Since any algorithm is “garbage in, garbage out” all of the inputs need to be perfect otherwise contamination could lead to a skewed output and actually cause more harm than good. The more comprehensive an artist’s career portfolio is, the stronger the neural network will be to determine the different styles of each individual artist down to their brush strokes. Using these other paintings to understand the style of brushstrokes a specific artist typically has, it develops a heat map for the painting to point out “areas of question” that indicate it might be a forgery.

This concept is not new and gained a lot of popularity 4 years ago when a paper was published by Rutgers and the Atelier for Restoration and Research of Paintings in The Hague that broke down 297 pieces of artwork by 4 major artists (Picasso, Matisse, Modigliani, and Hasse) into 80,000 strokes. The system detected the precise artist 80% of the time and fakes 100% of the time.

When a new, previously unseen artwork is being analyzed, the features are analyzed in comparison to the already stored one. If they match, the new image is labeled as original; otherwise, it is considered a fake. Probabilities to distinguish original from fake can be higher than 90% depending on the style and artist. 

With any sort of machine learning, bias is a huge problem, which is why the algorithm is trained to learn the relevant features of an artist itself. The only feature given as an “input” is the brushstroke. It is not easy to find out what features have actually been learned, so the heat map is produced to provide a visual interpretation of the decision process.

This makes catching forgers a lot simpler for a variety of reasons:

  1. It runs on images alone so the work does not need to be transported. This is helpful because the original assumption is that this is a valuable piece of artwork, so transporting it physically opens up the opportunity for physical damage, loss, or theft.
  2. An inquiry can be responded to quickly as the algorithm can run in a few hours. Traditional committees that judge whether a painting is a forgery or not can sometimes take months to make a determination, with no more accuracy than the computer.
  3. The process is not invasive. Oftentimes, a sample is removed to do a chemical analysis to make sure materials used are similar to what was around in that time period, or compared to what an artist would normally use, but now that does not need to occur.
  4. Finally, there is more objectivity present in the use of a machine versus an individual’s opinion. Two experts could arrive at different conclusions based on less definitive or scientific information.

While spotting forgery is important and valuable work, this technology is also helpful in assigning who should be credited with a piece of work. Throughout history, artists might have the same theme multiple times or their students might have filled in paintings that were started and still given credit to the original for the idea.

Like everything else, this technology is a tool building confidence in identifying original artwork. As noted before, there are some downsides to AI and the algorithms can have bias if they tend to get too much information from a specific period in an artist’s life, particularly if their styles changed over time.

While some are skeptical of how far the technology can go, people will still need to have the final call in determining if a piece of art is authentic or not. However, we can be more confident in whatever verdict is determined. Additionally, this was tested on only a few artists within a specific time period. For other artists who have many strokes in a painting, using this technology will be more difficult as it means there is more information the computer will have to sift through to determine authenticity. Finally, there have been challenges with older paintings as well, as these might have been restored or overpainted several times.

I guess at this point, until blockchain becomes so ubiquitous that we can determine the history of a work from its inception, the only thing we need to worry about is the machine being smart enough to be able to create a perfect forgery itself…

For further Reading:

https://www.artnome.com/news/2019/9/12/can-ai-art-authentication-put-an-end-to-art-forgery

16 comments

  1. therealerindee · ·

    I LOVE this. My wife and I recently watched the Netflix documentary Made You Look which was about forgeries that were sold for millions of dollars out of a respected art gallery in NYC. Highly recommend. But it very clearly demonstrated that the luxury art market is based purely on trust in “experts” who have studied the works of specific artists. I couldn’t believe that the authenticity of a painting that sold for $20 million in the 90s was based on one guy who was like “yeah, looks good to me.” Your post is giving me hope that through the use of AI and blockchain, the art market can stop basically living on fake news and have some sort of data to back up these masterpieces. Using AI to do what it does oh so much better than humans will actually allow buyers and sellers to have some sort of faith in the market. So cool.

  2. This was a very interesting blog. I have no real knowledge on artwork so this was informative and thought provoking. I could see uses for AI in artwork to go beyond just identifying fakes. They say beauty is in the eye of the beholder, but I wonder if there is a way for AI to provide estimates of values of artwork in the future. The algorithms could be fed data on the sale price of items and study their characteristics. And then it could use that data to provide valuations on other pieces of work. I’m sure if its something that I have thought about, its already either in existence or someone else is designing it.

  3. abigailholler1 · ·

    Very interesting way of applying AI to a traditionally non-digital industry – thanks for highlighting this in a blog post! This application of AI for authentication towards non-typical art formats is quite fascinating, and likely just the beginning of a shift to digital for the industry. A couple years ago I read about a sculpture whose design was created using artificial intelligence and whose materials were from melted down computer parts. I found it fascinating, as the artist (Ben Snell) used a computer algorithm with training data from over 1,000 famous statues to create a design for his sculpture, Dio. The training data included Michelangelo’s David, among other famous works. After creating the design for Dio, he broke down the computer used to create the design and used these materials to create the sculpture. I found this article quite thought provoking (linked below); it had me questioning if Snell could really take credit for artwork created almost directly from other famous artist’s work, while at the same time applauding the unexpected and highly unique use of technology. And I have to say, a sculpture made out of computer parts actually makes NFTs seem less crazy!

    https://www.theverge.com/tldr/2019/4/12/18306090/ai-generated-sculpture-shredded-remains-ben-snell-dio

  4. Scott Siegler · ·

    This is such an interesting topic that I didn’t know anything about before reading your article. This is opening up my mind to so many other ways AI could potentially be used for verification and authentication purposes. Also, capturing rates of forgery is incredibly useful in terms of providing a snapshot of where a market is currently sitting and how critical intervention and policing may be. I feel like AI is going to be a huge win for collectors, and ultimately increase values of collections across the board as lemons get weeded out more easily and efficiently.

  5. conoreiremba · ·

    Similar to others above, my knowledge of art is quite poor and not to mention how digital technology is being implemented in the art world. As someone who grew up with a picture of Vincent Van Gogh’s Sunflowers hanging over my dining room table, my appreciation for original artwork is definitely not high. But thank you for continuing the trend of this class which has given meaning to the phrase “learning something new every day”. My concerns around items of value are similar to the last point you mentioned and the idea that technology could get to the point of being able to perfectly recreate original work in a way that is impossible to tell the difference. Nonetheless, it was fascinating to see the application of AI in distinguishing genuine artwork from forgeries and the level of accuracy to do so based on brushstrokes is mindblowing. With so much attention (especially in our class) now being given to NFTs, it is great that to see an example of how technology is being used to enhance one of the more traditional bedrocks of collectibles. Tying onto Scott’s comment above, I wonder how similar methods are being used to spot forgeries across other items of value to give more confidence to collectors, for things outside of art like sports memorabilia or comic books, even bottles of wine, etc (although admittedly not quite the same value as original artwork). Really interesting post.

  6. olivia_levy8 · ·

    The heat map technique is fascinating and something I was definitely not aware of. This is a great application of AI and something that could be very helpful, as I have seen the fair share of documentaries about art ownership, theft, forgery etc. I like the idea that artists have consistency and even if it is nuances, this is where AI and machine learning really kick in. My one question or thought would be about an artist that is possibly known for their style being very vast and different, unlike Van Gogh that may have a very similar style in which brush stroke could be detected. Seeing this application on a Pollock piece or something very unpatterned and not as AI friendly would be valuable.

  7. Great post. This is a great application of AI. I actually think this application is one in which the black box nature of AI is a real benefit. Since forgers don’t know why AI is calling something fake or not, it will be harder for them to learn how to trick the system.

  8. shaneriley88 · ·

    Awesome post! I’ve heard of the Secret Service using similar technology to track counterfeit money. This certainly gives a whole new – 21st century – spin on art provenance. It will be interesting to see if major auctions houses start advertising that they use these techniques or if IS/data science education makes it’s way into MFA programs. This is certainly an application where the complexity of AI and cryptography of blockchain will have interesting impacts as DX moves along in the space.

  9. Very interesting post Ryan! The counterfeits seen in the art market are very similar to the Netflix documentary Sour Grapes. It will be very interesting to see if AI can become good enough to tell the difference between wines as it has been able to do with art. It is fascinating to me that there is so much money in the art market, especially for pieces that are not authentic.

  10. changliu0601 · ·

    Interesting Post!I just read an article about an AI “artist” got a solo show at a Chelsea gallery.Some people think it is a way to reinvent art.But some people think AI artist destroy arts.But i think it is really meaningful to apply AI to identify artwork.

  11. alexcarey94 · ·

    Cool post- very informative on a topic I did not know much about. I actually didn’t know there was a lot of art forgery that even went on in the world. This also makes me think of the application that this could be used on signatures that could identify fraud on credit cards or on legal documents. Related similarly to the blockchain video in class on how the FBI used it for the good to determine the real “bad guys”. I think AI overall will make fraud detection easier and it will be interesting to seeing the applications in other fields in the coming years.

  12. williammooremba · ·

    Great post. One thing this makes me curious about is if either now or in the future AI can be used to help create forgeries. Even if the AI can’t generate sufficiently good new art which would pass scrutiny, it could serve as a pass-fail metric for sophisticated forgers. I would think a forger could take the heat map of problem areas and then try and fix them until the entire painting passes. While a lot of effort, the lucrative nature of these artists work could make the effort worth it. It seems that one of the challenges of technology advances is that it can likely long term be utilized by everyone. As the authentication techniques improve so can forgery techniques.

  13. sayoyamusa · ·

    Awesome post!! This is so cool and I love this fascinating application of technology, but I’m simply curious whether it is OK for art collectors to rely on machines, not on their own eyes for beauty…? On my second thoughts, however, it might be the mix of them like we’ve learned that the best chess champion in the world is not an AI, nor is a human but it’s the team of a human and an AI. Also, your concern that machines can create a perfect forgery itself really resonates with me… but one thing I’m sure is that machines will never be able to create a new art which can disrupt the precedented methods. They can imitate Picasso for sure, but never be a second Picasso who will invent a totally new method and bring a totally new value to the art world. Humans are good at innovation, I believe.

  14. lisahersh · ·

    Really interesting post! I think this technology has a lot of other applications outside of the art world too. I could really see it being applied to a variety of forged document detection, such as IDs, currencies, historical letters, etc. Since it takes pictures, it might also be able to be used for historical 3D artifacts within the field of archaeology too. I remember watching the movie Catch Me If You Can a year ago and thinking that there’s no way his counterfeits/forgeries would fly today. Even thinking back on the TV show White Collar, I think AIs abilities to detect such minute patterns would make a show with that same premise simply impossible.

  15. Really Interesting post-Ryan! I like how you characterized the industry as well as characterized how potential flaws could be created even with blockchain. What I think will be interesting is if there can be a hardware component like an RFID/QR type imprint for physical art that could be administered by the big 4 accounting firms that would digitize authenticity onto a stamp that could be imprinted on the back of the painting.

  16. courtneymba · ·

    Interesting (and beautiful) article! That is incredible about 100% fraud detection in the Rutgers algorithm. Reading the comments above also reminds me of the introduction in Malcolm Gladwell’s Blink book where art historians are assessing if a sculpture is an original or counterfeit. They felt it was because something seemed “too fresh” about it, but the data was showing the age of the calcite deposit was thousands of years old. Based on the aging, they felt compelled to rule in favor of “real.” er by). I wonder how AI could be used in more 3D kind of art applications, as clearly there are holes in dating methods.

%d bloggers like this: