Digital Transformation in Grocery Stores — Business Applications of Computer Vision, Sensor Fusion, and Machine Learning

Last year, as the Covid19 continues hitting us, Amazon announced that it will offer its “Just Walk Out” technology used in its own Amazon Go stores to other retailers, enabling a cashless grocery shopping experience to traditional grocery stores that moves into digital transformation. Amazon Go first opened back in 2016 in Washington, Seattle, which allows customers to grab the items and exit the stores without going through the old-fashioned checkout systems. However, before talking about the technology used in Amazon Go. Let’s start beginning by reviewing the current grocery store shopping experience.

There are roughly six steps in the traditional grocery shopping experience – walk-in to the store, grab the shopping cart, find the put the items into the shopping cart or shopping basket, wait in the checkout line whenever ready, start the checkout process, and finally put purchased items into the shopping bag. This complicated, time-consuming, and inefficient process has a lot of problems. First, customers waste their time waiting in the long checkout queue when it could be saved. Second, after a long time waiting in the queue, the customers finally got to the checkout process. They need to grab all the items to the counter to let cashiers scan every single item customers purchased. Then customers need to choose their preferred way of payment, coupons, discounts, etc… The last step is to, then, put all purchased items into the shopping bag. On the other hand, the inefficiency also increased stores’ costs. E.g it’s really hard to measure the traffic of the grocery stores, and the customer data collected is so limited to empower the grocery stores on cost-saving and inventory controls. The overall shopping experience can be improved by modern technology. It’s time to introduce digital transformation in grocery stores.

Nowadays, we have seen a lot of grocery stores started the digital transformation, especially in the supermarket industries. Walmart and other supermarket giants introduced e-commerce on top of their existing in-store shopping. Target also announced “order online, in-store pick” to alleviate efficiency problems and improve customer service. Most stores, such as CVS, have self-checkout systems which significantly reduced the checkout time of customers. Besides, there are tons of delivery companies, in partnership with grocery stores, started to offer same day or 2-hour delivery service. The process is 1. Customers order online 2. Delivery companies send shoppers to shop on behalf of customers 3. Delivery of the ordered items directly to the customers’ house. Because the browsing experience online is nothing comparable to the in-store browsing experience. (pictures versus actual feel) Although all of these are solutions commonly adopted by the grocery stores market, the questions remain. How do we provide an efficient in-store shopping experience?

To solve this problem, unlike other solutions, Amazon introduced “Just Walk Out” technology back in 2016. In the first marketing video Amazon posted, it specifically says “no checkout. Seriously, no” Amazon go adopts most advanced technologies – beacons, computer vision, and machine learning—that are used in autonomous cars in the grocery shopping experience. The whole process becomes 1. Download the Amazon Go App 2. Scan App QR Code to enter 3. Choose your items 4. Leave 5. You are charged automatically for the products you selected. There are no checkout lines. How is it even possible? Let’s dive deep and discuss some technology used in Amazon Go.

SEATTLE, WA – JANUARY 22: Shoppers scan the Amazon Go app on the mobile devices as the enter the Amazon Go store, on January 22, 2018 in Seattle, Washington. After more than a year in beta Amazon opened the cashier-less store to the public. (Photo by Stephen Brashear/Getty Images)

The heart of the store is the computer vision-based machine learning that is used to track the movement and the intention of everyone in the store. “According to Dr. Medioni, the Go store was a “Computer Vision Complete” problem, a reference to the NP-Complete class of algorithmic problems from Computer Science. Within that top-level problem, 6 core problems needed to be solved to provide the experience.” Says Ryan Gross. The first one is Sensor Fusion, which aggregates all the signals across different sensors such as phone and beacon. The second is Calibration, which lets each camera know its exact location in the store. Person detection can constantly identify and track each person in the store using machine learning. Object recognition is used to distinguish the different items being sold. What exactly each person near a shelf is doing with their arms is detected by pose estimation tech. Finally, activity analysis determines whether a person has picked up or returned an item. In the combination of all technology, amazon go provides an astonishing cashless shopping experience.

Although the technology seems appealing to us, there are several implications and impacts associated with it. First and foremost is the impact on the workforce. “According to the Bureau of Labor Statistics, in 2016, there were 2.7 million people identified as being employed by retail grocery stores, 856,850 of whom are employed as cashiers [5]. This number doesn’t even reflect the 3.5 million cashiers that are employed across all industries, not just the retail grocery industry ” A large number of employees may be affected by the disruptive technology brought by Amazon Go and its cashless model. The second is the reshaping of customer’s shopping behavior and their expectations of grocery shopping. As tech is adopted widely in the industry, customers may expect such a convenient way of shopping. For traditional grocery stores, it’s gonna be a big threat. And they need to improve their customer service to compete with such technology.

Lastly, covid19 has changed the way of people shopping, and it especially accelerates the digital transformation in the grocery retail industry. “In a matter of a couple of weeks, we were already ahead of our end-of-year goal. A week later, we were ahead of our 2021 goals, and a few days after that, we were ahead of our 2022 goals. And so, at a certain point, we stopped counting.” – Instacart. As people are more used to the new way of grocery shopping, and a lot of solutions introduced to digitally transform the grocery stores, I always have this question “What’s the future grocery shopping will look like?”


  1. abigailholler1 · ·

    This amazon go store is one that I was dying to go to pre-covid! I can really see how it will change the behavior of both customers and the industry. I do however wonder how COVID will impact the success of this shift in the model – many people have transitioned to the grocery delivery example you mentioned above, and I wonder how many of those consumers will switch back to in-store shopping. The other point you noted above which will be super interesting is the data collection aspect; as Amazon Go tracks location of shoppers, how will this impact product placement and sales tactics going forward in the grocery industry? Very cool technology, and I’m holding out hope that the next pilot location is somewhere in MA!

  2. I liked how you tied in the impacts this can have on the grocery stores and the people who work there. I never thought much about the operating model of grocery stores but I thought you did a great job outlining how inefficient it is. By moving to a digitally focused model, the grocery stores should have opportunities to minimize costs while also finding new ways to push consumers to purchasing more items. The jobs that could be lost in the process of becoming technologically advanced hopefully will be somewhat offset by new jobs that are needed. I could see how grocery stores will have a much greater need for staff to manage their technology operations.

  3. ritellryan · ·

    I agree that the grocery store is one of the more operationally inefficient places I frequent. When I was growing up our stop &shop (and now the Wegmans in Chestnut Hill) have the barcode scanners so you can scan and bag as you go. I love being able to do this because then I can go to a checkout area and point the scanner at a barcode and I am basically done. It saves the time of putting things on a belt and bagging (not to mention if there is a line it moves much quicker).

    While the concept of the Amazon store is great, the one concept I struggle with would be in regards to produce since the price is dependent on weight. I don’t think Amazon sells anything that has to do with that, so it is something they have not had to think about. I am sure the capability exists, or can be developed rather easily, but that is really the only hurdle that I can see that would keep this from being adopted very quickly. While more things are getting delivered I believe people do like being able to pick out their own food in many cases, and for me sometimes walking the aisle does trigger things that I want to buy for the week that I wouldn’t get online (just filling and emptying the cart).

  4. therealerindee · ·

    Awesome post! When Amazon piloted this idea, I was fascinated. I think the US is way behind in terms of digital payments, and I think the fact that you can just pop in, grab what you need, and walk out is amazing. You did a great job at highlighting what this does to the grocery workforce, but I am also curious how this impacts customer behavior. There was a very steep learning curve when self-checkout became the norm, and I would imagine the implementation of this tech in more stores would also cause quite a traffic jam. I would also very much have to get over the fact that it would feel like stealing if I just walked out without paying. May take me a few times with that one.

  5. When it comes to revolutionizing the shopping experience, I think Amazon has been a leader in this category. They were one of the first companies to offer buying recommendations and personalize the digital experience. Very interesting in terms of how they are working to digitally transform the physical experience. While certainly disruptive, I think that the workforce that is laid off by the grocery chains will be absorbed by personal shopping or delivery labor like Instacart or Doordash. These workers will still benefit in the long run as the personal delivery companies will be able to process more throughput/orders. In the end, a win/win for all of those involved.

  6. Jie Zhao · ·

    I remember watching a video about Amazon-Go a few years back and have also wanted to visit since (sadly haven’t had the chance yet), but I think this is another one of those technologies that can be a little scary as it seems like every movement is being tracked and saved down to a server. I also think as more stores use this technology in the future, it will take some time for customer behavior to adapt and walk out of the store without feeling something is missing. I also liked how you brought up how it will disrupt our workforce since these technologies will continue to replace many jobs, and I wonder how/if companies have plans to provide resources and training for people impacted to be qualified for jobs that aren’t taken over yet.

  7. sayoyamusa · ·

    Great post, Lewis! I’ve been curious about Amazon Go and it is so nice to learn the details from an expert like you! Your explanation about solutions for core problems is clear and I’ve learned a lot from tech-system perspective.
    I also like you’ve brought up the possible impacts on labor force and consumer behavior. Especially the latter one is extremely important to the other consumer product companies like us. I remember that when we needed (and still struggle) to adapt to the rising e-commerce market, we should promote conventional consumer behaviors such as cross-buying and impulse buying in a totally new way. Again, we should update our marketing tactics to keep up with changing consumer insights and behaviors.
    One thing I’m just wondering is if/when Amazon will implement the technology into Whole Foods. I can imagine that Amazon might want to start from the smaller, controllable store and to appeal the news with the catchy store name under Amazon brand. But it will be interesting to see how Amazon integrate its own retail stores along with its online site.

  8. courtneymba · ·

    Great post and presentation yesterday! On the creepy/cool line, this “just walk out” concept is super cool! Like others mentioned, I was so curious to try this out myself and befuddled as to how it worked. To Sayo’s point, I really hope they implement in Whole Foods so I can actually experience myself. Check out lines are awful, especially with young kids. They’re antsy and there’s a ton of candy that I’m constantly saying ‘no’ to, and it’s just miserable for me, the kids, and anyone before or after us lol!

%d bloggers like this: