Our Team!

---

Table of Contents

1. Project Idea
2. Synopsis
3. Overview
   3.1. Video Demo
   3.2. System diagrams
4. Parts List
5. Documentation
   5.1. Database
   5.2. Server File Descriptions
   5.3. Arduino Feature Descriptions
6. Milestone Videos

Project Idea

Have you ever seen an item at someone’s house that you are really interested in, but you’re not sure where to buy it from - especially at the best price or closest location? Or with things around your own house, if you run out of something and want to purchase more but don’t remember where you got it? Our project aims to fix that problem through adding a camera to our ESP setup to scan an item’s barcode or use image recognition APIs to identify the object. We would then return the closest or cheapest place to buy the item, depending on the user’s preference.

Shopper Helper

For our project we created a device that allows users to create and maintain a shopping list by taking photos of items and scanning barcodes. We integrated the ArduCAM camera to both display images on the TFT screen and allow the user to take photos of images. These images are then sent to the server where they are processed and an item description is returned to the user to confirm. The user can then view their current shopping list and alter the quantity of items as well as find stores near them (based on GPS location) that are optimized for item price and location.

Overview

Our system consists of a camera module for the user to take pictures of barcodes or purchasable products and an LCD screen to display images, shopping lists, and prompts to the user.

We have 3 options for the user to choose from on the main menu as shown on the LCD: Add Item, Edit List, and Find Store.

Add Item

• This allows the user to use the camera to take a picture of either the barcode or the product to run through our image processing pipeline. Once the picture loads and we get a result, the user can confirm the product or retake the picture in the case the image result is incorrect. The user will select how many of the product they want to add and then the item and its quantity will appear in the shopping list database.

Edit Item

• This allows the user to change the quantity of an item in the shopping list database or remove the item.

Find Stores

• When the user selects this option, they will get a list of the closest stores that have the items in their shopping list.

---

System Diagrams

Wiring Schematic

Block Diagram

Server Side

Parts List

• ESP32
• LCD Screen
• IMU
• GPS
• 2 Push Buttons
• 2MP Arducam

Documentation

Database

photo.db $\to$ Table “jpeg”

Column	data
Type	int
Description	image data where each row entry represents the RGB for a pixel
Example	114

shopping_list3.db $\to$ Table “products”

Column	user	item	quantity	UPC	closest	cheapest
Type	text	text	int	text	text	text
Description	The username that has the item in its cart	Name of a product	The number of that item that is in the user’s cart	The UPC code of the item (can be empty if unknown)	The name of the closest store to the user’s location that carries this product	The name of the store to the user’s location that carries this product for the cheapest price
Example	USER4	acoustic guitar	1	841060045279	Guitar Center	zZounds

Store_Image.py

Because of the large sizes of images, we need to send image data from the ESP32 in small batches. Every time a batch of information is sent, it is stored in a database file to be read out when all of the image data has reached the server. When all of the image data has reached the server, the data is then written into a JPEG file that can then be analyzed for the presence of a barcode.

Shopping_list.py

When a user confirms a new item to add to the database, the ESP32 sends the item description, the GPS coordinates of the user, the item quantity, and the UPC code if the original image was a barcode. This information is then used to query the product APIs that we used in order to find both the cheapest and the closest stores where the item can be purchased. The name of the item, the UPC code, the item quantity, the closest store, and the cheapest store are then all written to the product database to be read back later when the user requests to view the shopping list.

Server File Descriptions

Google_image.py

Goal: Recognize a product from an image

Our script sends a request to the Google API and identifies the image on the left as “chobani greek yogurt”

We utilized Google’s Cloud Visions API to identify objects a user takes an image of using our system. The API requires a key which we obtained, and we also wrote a Python script to send HTTP POST requests to the Google server. In our script we convert the image to a base64 format and convert it to the utf-8 form in order to pass it into the JSON request.

Initially, the results were very generic such as returning “natural foods” when we sent a picture of an apple. The API allows us to detect specific things using features like Labels, Logos, and Web Entities. Whilst testing the different features the API offers, we found that detecting using Web Entities and returning the bestGuessLabel value from the JSON response gave the most accurate results on average.

Find_barcode.py

Goal: Differentiate between image with barcode and regular image of product

In order to do this, we integrated functions from OpenCV in order to process the images passed to the request handler. We assumed the images would be passed in base64 form, so we first decode it before passing it into a numpy array. To follow along with the process of image processing, we included an example image below:

In order to process the image, we make it grayscale using the

greyedOut = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

command. Next, we find the x and y gradients of the image of the image and subtract them from each other so that we have an image with a high horizontal gradient and low vertical gradient. This allows us to look for distinct changes in the image. This is done in the following lines of code:

dx = cv2.Scharr(greyedOut, ddepth=cv2.CV_32F, dx=1, dy=0)

dy = cv2.Scharr(greyedOut, ddepth=cv2.CV_32F, dx=0, dy=1)

  

#combine gradients by subtracting

der = cv2.subtract(dy, dx)

der = cv2.convertScaleAbs(der)

Once this operation is performed, the image looks like this:

Next, we want to blur the image to try and fix some of the space present between barcode lines. This is done in the following code segment:

blurred = cv2.blur(der, (5, 5))

thresh = cv2.threshold(blurred, 225, 255, cv2.THRESH_BINARY)[1]

  
#this allows us to blur vertically or horizontally, depending on direction param

if direction == "Y":

kernel = np.ones((20, 10))

else:

kernel = np.ones((10, 20))

closed = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)

Once done, the images look like this:

Now we want to get rid of some of the smaller whitespots by ‘eroding’ and then enlarge existing white patches by ‘dilating’ in order to fill in some of the gaps. This is done in the following lines:

eroded = cv2.erode(closed, None, iterations=25)

dilated = cv2.dilate(eroded, None, iterations=25)

Once completed, the images look like this:

Finally, we pick the biggest white chunk and create a bounding box around the blob. If the ‘barcode’ it found was big enough (implying it actually existed), then we return True. Otherwise, we simply return False.

Barcode_reader.py

Goal: Integrate barcode recognition

We used the OpenCV library to detect barcodes in photos and then generate the UPC code from the barcode. We broke down the problem into the following three steps:

Finding the barcode in the image

Once an image is verified to have a barcode using the “Differentiate between image with barcode and regular image of product” step above, we need to find the barcode in the image and focus on this area. To do this, we run the image through several filters in OpenCV designed to pick out edges and then focus on edges that meet the criteria of fitting a barcode. Once we have found the general region of the barcode, we remove the rest of the image and focus exclusively on this region. A video of this can be found below:

Rotating the barcode to be straight

While it is important for the user to do their best to provide an image with the barcode already straight, the code we wrote has the ability to detect barcodes at up to a 20 degree tilt and straighten them before analyzing them. It does this step simply by bounding each of the barcode rectangles in a minimum area rectangle and then taking the median tilt of these rectangles and straightening the barcode by this amount. This is important as it allows us to correctly analyze the digits on the barcode in the next step. A video of this can be found below:

Getting the UPC number from a cropped and straightened barcode

The final step in the process is extracting the UPC number from the cropped and straightened barcode. To do this, we again use OpenCV to bound the barcode strips in rectangles before measuring their width and determining the UPC number from these values. A video of this can be found below:

Combining all of these steps

We then combined all of these steps to extract the UPC code from an image with a barcode in it. Obviously, this process still requires that the user provide images of the barcode in decent lighting with good orientation but allows for enough variability that this is not a requirement that is hard to implement in practice for the user. Below is the video of the entire system working properly on five product barcodes:

ProductInfo2.py

Goal: Find product information (price, stores, and availability)

In order to get the product information from barcode numbers, we integrated the UPCitemdb API. We send it get requests with the upc code passed in and then parsed the json result so that it displays all the available offers with the most updated price, store domain, and availability. Here are five examples of us passing in a barcode number to receive information about price and availability:

The other half of our objective was to find similar information for a product (the most updated price, store domain, and availability) given a description of the product as an input rather than a UPC code. In order to do that, we used the Barcode Lookup API (https://www.barcodelookup.com/api), which allows for a simple product description as an input and returns similar information about the product. We also passed in geographic information specifying we were in the United States in order to restrict the search to stores within the nation. Since the input is less specific than a UPC code, the API returns many products that match the description, with all of the availability and price information. We decided to choose the product with the most availability information and return all of the relevant information for that product. Here are five examples with product descriptions that match the five product UPC examples from above:

Shopping_list.py

Goal: Add a database that the user can add items to and remove items from with GUI on LCD

In the video, I…

1. Add 2 Nutellas
2. Add 1 Peanut Butter
3. Remove 1 Nutella
4. Cycle through items
5. Restart ESP
6. Cycle through items (notice all items persist)
7. Remove all peanut butter
8. Remove all nutella
9. Cycle through items (all items removed, so nothing shows up)

How this was done:

Server-side:

In the event of a POST request, we require 3 values: the string of the added item, the username of the user, and the quantity that we are adding or subtracting.

There are 2 cases to inserting an item:

1. The item already exists in the database - In this case, we query the database to find the number of that item already inserted into the shopping list. We update the table with this item to add the quantity sent in the form to the quantity already in the table. In the case where the addition results in a nonpositive number, the script deletes that row from the table.

2. The item does not exist in the database - In this case, the script simply adds the item and its quantity to the table if the quantity is positive.

Both the GET and POST requests return a string formatted with product names and their quantities separated by semicolons.

Additionally, this code finds the closest and cheapest stores to buy an item that is being added by using GPS coordinates provided by the user. Next, we pass the item name to another previously made piece of code (ProductInfo2.py) that queries an API with an item description to find a list of stores where it is available and the price of the item at that store. Once we’ve gathered all of that information, we then decide which store is closest to the user and which contains the items at the cheapest price.

Then we iterate through the possible stores for the item and return the store where buying the items is cheapest. We also make use of another API (TomTom Maps) to find the distance from the user to each of the possible stores. Since some of the stores are just websites, we discard those as possibilities here. Then we iterate through the possible stores for the item and return the nearest store.

Find_stores.py

Goal: Return the cheapest/closest stores to buy all the products in the shopping list

find_stores.py accesses the table products in the shopping_list3.db file. Depending on whether the goal is “close” or “cheap”, the script will return a list of new line seperated stores from the cheapest or closest columns of the table.

Arduino Feature Descriptions

Display camera input on LCD Screen to allow for easy user feedback

Here is a video of the deliverable: https://youtu.be/YdldBCKY-mY (embed)

There were two parts to this deliverable:

1. Adding the ability to switch between camera modes (JPEG and BMP) since JPEG is much smaller and can be uploaded to the server and since it is easy to extract pixel values from BMP.

2. Reformatting the image data from the camera and sending it to the LCD.

1. Switching Camera Modes

To allow for switching camera modes, we added button functionality that allows the user to press the button to take a photo and send it to the server. When the user is not pressing the button, it returns to display mode, where the output of the camera is displayed on the LCD screen. To convert between these two modes, we write to the registers found in the InitCAM ArduCAM function to specify which mode we would like to use.

2. Reformatting the image data from the camera and displaying it on LCD.

The first challenge was figuring out what format the ArduCAM was returning the BMP data in and what format the LCD needed. It turns out that in both cases it was RGB 565. However, In the case of the ArduCAM, it was returning the low byte and then the high byte for each pixel when in fact, the LCD needed the concatenation of the high byte first with the low byte second. After realizing this, it was simply a matter of creating an array of these 16 bit RGB 565 values and using the pushImage function of the TFT to display them on the screen.

Integrate product lookup systems so that the user can find the stores nearby that contain all (or most) of the items on their grocery list for the cheapest price

In this video, the user finds the closest and cheapest places to buy ritz crackers. Then, using Postman, they add nutella to the shopping list. This time, using cheapest, the location is the same. However, with the closest option, now both Walmart and Target are returned.

On the ESP side, we added another state for finding stores. In this state, long pressing the left button will change the screen to the main menu. Short pressing either of the buttons will alternate between selecting cheapest and closest stores. Long pressing the right button will confirm the selection and send a get request to the server side script. This request will be formatted with the selected goal (described further below) and the GPS’s longitude and latitude. Then, the output of the script will be displayed on the LCD, listing the stores.on.

On the server side, we made a request handler that integrates work from previous milestones. The request handler takes in three arguments - goal, lat, and lon. The latitude and longitude represent the user’s current location, while goal can be one of two arguments: ‘close’ or ‘cheap.’ The first thing the server does is read all items and corresponding quantities currently in the shopping list from the shopping list database that was already made. Next, for every item on the shopping list, we pass the item name to another previously made piece of code that queries an API with an item description to find a list of stores where it is available and the price of the item at that store. Once we’ve gathered all of that information, we then decide based on the value of goal whether to return a list of stores that are closest to the user or contain the items at the cheapest price.

If goal is “cheap”, then we iterate through the possible stores for each item and return a list of stores where buying the items is cheapest. If goal is “close”, then we make use of another API (TomTom Maps) to find the distance from the user to each of the possible stores. Since some of the stores are just websites, we discard those as possibilities here. Then we iterate through the possible stores for each item and return a list of the nearest stores.

Milestone Videos

Week of	Milestone	Demo
April 21
	Differentiate between image with barcode and regular image of product	Show 10 images (from online or taken with phone) being processed by python script with output classifying the image as image with barcode or regular image
	Integrate Google vision API product recognition	Show 5 different product photos (from online or taken with phone) being processed by a python script with output being the name of the product that can be used to query the store database
	Integrate barcode recognition	Show 5 different barcode photos (from online or taken with phone) being processed by a python script with output being the UPC of the product
	Set up version control	Show the github and log of commits

Week of	Milestone	Demo
April 28
	Integrate all three systems from last week (full pipeline from user taking photo to deciding what type of photo it is and then outputting UPC or product info). Additionally, work to ensure robustness of this pipeline and that if one part fails (UPC code cannot be scanned for instance), it falls back on a different part of the system (image recognition for instance).	Show 5 photos already on the server that are taken as input to a python script and then either output the UPC (if barcode) or a product description (if not barcode)
	Add camera to board and get photos uploading via post request.	Show photo taken on board uploaded to the server and then shown on a separate computer by initiating a GET request.
	Find product information (cheapest price, stores, and availability) using multiple APIs such as Walmart API, UPC code API, etc.	Show as input 5 UPC labels and product descriptions and as output show the prices, stores, and availabilities of these products
	Add a database that the user can add items to and remove items from with GUI on LCD	Show LCD screen with list of items and ability to add or remove items. Show that this list of items remains unchanged even after the device is powercycled.

Week of	Milestone	Demo
May 5
	Display camera input on LCD Screen to allow for easy user feedback	Show input to camera being displayed on screen
	Integrate all image processing systems created so far so that user can take a photo, have it upload, and add the information to the grocery list which can then be modified for quantity/add and delete	Show user taking photo and having the item appear on the grocery list.
	Integrate product lookup systems so that the user can find the stores nearby that contain all (or most) of the items on their grocery list for the cheapest price	Show user looking up best store to buy the list of items from considering current location and price
	Display product look up results for the user on the LCD screen for confirmation before looking for stores	Show input as image and having the LCD confirm it is the correct product

Week of	Milestone	Demo
May 12
	​Integrate all systems into one working product	Video showing the entire project including taking photos to items to add them to the grocery list (with both barcode recognition and google vision), confirming items, removing items from grocery list, finding optimal store (based on location and price), and changing quantity.
	​Solve blocking issues that occur while displaying photo on LCD	Show in video of system that performing a task of the ESP32 is not prevented while images are being displayed on the TFT
	​Use store locator to choose optimal store based on distance and price	Show a store being chosen and returned to the user from a list of stores based on some predetermined or user chosen metric of distance and price