The purpose of this project is to develop a Machine Learning model that can predict a restaurant's success based on its features. Success is measured through Star Ratings - from 1 to 5 stars.
We gathered data from Yelp on restaurant attributes and ratings, reviews and users.
In an attempt to achieve these predictions, we set out to develop the best Machine Learning Model that can accurately predict the top restaurant attributes associated with higher or lower ratings.
Testing a series of Machine Learning models, we hope to establish which one yields the strongest performance. Based on all possible combinations of attributes and additional features, including census data on the postal codes of the restaurant locations in our dataset, the models will make rating predictions. We assume that the model with the strongest performance will be able to accurately make predictions.
The COVID-19 pandemic has presented a multitude of constraints and economic downfalls, in which the restaurant industry has taken a big hit. Now, more than ever, restaurant owners can greatly benefit from the insights presented by what consumers currently value the most in the food industry — to best stand out from other local competitors.
By accessing data on the top-rated attributes, restaurant owners and investors can learn how to best appeal to the public in such ever-changing times. Just as the nature of our social interactions has changed, so have current demands and trends. Therefore, it is of critical importance for business owners to remain well informed on the story the data tells to better understand social transitions.
The predictive model developed in this project can be applied to other regions and can also be used by various stakeholders, in analyzing upcoming opportunities and their advantages and disadvantages.
In this group project, responsibilities were shared equally among all team members. Each team member participated in generating content for the various areas of the project.
This is a layout of responsibilities per area that each member oversees:
| Member | Responsibilities |
|---|---|
| Aigerim Zhanibekova | Machine Learning; Manage the GitHub repository; ERD diagram and SQL schema; README write-up; Dashboard Creation |
| Ana Bisker | Data Cleaning and Merging; Machine Learning; README write-up; General Project Manager; Dashboard Creation |
| Hristo Bustamante | SQL Database Creation; Data Cleaning and Merging; Machine Learning; README write-up; Dashboard Creation |
| Leila Corvera Fontela | SQL Database Creation; Data Cleaning and Merging; README write-up; Dashboard Creation |
| Ngoc Chau | Data Cleaning and Merging; Data Extraction; Data Transforming; SQL Database Creation; README write-up; Dashboard Creation |
-
What attributes are the most important in making a restaurant more or less successful?
-
How are the combination of attributes correlated with a restaurant's rating?
-
Can we predict a restaurant's rating based on its current or potential attributes?
-
Examples of important attributes:
- Review Count
- Median Income(dollars)
- Total Estimate Households per Postal Code
- Noise Level
- Wheelchair Accessibility
- Delivery
- Takeout
- Outdoor Dining
- Vegetarian Options
- Free Wi-Fi
-
Are there any significant differences across regions?
- Null Hypothesis = H0 = There is no correlation between the rating of a restaurant and a particular combination of attributes.
- Alternate Hypothesis = Ha = There is a correlation between the rating of a restaurant and a particular combination of attributes.
The data used in this project is Yelp's businesses, reviews, and users subset data extracted from Kaggle.
-
The original raw data contains business information across metropolitan areas in the USA and Canada. For this project, we will focus on the USA area. Our raw data included the following states: AZ, CA, CO, DE, FL, HI, ID, IL, IN, LA, MI, MO, MT, NC, NJ, NV, PA, SD, TN, TX, UT, VT, WA.
-
The data was last updated on February 16, 2021.
-
There were 150,346 records in total, which were filtered to only restaurants.
-
After filtering and cleaning, we have 27,137 records. 13 states are covered by our analysis: AZ, CA, DE, FL, ID, IL, IN, LA, MO, NJ, NV, PA, and TN.
We visited Census.gov and filtered by state, income, and total household for our census data table.
- Our original data contained 9,745 records.
- After cleaning, we had 9,045 records.
As part of this project, we have developed a database in PostgreSQL using pgAdmin. As demonstrated in the Entity Relationship Diagram below, the database consists of four tables:
- Business
- Census Data
- Reviews
- User
The ERD below demonstrates how each table is related to each other and specifies a data type for each column.
-
The Census Data table has a one-to-many connection to the Business table through the Postal_Code column.
-
The Business table has a one-to-many connection to the Reviews table through the Business_ID.
-
The User table has a one-to-many connection to the Reviews table through the User_ID.
We used three tables with synthetic datasets generated through generatedata.com.
These were our custom-tailored synthetic datasets:
We merged clean data from Business and Census tables on our SQL database and titled the new table Merged Data. We used SQLAlchemy on a Jupyter Notebook to create a connection for the merged table to be read into the Machine Learning model.
We aim to upload our Reviews and User Tables to our SQL database in the upcoming segments, for visualization purposes only.
These are the datasets that we have, in full clean and operational CSV format:
The query created on PgAdmin to create tables in the yelp_DB SQL database can be visualized here.
Below are screenshots of our SQL database per table:
We attempted to upload our Reviews and User Tables to our SQL database in the upcoming segments, for visualization and future uses only but were not successful as the datasets were too large.
Instead, we stored our datasets in a shared Google Drive which can be found here:
For future use and further research, we would use AWS and Postgres to store and manipulate the data.
Business, Reviews, User tables - came in JSON format, and it required significant cleaning.
We executed the following data cleaning tasks:
- All categorical and text variables were converted into numerical data.
- Filtered the data for restaurants.
- Checked for data types, duplicates, and null values.
- In the Business JSON dataset, all attributes of interest were in one column. Each attribute was extracted into a separate column.
- Renamed and reordered columns in a readable format.
- Split the Review JSONs into 11 files, as data was too large to be manipulated.
- Finally, all JSON files were converted to Pandas DataFrames and then to CSV files.
Census table - imported from census.gov in CSV format.
- Columns were separated according to the data target.
- Dropped columns that were not needed and merged the ones that were of interest.
- Renamed and reordered columns in a readable format.
- Checked for data types, duplicates, and null values.
All data CSV files were imported into Postgres according to Schemas that were created by queries.
We conducted extensive exploratory analysis before going into machine learning.
The code available here.
We examined the rating distribution and its relation with each restaurant attribute and income and population data from Census. Each relationship was examined through boxplots, scatterplots, and bar charts.
- In most cases, the average restaurant ratings had minimal difference across the selected restaurant attributes.
- Review Count control seems to be important. Restaurants that fall under 4 of 4.5 rating categories tend to have the largest number of reviews, on average.
- The attributes that had the most difference in the mean star ratings are:
- Wheelchair Accessibility
- Noise Level
- Good for Kids
- Good for Groups
- Data from Census seems to show some difference across rating categories.
In this section, we will describe our progress in Machine Learning model implementation. We have fundamentally decided to use Supervised Machine Learning methods. We measured success through Ratings (from 1 to 5 stars), as our target variable.
We began with a Multiple Linear Regression model, on Segment One. Based on the results from the Linear Regression Model, we moved to classification models with a categorical target variable for Segment Two.
On Segment Three we scaled the data and experimented with different kernel settings in SVM.
We used Multiple Linear Regression as our method. Our prototype was built with synthetic data that returned numerical predictions which corresponded to ratings. The code can be found here.
Going forward, we intended to test our analysis and metrics using our real data to select the best predictive model.
Subsequent to the submission of Segment One, we input our actual data into the Linear Regression model. Doing that, we observed that the R^2 of this model was no higher than 0.14. The code for this stage of Machine Learning testing can be found here.
After inspecting the data more closely, we found that the target variable of 'stars ratings' follows a discrete distribution with only 9 unique values. That fact alone eliminated the prospect of keeping the Linear Regression model, which requires the target variable to be continuous values.
We moved to classification models with a categorical target variable. Before arriving at our strongest-performing model, we went through a series of due diligence stages. In each stage, we tested the following models:
- MultiClass Classification Logistic Regression and its Resampling Techniques to address the classes imbalance
- Balanced Random Forest
- Easy Ensemble AdaBoost Classifier
Steps to Arrive at the Best Performing Model:
First, we classified the ratings into four classes: poor, average, good, and successful.
- We started with 10 features and added 5 more restaurant features, which increased the accuracy.
- Furthermore, we added socio-economic controls such as income and total households, which slightly increased the accuracy further.
However, we obtained a stronger performance with two classes only: lower- and higher-performing restaurants.
- We adjusted the threshold dividing the two classes to eliminate class imbalance. Having balanced data gave more equal prediction rates across classes and, therefore, the most unbiased overall accuracy.
- Finally, adding the 'state' feature also slightly improved the model indicating small structural differences across examined regions.
We compared each model based on its overall accuracy score and an F-1 score for each class, a harmonic mean of precision, and recall rates. The F-1 score metric was used since neither precision nor recall has higher importance, given the topic question.
The Easy Ensemble AdaBoost Classifier model predicting 2 classes yielded the highest and most unbiased accuracy of 66.68%. The model features included all 15 restaurant features, Census income and household data, and region control. The code for the Final Machine Learning Model can be found here.
Since the first stage, a series of verification tests lead us to over a 20% accuracy increase in the strongest model.
You can find more details on our Machine Learning model development process here:
- Here you can read a detailed description of every classification model and method we tested, and their comparisons.
- All the code for classification models is saved here.
To note: Initial experimental stages of the Machine Learning process were conducted using tables that were merged using Pandas and read directly from CSV files. The data for the final model had 24,739 observations and yielded 66.85% accuracy.
Subsequently, some of the observations had damaged address entries and had to be dropped during the SQL table merge in the Database Creation stage.
The final model that connects to the SQL database and table merged there now only uses 24,681 observations, which dropped the accuracy to 66.68%. This incident might indicate that having more observations could give us more accuracy.
The following steps did not improve the strongest model further:
- Scaling data and running all models again.
- Testing further with the Support Vector Machine model:
We did research and tested various Support Vector Machine or SVM machine learning methods. After scaling the data with StandardScaler(), we reached the best performance with rbf kernel, random_state=1, gamma=0.05, C=0.01.
There was zero overfitting but the accuracy was still much lower than the AdaBoost Ensemble classifier. We acknowledged that there may be parameters we are not yet familiar with in SVM. Had we had more time and a larger number of observations, this model may have been a promising prospect.
The code can be seen here.
During our machine learning stage, we examined features by three methods: variation, correlation, and importance rank. Each method provides different criteria for selecting a feature. In the end, it is a researcher's judgment call. For us, these methods helped to choose the Median Income feature over the Mean Income.
The code is available here.
Higher variance means more heterogeneous data. Potentially, features that have more variability add more representativeness to the dataset.
Before running our models, we examined the correlation between all features and ratings. As it can be seen, most of the features do not have a very strong correlation with the ratings. Nevertheless, these features constitute an important part of the restaurant's performance as shown by the Machine Learning models. The variable with the highest correlation is WheelChair Accessibility.
We can also see a relatively high correlation among the Census population features of the Total Households Per Zip, Total Married Households Per Zip, and Total Non-Family Households Per Zip. This observation is expected as all these values are representative of the population. However, leaving only one population proxy did not affect the models.
Although the Balanced Random Forest produced overfitting results in the majority of settings, its feature importance calculator is useful. In the chart above, all features are ranked according to their importance. Although we tried removing less important features, the accuracy score dropped slightly, i.e., all less important features collectively make a meaningful contribution to the model.
We used Tableau to create and host our dashboard. We made a Tableau storyboard that can be seen here.
Screenshot below:
Google Slides will be used for our live presentation. Our slideshow can be viewed here.
Screenshot of our slide structure below:
-
Sentiment Analysis on Reviews
- The unexplained part of the model may come from other features that might be restaurant-specific such as cleanliness, menu and food attributes, service attributes such as the number of servers, politeness of staff, etc.
- However, these data are not available in Yelp attributes but can be extracted based on keywords from review comments. In general, review comments analysis might reveal other factors determining success in the restaurant industry.
-
A deeper study into User Yelp profiles
- Analyzing user profiles based on data from the User table on Yelp data
-
Capture even more recent data to have a wider scope, and keep track of trends year-by-year and oscillations due to behavioral shifts stemming from the pandemic, and society's transition out of isolation.
-
With more time and additional data points we could have gone further into exploration of a more effective predictive machine learning model.