Web Scraping

Condo Price Per Square Foot Near Rittenhouse Square


Wondering the price per square foot of condominiums overlooking Rittenhouse Square in the center of Philly?



Philadelphia Property Database
Web scraping source
In this project, I calculate the price per square foot of condominiums overlooking Rittenhouse Square by scraping thePhiladelphia Property Database.

STEPS
1. Obtain the full list of condos and units for web scraping (in total of 621 rows of condos and units)

2. Connect to Selenium Standalone

3. Write code to scrape the needed information from Philadelphia Property Database
    -  Information I need: the most recent market value and improvement area
    -  Instead of jumping into scraping all needed info for all units, I started by scraping needed info for the first unit
    -  I first navigated to the website, and sent address and unit number to the search bar, then found the correct CSS selector to retrieve needed info, and finally stored the market value and area info to a data frame

5. Write the loop to scrape needed values for all condos units
    -  After successfully getting what I need for the first condo unit, I started writing a loop for scraping info for all condos units
    -  The loop starts with going back to the original search page, getting address and unit for each new search, then going through the steps to scrape needed info, and finally store every result to a data frame
    -  The trick is to include a long-enough pause after each search to prevent failure of the loop

    Below are screenshots of the final results I obtained from web scraping. 
     
Web Scraping Results
6. Calculate the average price per sqft for each condo property
    -  Now I have 605 (out of 621) condos units with market value and improvement area info, as some of the condos units do not have either market value or area info
    -  I calculate the total price and the total area in sqft for each condo, and then calculate total price/total area to get the average price per sqft for each condo.

7. Geocoding
    -  I use a loop again to geocode each condo using its address

8. Mapping using ggplot2
    -  I also create a bar plot showing the calculated price per sqft for each condo from high to low

   The final map created is shown below! 
Condo Price per SQFT near Rittenhouse Square
Philadelphia, PA
RESULT & ANALYSIS
From the final map we can see that, average prices per sqft tend to be similar when the condos are nearby. For example, for 224-30 W Rittenhouse SQ and 220 W Rittenhouse SQ, the difference between their average prices per sqft is only about $13. For 1900, 2830 and 1820 Rittenhouse SQ, their average prices per sqft are very close as well. There is exception, though. Although 1806-18 Rittenhouse SQ is close to 1820 Rittenhouse SQ, its price per sqft is almost $80. Another interesting pattern is that the condo with lowest price per sqft is located the farthest from Rittenhouse SQ.  

Yinuo Yin’s Portfolio 2018