Find the best apartment out of thousands using Web Scraping, Google APIs and Sheets

Rental hunting is hard, time consuming, and repetitive — let’s fix that.

The lease was ending on my current apartment, and I needed something new.

I wanted something

• Within walking distance of the beach
• A short bus to the gym
• A close enough drive to work

I wanted to optimize for the best possible one bedroom apartment across those three metrics, within Sydney Australia.

I looked through apartment listings one by one on Domain.com.au, a popular rental website in Australia. In each listing I had to

• see if the apartment looked nice and if so,
• Pull up google maps
• find out how far the place was walking from the beach
• find out how far the place was bussing from the gym
• find out how far the place was driving to work,

and record all of this in a spreadsheet for rating later.

As an engineer, it wasn’t long before I became overly frustrated with how repetitive task — there HAD to be a way to automate this.

So I whipped up a quick project.

JoeGaebel/rentifier

It takes a few inputs.

Firstly, your points of interest, and how you want to arrive there.

In my case, I cared about drive and walk time to a few beaches, and the drive and bus time to work and the gym.

Then, you provide the search query URL from Domain. There, you’re able to set the amount of rent you want to spend, how many bedrooms, what areas, etc.

Here’s my example query URL: https://www.domain.com.au/rent/?suburb=bondi-nsw-2026,clovelly-nsw-2031,bronte-nsw-2024,bellevue-hill-nsw-2023&ptype=apartment-unit-flat,block-of-units,duplex,free-standing&bedrooms=1-3&bathrooms=1-any&price=0-550&excludedeposittaken=1

This URL was generated from Domain after I applied the filters I wanted. As you can see, there are a few suburbs listed, it’s set to look for apartments, and my price range was about $550 per week (keeping in mind COVID bargaining would be occurring).

Following that URL returns almost 500 properties to look through — WAY too many!

Using browser automation in the form of TestCafe, Rentifier traverses your search results from Domain. From each of the results it records the price, the area, and most importantly, the address.

From there, the program reaches out to the Google Maps Distance Matrix API, and provides the list of all the addresses it scraped from your search results, along with your points of interest, and how you’d like to make it to your POIs. It does this in batches, so you can scrape thousands of results at a time.

Google Distance Matrix API then returns the amount of time it takes for each address to each point of interest.

Some parsing occurs, and voila! A CSV is spit out, one row per property. It includes the address, the price, and the time in minutes to each of your points of interest, in the mode of travel you’ve specified.

Here’s the program running on a smaller result set

The beauty of the CSV is that it can be imported into Google Sheets with an easy drag and drop.

Which gives you something like this:

Based on your search criteria in Domain, this could potentially return THOUSANDS of properties! So more filtering magic is required.

From there, you can add Conditional Formatting. For example, adding a gradient on the rent amounts, and the travel time.

Here’s the above spreadsheet, with gradients applied to rent.

The cheaper the rent, the more green it is, trending to red for the most expensive

This allows you to be able to judge the property immediately.

From there, you can apply the gradient to the travel times as well, and most importantly, SORT by your criteria, e.g. time to your favourite beach.

Here’s a close up of all the conditional formatting rules applied, and a sort on the closest places to Bronte beach.

Looking at the first row, you can see quite quickly that the walk and drive to Bronte is quite fast, and due to how dark that green colour is, you know it’s pretty close to the other beaches as well, compared the the rest of the properties in the sheet.

However, you’ll notice that the rent price is slightly yellow, meaning it’s good, but on the slightly more expensive side, relative to the other properties.

And as you move to the right most columns, the time it takes to get to Unity Gym, and Pivotal Labs is a strong yellow, meaning average across the rest of the properties.

From there, I was able to apply different sorts and filter views to this sheet. “Closest to Bronte”, “Closest to work”, etc, and sample the top properties of each of these filter views.

This makes sifting through these properties go from THOUSANDS down to HUNDREDS as you can get at the top properties based on your criteria, ie. cheapest, closest to work, closest to Bronte beach, etc.

From there, I would view the property listing on Domain using the handy link in the second column, and rate them on a scale of how excited I was about them. As a nod to some endearing apartment hunting experiences with friends, that scale ranged from Nopesauce, all the way to Totally F***ing Dope Sauce. I left that rating a column.

Sample ratings of a few properties, red meaning the worst, dark blue meaning the best

This narrows the results down from HUNDREDS to TENS (in my case it was about 30 TFDS places)

From there, you can create further filter views to show only your most favourite properties.

As you can see from the above, each property is rated the best (TFDS).

This is still too many to choose from, and I needed another way to narrow it down, so I introduced another scale of 0 to 10. 0 being perfect beyond my wildest dreams, and 10 being bleh.

This further focuses down your efforts on to your most favourite properties. Allowing you to focus on scheduling viewings, and adding more notes as you decide which property to move forward with.

In the end, I had scraped 1500 properties. From that list, I viewed the listings of and rated about 200. From that, I favourited 40. From that 40, I narrowed that down to a top 10, from that I visited 8 in person, from that I applied to 5, and from that I now live in 1.

1500 places down to 1.

Which is where I live now, close enough to walk to the beach, and a short drive to the gym, with warm streaming light for working from home, and some very sweet neighbourhood cats.

To get this to work for you on Domain.com.au will be quite easy, just clone and follow the readme listed here:

JoeGaebel/rentifier

However, this can be readily adapted to work with any other property website, given you update some selectors.

I hope this makes your rental search faster and more systematic, so that you too can live in a totally f***ing dopesauce place.

With Love,

Joe ❤