How I built a Slack bot to help me find an apartment in San Francisco

When I was searching for a flat in Berlin I wrote a bot that would continuously crawl EBay small ads and look for suitable apartments (based on surface, neighborhood, price, ...). When finding a compatible one, it would write to the poster and if a phone number was given send it to me via Slack. I would then always be the first person to call or write, and this helped me to find a really nice flat within just three weeks. The hardest part was pretending to have read the ad when people called me back after my bot wrote them, or explain how I could already call them when their ad was online only for a few seconds :D

Here's the code btw:

https://gist.github.com/adewes/c9b2a71457c6c6f01f2f

I like this post not for its technical merit but for its pedagogy. Scraping Craigslist and posting the results to Slack is not rocket science, but the project is described in enough detail to make it accessible to -- and potentially reproducible by -- someone who didn't already know how to do it. IMHO that has a lot of value.

Even with tools to automate the search through listings and get you ahead of other buyers, one of the tricky parts of apartment hunting is its “optimal stopping” structure: you have to decide when to commit without knowing whether an even better apartment might be out there.

This is a famous math problem (the “secretary problem”) with the lovely solution that you should define a time interval over which to search and not commit to anything until you’ve spent 1/e of that time -- 37% -- noncommittally exploring your options.

https://medium.com/galleys/optimal-stopping-45c54da6d8d0#.lj...

I went hunting for a flat a while ago here in New Zealand, I gave up pretty quickly and just ended up staying with relatives. I've now moved cities and I live in a backpackers because I can't be bothered finding a flat (plus interesting people here!).

It's just so draining to have to contact all these people, and view their places. I guess it's like job hunting, except you're paying money. It doesn't help that I'm not very good with strangers.

To find our current apartment in NYC, I just used IFTTT, plugging in the URL of a custom search (neighborhood, price, bedrooms, etc) on craigslist. Worked a treat

Those real-time notifications definitely make a difference.

When I was searching for apartments in the NYCish area, I asked Zillow to ping my phone when a good listing came in.

One day, I got a ping during class and called the broker as fast as I could. "Yeah," he said, "I posted that listing 20 minutes ago but you're like the 6th caller. You almost certainly won't get it, sorry."

While this is technically fine, the process strikes me as unnecessary. I suppose that's not too surprising for somebody new to the area and mostly experiencing it from scary anecdotes.

The list of "San Francisco" neighborhoods being searched are almost all more than 1.2 miles from BART. I'm sure it's hard to find an apartment if you have a robot excluding nearly all listings.

FWIW, there's been an apartment vacant in my building for 2+ weeks, with very few open house attendees. Other nearby buildings are much the same. If you're willing to look slightly outside the most desirable neighborhoods, it isn't that hard to find an apartment. Affordable? Well that's another story; this is SF after all.

Well I also built a program[1] to help me find an apartment in Vienna, in 2011. We actually needed two apartments, and we wanted them close to one another. We quickly found out how bad real estate websites are with moderatly complex queries even when wanting only a single apartment. With the additional constraint of needed two apartments close to each other... yeah, they were mostly useless.

So I built some shell scripts that scrapped all the websites, and generated some tables of pairs of houses sorted by the distance between them that can be easily read by awk. Then I could run queries on them.

All the scrapping was done with regular expressions, no fancy HTML parsing here.

For the distance calculation, I got the geographic coordinates by piping the address of the residence to Google Maps. I then calculated the geodesic distace between them in my scripts. Initially I wanted to let Google Maps calculate the more useful walking distance between residences, but that made the algorithm O(n²), and I ran into Google API free quota issues even with O(n). Geodesic distace was a good enough proxy though.

Being able to use awk, I could use any kind of arbitrary query I could think of. I filtered all residences that were not direct sales (used an agency), they were unfurnished, that were in a place where I didn't want to live (few such places in Vienna though), that were outside my price range, etc. basic stuff.

However, I could create arbitrary utility functions. For example it was really important for me that apartments were close together. So I was willing to sacrifice location, or the total area, or the numbers of rooms if they were really close, but if they were further away, I required more rooms or better location. No real estate agent or website will be able to do this for you.

In the end, it was too much trouble to rent two apartments, so we only rented one. However, the software was still very useful as it presensed all data in a much useful format, multiplexing data from all websites, and the data coming already filtered.

Plus being all text-based, and this being Unix and all that, I could easily manually input the 100 or so metro stations as "houses", so we could sort apartments based on the distance to the nearest metro station. And again I could create arbitrary utility functions. For example the U3 line is much more important for us than the U6 line, and we really don't care about U2 at all.

[1] https://code.google.com/archive/p/operation-housefinder/

I have a feeling, from the title alone, that some people are going to find some way to mock the OP. "You built some thing to do some task? I do it differently. Loser!"

Is scraping CraigsList legal?

I'm currently trying the same approach to get an apartment in the Hague area, Netherlands. I find it very unreliable, but that's probably due to the poor state of house listings here.

The whole experience of searching a suitable house is just aweful, all listing sites are basically full of spam and scams:

- most houses are already taken, but they leave them on the site because free exposure/advertising for the realtor/landlord.

- pictures that are not taken in the actual apartment (they just reuse pictures of a different apartment that kinda looks like it).

- ads that look like normal list items (dark ui).

- super good looking houses for low prices, that turn out to be fake, just to promote the realtor/landlord name.

- loads of hidden costs (service, administration, VAT, parking etc).

- all sorts of requirements (based on sex, income, ethnicity, type of work and what not).

Recently moved to SF and wanted to be able pick prospective apartments from Craigslist based on how close they were to BART/Muni stops...so I hacked together a little bookmarklet to help with this. Check it out if you are in the same boat:

https://github.com/christopher-skeels/add-walking-times-to-t...

Cool project. A chat bot plugin to handle intro convos and scheduling with the brokers would be great :)

Honestly suprised that Craigslist doesn't embrace people interfacing with them via an official API; yes, I know lots about CL, but the after bring on (forgot his name) they changed their back an a lot and a bit on the front too.

Improvement: Calculate real walking distanse using Google Maps or OpenStreetmap.

Anyone else have beautifulsoup crashing at line 160 of __init__.py?

160 p_text = row.find('span', {'class': 'p'}).text

'Nonetype' object has no attribute 'text'

By documenting the process and making this open source it seems like it will be even harder to find an apartment in San Francisco.

Might be worth trying this code for other cities though :)

I built something like this a couple of years ago for my own apartment hunt, but my IP got blacklisted. Has Craigslist changed their policy on scraping?

Has anyone tried to post in the housing wanted section?

Craigslist has a built-in version of this -- email alerts on specific searches. That's the layman's solution. :)

Aren't there existing Craigslist apps that do most of this already?

I feel like all these newfangled slack/whatever bots fall into two categories: stuff that used to be or would otherwise be a (toy) IRC bot, and what basically amount to annoying phone support menus, in text form. Not impressed.

The headline is slightly click-bait. Most the work is outside slack - parsing Craigslist, classifying etc.. The slack channel bit is one part of the puzzle. The article itself says there are 4 steps and devotes 2 paras to slack.

I guess the author chose slack in the title because it will result in eyeballs rather than saying how he used Python, craigslist because they aren't "sexy".