Hacker News Reader

Comment by lachyg | Hacker News Reader

PHP. We've got to collect about 10 XML feeds. Basically the site loads, but it takes time for data to come in.

We collect the data, then filter it, then display it. Will do a Review My Startup in a few hours :)

Mate, just send out a naked response containing all the static elements, then refresh the content via ajax.

The users will see the full page instantly, but all the data-heavy tables will have spinner.gif laid over them until they're loaded.

You should look into caching the results from the XML feeds in a database, as well as caching the output of your app. For example, if your app needs to pull weather data from one XML feed by city, and stock quotes from another by ticker symbol, consider using a cron to pull these at 5,10,15 minute intervals and store them in a database. Then, have your app always look at the database for the data. You can even cache these queries using memcached, etc. Then, cache the output of your app in a static file and serve that with a 5,10,15 minute timeout. Most of the time your users will get the static cached file, and if not, they will get a page generated from a memcached query result, or pulled from the local database.

You never want to serve up that many XML feeds live because when one of them is slow or down, so is your site.

Second this.

I built a similar site a year back and what I did was pull in the UK XML data daily (the feed was updated daily) and database that mother. I'm not sure how often Expedia updates its hotel prices, but you could maybe poll every couple of hours instead.

We also had a free search for anywhere in the world, so we'd pop up a spinner and grab the XML. Then we'd cache that result set in the DB with a rudimentary script and store it for the next 24 hours. That would then be served to anywhere searching for the same thing in the meantime.

Overall, it sped things up dramatically.

Don't do that as the page is rendered. Have an offline job pull in those feeds and store them in a database, then have the frontend pull from the database. Even if the feeds are highly specific for the page loaded - In that case, model the database as a queue where the frontend requests the feed and the offline jobs pulls them. This will allow you to parallelize the feeds, which is otherwise neigh impossible to do in php.

Maybe put the data feeds into frames on the page so the main page loads quickly and the frame data can load as it comes in. Then use a simple progress bar for each frame.

Progress bars are not an optimal solution to this. Many websites shoot for 3 seconds or less load time because a lot of people will not want to wait.

Another tactic might be to prefetch the data somehow as soon as the first page loads into a visitors browser and feed the data to the user before he asks for it. If he does not ask for it you will burn unneeded bandwidth preloading it, but something needs to be done to get load times down from 10 seconds to avoid losing users.

Do you display someting like 10 entries per XML feed ? If you only display one, just get the first one, then in the background, get the remaining !

Probably won't work. Fetching a feed is usually heavy because of the connection overhead - not the actual transfer or parsing.

Reply on news.ycombinator.com