ezmobius

nginx hands down. nginx is probably the most stable piece of software with the least problems we run here at engineyard. We've had thousands of nginx instances in production for years now and I can truly say it is one of my favorite pieces of software.

BSD, MIT or Apachev2 is my preference. I really don't like the GPL and it does tend to turn off some folks from using open source projects. I would not be too concerned about someone taking the work closed source even if you do go BSD.

Redis has a different use case then tokyo does. I think tokyo is beter for long term persistent data storage. But redis is much better as a state/data structure server.

Redis is faster then tokyo by quite a bit but is not immediately durable as in the writes to disk happen in the background at certain trigger points so it is possible to lose a little bit of data if the server crashes.

But the power of redis comes in its data types. Having LISTS and SETS as well as string values for keys means you can do O(1) push/pop/shift/unshift as well as indexing/slicing into lists. And with the SET data type you can do set intersection in the server. This allows for very cool thigs like storing a set of tags for each key and then querying for the intersection of the set of tags of multiple keys to find out the common set of tags.

I'm using this in nanite(an agent based messaging system) for the persistent storage of agent state and routing info. Using the SET data types makes this faster then keeping the state in memory in my ruby processes since can do the SEt intersection routing inside opf redis rather then iterating and comparing in ruby:

http://gist.github.com/77314

You can also use redis LISTS as a queue to distribute work between multiple processes. Since pushing and popping are atomic you can use it as a shared tuple space. Also you can use a LIST as a circular log buffer by pushing to the end of a list and then doing an LTRIM to trim the list to the max size:

redis.list_push_tail('logs', 'some log line..') redis.list_trim('logs', 0, 99)

That will keep your circular log buffer at a max of 100 items.

With tokyo you can store lists if you use lua on the server, but to push or pop from a list you have to pull the entire list down, alter it and then push it back up to the server. There is no atomic push/pop of just a single item because tokyo canot store real lists as values so you have to do marhaling of your list to/from a string.

I added consistent ring hashing and a rediscloud class to the ruby client library today. so you can now use a cluster of redis servers and have the client use ring hashing to determine the server for a given key.

Umm redis is really cool. Nice job on this antirez. I've just finished implementing the entire spec here:

http://github.com/ezmobius/redis-rb/blob/0951204e6a84ca335ac...

And I am really enjoying being able to store lists as values and pushing/popping/indexing. Looking forward to where you are going with this.

Although the current release doesn't seem to support all the documented features yet. incrby and decrby don't work yet.

Can you get in touch? I'd love to pick your brain a little about redis.

I've knocked up a quick prototype ruby client library for redis. So far it supports GET, SET, EXISTS, DELETE and KEYS. Should be able to have a fully compliant library in a few days.

http://github.com/ezmobius/redis-rb/tree/master

yeah backgroundrb sucks, who wrote that shit? (me and i wish i never did)

at the high cost of crippled one line lambdas, no thanks

nope, rails will still buffer up the whole template render before it sends it down the socket. I think he means progressive rendering directly to a socket rather then buffering it before shipping.

rack, merb, sinatra and rails

We have ey-flex coming in a few months. Flex is the multi slice clustered version of engineyard on AWS, replicated db's autoscaling app servers and load balancing, message queueing plus logging and monitoring aggregation services.

So the answer is yes you will be able to replicate a traditional EY cluster setup on AWS in a few months.

The ec2 stuff is transparent because we aren't trying to hide the underlying platform. You can still utilize the full power of AWS form our system, you just don't have to unless you really want to. Sane defaults but major flexibility is one of the goals of this platform. That being said you may be right and we may need a tiered pricing system tha includes certain amounts of bandwidth and storage for a known monthly price. Time will tell as people start top use the platform.

Also you can run multiple applications on one solo instance. Since you have 1.7 gigs of ram you can run 3-6 little apps pretty easily if you use the apache + passenger stack.

ey-solo will be generally available on the 28th of Jan.

The main difference is that ec2 and aws in general just gives you very low level building blocks. It gives you an api for 'give me a new server', 'attach an ip address', 'create a new volume', 'attach volume' etc. So you have to learn this stuff just to get your basic hardware in place. once you have that done you will still be faced with all the sysadmin tasks involved with configuring everything to mount and format these volumes and configure every service you need to run. Of course you need to know what the proper setup for a ruby app in production is and learn all the different webservers and monitoring tools.

The idea behind ey-solo(and the soon coming ey-flex) is that you don't want to have to learn all that stuff. We have been running and managing high volume ruby apps in production for 2.5 years now and we have taken our whole battle tested stack and are now offering it as SaaS.

So the goal is to save you time and hassle. We will maintain the whole stack with constant feature and security updates as well as bug fixes and all you need to do is write your app and leave the deployment and scaling hassles to us.

AWS is only the beginning, this platform will soon be available on other clouds as well as be downloadable so it can be run on any hosting provider if you run our linux and install our agent.

This means that eventually you will be able to host anywhere and still use our automated stack to manage the servers.

news.yc always adds that leading period to my domain when it gets posted here for some reason.

Yes this was mostly support staff. The main reason for this is that we have become much more efficient and automated so we did not need as many support staff anymore to support our systems.

I've played with this and I like the idea a lot. but I found that BDB sucks still and gets corrupted too easily.

So Instead o've been using tokyo tyrant. It supports memcached protocol as well as HTTP and native client protocols.

http://tokyocabinet.sourceforge.net/tyrantdoc/

Much better then memcachedb and serves the same purpose.

ruby 1.9 does not run rails apps 3-5x faster. More like maybe 10-20% and slower on some things.

Supermicro servers have served me very well. All of our clusters at engineyard.com are built on supermicro 1u's with 32gig ram and 8cores. Highly recommended.

Drb is nice for very small low traffic things. But it is very network chatty and is not really viable for anything that needs to scale or deliver high throughput.

Use this nginx module: http://www.grid.net.ru/nginx/upload.en.html

It is highly scalable and spools the uploads to disk and does the mime parsing in efficient nginx C code> then once it finishes it just passes some params to your backend processes with the location of the file on disk and you can process it however you want.

and it appears that two out of their 3 availability zones have to go down in order for anyone to collect on sla. This effectively makes the sla meaningless afaict.

thanks!

Working on the following open source projects: Merb Rubinius Nanite Vertebra

Also working on some top secret engine yard projects that I will be able to talk about soon.

thats a killer domain name ;)

nanite is for individual application where you know every part of the system is allowed to talk to one another. Vertebra is for wide area networking and assumes hostility so does a lot more security isolation.

Vertebra is for controlling clusters and clouds at large while nanite is for building scalable backends for individual apps.

The file streaming works by chunking up a file and broadcasting the chunks to an exchange. Any agents subscribed to the exchange will get a copy of the file. There is a small state machine for re-assembling files and each set of file packets is tagged with a token for easy re-assembly even on a multiplexed channel where multiple files are streaming at once.

I was definitely inspired by the erlang work I've been doing lately. Took a lot of those principles and applied them back to a ruby mentality.

sorry about the slow download speeds. We will get it on a faster mirror later today and I will repost the link.

Yes there is a big advantage to keeping your memory consumption low with ruby apps. Ruby's garbage collector is not the best and has to walk all the objects in the process when it GC's. The more memory your process uses the longer and more often GC will happen. This will degrade performance the more memory you use. In fact I've seen really leaky apps spend most of their wall clock time in the garbage collector.

Sure, throwing more ec2 instances at the problem is one solution, but if you care about your apps performance you will try to optimize for smaller memory footprint.