necro

So I may be a 30 something professional who's resources vs cost point is that I don't need to look for crappy versions of files. My time is much more valuable, but I believe that everyone is not like me. My contention is that if you took a sample of the younger population, or perhaps less affluent population you would find that their pain threshold is at a different time/cost level.

Don't kill the messenger, but here is a real scenario.

My "friend" used to use megaupload, filesonic, and other services to watch all his tv content every night, and even paying for a megaupload account (never putting any money back to the creators of content). It was really convenient and the price was right. Now without it, even in the last few days, the pain point has shifted enough that he started buying the seasons of shows he is interested in on itunes. (giving a cut to the creators)

There was a point even a year ago that he could get the latest film on piratebay so easily that there was no point going to the theatre. Now it seems a little harder to find the releases, or at least more inconvenient, so my friend has gone to the movies a lot more.

Now I personally get the whole internet freedom side of things, but at the same time I have seen my friend screw the content creators out of their cut because some middle man made it a lot more convenient to get the content.

I get the idea that some people have used these services for personal content, but if the business model of a service is _primarily_ around the sharing of copyrighted content then something should be done. And especially if the service pays the uploader based on how popular the stolen content they upload is.

At least in my friends case, he spent $100 on itunes in the last day, _only_ because megaupload was not there, and there was enough pain to look for another source.

If there are millions of people out there like my friend, that is a lot more revenue for the creators/studios.

And no, I'm not a troll from the film industry. This is just a real observation.

We use google for our domain email. In order to handle uploads via email on this same domain we simply create a new account and send emails to this account. You then can have a hook (imap push) in google imap to detect when an email is received per account/box and process is right away. Or you can just run a cron every minute the check if new emails have been received. We then fire a script that pulls the email via imap, and then processes the emails/images into json format for use.

This allows us to keep everything on google and we do not need move our MX somewhere else. Also we keep our "upload" email on the same domain.

Pretty simple solution with no additional costs.

That is the scary part isn't it. I'm sure only ruby users exist and that's awesome, but I would wager that most ruby users in the web field are rails users, and many blur where one starts and the other begins.

I think we're getting at the same thing. The time example is used as a comparative generalization that there is a need to focus and understand the fundamentals. The point being that given some period of time, if an average individual chooses to learn 10 languages at once, they probably wont master the fundamentals, compared to mastering those in one language. Of course there is the side case that comparing different languages and their patterns will actually be even more valuable in overall understanding, I would have to side that one first needs a deep enough understanding for that to be an advantage.

I was at conference recently talking to a group of "developers" and one finally asked, so what kind of language requirements do you have of the people you hire. I answered that I don't really have any requirements of the specific language. They all looked at each other and nodded, oh, so you must not be too advanced as you are not looking for people with experience in new advanced languages. I was a little surprised because I was thinking the EXACT opposite. I work with _developers_. I don't really give a shit what syntax they prefer of are used to. If you can create your own framework error handling, message queues, custom db libs, etc then that is useful. A real dev can pick up a new language and syntax in a week, so just because you can write hello world in 10 languages or use some rails ORM to output some stuff does not make you a developer.

It's all starting to take me back to my youth watching Robotech and remembering the Zentradi, an advanced race with awesome technology and tools that were developed for them generations ago, but none of the understood how any of it worked or how to fix it, they could only use it. If it broke, they just start over with a new copy instead of debugging it. I hope that the our new generation wont end up like that.

I recently interviewed a recent CS grad that had no idea of the correlation of array of characters and pointers to those characters. You know that it's all just addresses to memory, right?

Anyway, of course there are those who truly dissect and get all the languages and their benefits but are few and between.

Give me a dev that "gets it" in one language any day, instead of someone who thinks remembering the function call or some syntax in 10 languages is of value.

If I hire someone here is how I rank them... 1. 5 years experience in C 2. 5 years experience in PHP 3. 5 years experience in Rails 4. 5 years experience in Rails, Python, erlang, java, php, c, other languages dujour.

Ok, there are people that are just picking up PHP and don't know squat, but at least they know they are beginners and are teachable.

You miss the point. You would have already jumped off a cliff. This is one thing you do before. Of course try all the other things. But if you _are_ on the ledge, consider this before. If you think about it, is the consideration of this is similar to "who will find your body, how will they feel". It's one additional stop gap to prevent the worse outcome.

If someone is at that stage where you are contemplating this, here is a tip....just do it. By that I mean commit suicide of your _current_ life, not of life. Most of what weights someone down are the fears, struggles, embarrassment, your peer groups, jobs, etc. If you can't find a way through it, just leave it. Give all your stuff away, give away your money, give up your computer, leave your friends and family, pack a backback, travel to Africa, resign your life to helping others. Think of it as you ended your life and now you're just a servant to others. You don't have any aspirations of success, any ladders to climb, just live to help others.

Go without any plans, any future, just go and see what happens. You'll be free. It may be dangerous, you might not make it, but if your other choice was death the day before, then everything now is just gravy anyways. Don't worry about it.

This may sound silly and simple and extreme, but if you _are_ at that stage then this should be your stop gap. From now on when you're in that dark spot just think about when you're going to pull the trigger on this, not on yourself.

Of course there are lots of things you can do before it comes to this, but if you're past that, just leave your current life totally.

The temperature slider at the top which when changed dynamically shows the gas/liquid/solid state of each element is a nice touch.

Speaking of trusting google... there was a situation at a company I worked at where the management strongly was against transferring the company mail system to google apps/mail because of the concern of google may read their email. They wanted to keep their internal mail infrastructure for privacy reasons of the company.

The cto at the time convinced them that there is really no point of keeping things inhouse since the majority of the management, for convenience, forwarded all their company email to their personal gmail account anyways. Funny how security is not about what you think but what you do.

Im a bit conservative when it comes to hardware like raid controllers. My choice was 3ware. They are by no means the fastest, in fact the performance sucks compared to others. I went to a company that builds storage systems, but will build any kind you want, not locked into any controller. I trusted them when they recommended that by their experience is returned/fails the least. Of course everything fails, so it's just a matter of time. We have tripple redundant storage for file backup... active, 5 minute backup that is ready to be swapped in at one click, and long term. If something goes wrong with the active set or slows down, we just flip a switch and all our app servers use the new system that at most is 5 minutes behind. Old system gets shot in the head, and can be diagnosed off line. Shoot first ask questions later.

I have to agree. I put our stuff in a colo 2 years ago and never looked back. Pretty much all servers come with some kind of remote console interface IPMI, and that's not terminal redirection, thats actually a totally self contained microprocessor and ether port that you can run on a separate subnet and control your server even if it's off. I updated the bios, reinstalled OS's, all via IPMI which is part of the motherboards. Add to that power strips that you can also control remotely and you're all set. Our servers are in the Bay Area, I'm in Canada. I have NEVER had to drive/fly to fix anything. Never even had to use remote hands for anything. Sure some drives died, but standby drives are in place.

The costs are dirt cheap these days. You can get a full rack, power and a gigabit feed for about $800 in many colos in texas. We opted for equinix in san jose, which is all fancy with work areas, meeting rooms, etc when you are there, but the funny part is, we're never there!

I do like the virtualization for some maintenance/flexibility so we have a few servers that are hosts and we run our own private cloud where we get to decide where/what runs. Database servers on bare metal with ssd drives in other cases. Best of both worlds.

It's so cheap you get a second colo in a different part of the country to house a second copy of your backups, and some redundant systems just in case something really bad happens.

Oh yeah and don't get me started on storage. We store about 100TB of data. How much is that on S3 per month? $12,000/month! A fancy enterprise storage system pays for itself every couple of month of s3 fees.

The quality is relative. As HA has gotten more popular the level of experience of the average user has gone down, and they now dicate more what "quality" is. In the past you had a higher percentage of core "hackers", and they were posting things that they find of quality, but now you have much more varied submitters and voters. More people equals lower lowest common denominator.

To fix you need segregation. A ultra code/tech area, a business/start up area, and a fanboy/fluff area. You basically want to give high signal to noise ratio for the different groups of people. For example for me, I would just visit the code/tech area and not have to deal with all the noise of the other sections.

Perhaps this then...

if ($network=='at&t') { $initcwd=1; } else { $initcwd=10; }

I think a reasonable increase in initcwd for mobile makes sense. I imagine most mobile pages are smaller too. The way I look at it, I want to increase initcwd to fit the page size in it.

Great writeup. Have been hacking initcwd to way back when it was just hardcoded in the kernel. Other parameters not mentioned that control congestion:

1. net.ipv4.tcp_slow_start_after_idle By default this is set to 1, which means, in a longer keep-alive connection, your connection will revert back to slow start (whatever you ended up setting it as per article) after "idle" (idle is 2 seconds last time i checked). Of course in a keep-alive connection you may have opened up your window from slow start to max, and you probably dont want it do default back to initcwd on subsequent requests. Especially applies on static file/cdn type servers where you probably are running longer keep-alives. You can just set this in sysctl.

2. the last time i checked (years ago) if you run conntrack and turn off tcp_no_metrics_save, you get the benefit of the window opening up to the last remembered window size from your previous connection.

The concepts on increasing initcwd are particularly advantageous in the mobile arena where the rtt is usually much higher.

I worked at one of the major GPS manufacturers as a hardware/firmware designer and we stopped tracking above a certain altitude and over a certain velocity due to government policy. I remember one time there was a bug report from a customer complaining that our GPS would not work after mach 2 or some extreme speed. This ended up being a "qualified" foreign government client, so a new firmware version was sent which removed the restriction.

Perhaps the most obvious difference is that with siege you can specify a list of URLs that you want accessed ( some could cause reads and others writes) and in so more realistically load your website as a whole. Testing a single url with ab gives you different kind of information that is more artificial compared to the overall function of a site, but sometimes it may be the data that you are looking for. You can achieve this also with siege. Older versions of siege seem to keepalive connections not working...from the man "# TRIPLE CAUTION: don't use keep-alives until further notice" and if you find that you need to test with keep-alives you are back to ab. NOTE: the trunk of siege seems to have keep-alives functioning. These are at least a couple of differences.

I do use siege, but it seems that the keepalive support in siege is broken. At least their man says - DONT USE IT. So many people use ab, for whatever reasons, and if they do, they should be aware of this quirk.

You serve all you content via HTTPS? Including most static files/images, even though you have some non HTTPS objects which are not encrypted (and the browser notifies users of this).

Perhaps there is some business reason you have for this but it seems like a lot of wasted cycles.

That is a very small amount of traffic to experience issues with. If you want to inform about how much load was put on your systems, you should show pageviews and also show/look at stats on the static content (if you are using one server to serve it all).

Here is our 10/s crawl stats too. Thought I'd share to get a contrast. Though about half the page crawls per day. Note the page load times :) Oops, recent regression due to building an internal cloud. So you're right, not out of the ordinary.

http://www.pinkbike.com/photo/6365090/

Based on that interface I would predict...

middle number: for the graphical choices as it's common/easier to get a valid first click in the middle of the slider.

high number: for the key entry as if most people are right handed it's easier to get to the high numbers instead of going across the keyboard.

all that is predicated on the fact that people are lazy instead of random when using this.

That is a possibility for sure. Since pb is a younger audience my gut feel is that there are a lot of users who are simply clueless about what browser they are using and just don't know better. Whether they do anything about is another story.

So IE6 still accounts for about 1 million monthly page views on our site...about 1.5% of the traffic. Yesterday we took the plunge and decided to display an eyesore of a box on every page on our site if you surf to it with ie6.

Example of message: http://www.pinkbike.com/photo/6305769/

Imagine if many sites did this to spur the upgrade process. I'm really curious to see the results of this...will traffic plunge by 1.5% or will the usage of ie6 switch to other browsers. Note: no ie9 upgrade choice.

Yup. I actually wrote a filter to our framework when it was built years ago, but as new people help out on dev they don't always use the framework. My bad and I do appreciate the smack. I humbly bow to you with egg on my face.

It's a good point, and in retrospect I may not have used the username.example.com pattern, but this was a choice made may years ago. Users seem to like this, and unfortunately these recent changes in browsers make this pattern less desirable. Some other sites that do this subdomain style are the likes of tumblr.

The thing to note is that implementation of browser prefetching is further putting DNS requests out of the control of the publishers. 10 billion queries was something like 5000 dns req/s and that is considerable resources of some dns service. I wish dns was not per query costs, but it would seem that is the metric that resources would be based on. The point of the article is to let people know that if you fall into this dns pattern, it maybe a result of the recent prefetching done by browsers, and you may have a way to resolve the issue.

Difference of the art unzipped maybe 500 bytes but the difference gzipped is only 150 bytes.

Also this argument of comparing home internet bandwidth overage is only valid if you are serving a site like this from your home :) If you think you're going to get impacted by this overage because of your surfing habits, you'd have to surf...

1GB/150= 6 666 666 pages in the month

since there are 2 592 000 seconds in a month, you're looking at 2.5 page loads per second. You're probably not there.

Also the main page load is 1.5M (including all the resources) so the 150 bytes is even less of a burden.

I think you'd be better off looking at optimizing/saving money somewhere else ;)

I don't see much benefit to support HTML5 in webm or theora. One of the benefits of html5 over flash was the prospect of better resource handling and smooth play, but now as systems get faster, and flash better, there is less and less reason to go down this path. h264 is smooth enough in flash now, and it's about to get even better in the new flash release.

There are no real hardware/product reasons for websites to support webm right now.

h264 on the other hand gets the benefit of working with all the iphone and more recently apple tv. Promised new update this year will even allow HTML5 video to be streamed to apple TV directly from the browser of iphones, ipads, and i imagine safari. This is one of these technologies that will really increase the use of h264.

I run the larges cycling site and we handle 100s of niche video uploads per day so I follow this closely, and as much as I'd like to jump on webm, I'm going to definitely hold of. Currently we convert videos to 3 formats to try handle all cases, and having to now multiply that by 2 with another codec is a lot of extra resources.

2.5 mbit h264 web HD/appletv h264 1 mbit h264 for web SD/iphone 300 kbit mobile

- webm makes no sense in the short term. - you need apple support to make webm happen faster

Think about all the extra resources, time, effort that you are asking from companies in the world to support the 2 formats. If you want to be efficient with society, keep the support of h264 while webm development happens, transition once all the big players support the new format. Alternatively take all the money/time/efforts and get apple to transition. If apple does not jump on the wagon, it's going to be years and years of wasted resources in society.

There was a large discussion earlier about the subject. I posted detailed comments in that thread so I won't repost but just link. http://news.ycombinator.com/item?id=1143317

Most of these are micro optimizations that matter most on a larger site. Most large site do not use Apache for static content. I agree, nginx would be a better first choice for a module.

One thing that may not be obvious about minimization is that it does not yield much size reduction when coupled with gzip. You may see 30% decrease in size with minimization on the plain file, but when you gzip both the plain and minimized you may only see a difference for a couple of hundred bytes. If you are worried about speed, you should be using gzip for static content.