HNHacker News
TopNewBestAskShowJobs

cachemiss

278 karma · joined April 15, 2015

submissionscomments
cachemiss··on AWS mistakes to avoid
So I'd modify these a bit. We run a very large AWS infrastructure as a engineering team (no dedicated ops).

1. Use CloudFormation only for infrastructure that largely doesn't change. Like VPC's, subnets/ internet gateways etc. Do not use it for your instances / databases etc, I can't recommend that enough, you'll get into a place where updating them is risky. We have a regional migration (like database migrations) that runs in each region we deploy to that sets up ASG, RDS etc. It allows us control over how things change. If we need to change a launch conf etc.

2. Use auto-scaling groups in your stateless front ends that don't have really bursty loads, it isn't responsive enough for really sharp spikes (though not much is). Otherwise do your own cluster management if you can (though you should probably default to autoscaling if you can't make a strong case not to use it).

3. Use different accounts for dev / qa / prod etc. Not just different regions. Force yourself to put in the correct automation to bootstrap yourself into a new account / region (we run in 5 regions in prod, and 3 in qa, and having automation is a lifesaver).

4. Don't use ip addresses for things if you can help it, just create a private hosted zone in Route53 and map it that way.

5. Use instance roles, and in dev force devs to put their credentials in a place where they get picked up by the provider chain, don't get into a place where you are copying creds everywhere, assume they'll get picked up from the environment.

6. Don't use DynamoDB (or any non-relational store) until oyu have to (even though it is great), RDS is a great service and you should stick with it as long as you can (you can make it scale a long way with the correct architecture and bumping instance sizes is easy). IMO a relational store is more flexible than others since you (at least with postgres) get transactional guarantees on DDL operations, so it makes it easier to build in correct migration logic.

6. If you are using cloudformation, use troposphere: https://github.com/cloudtools/troposphere

7. Understand what instances need internet access and which ones don't, so you can either give them public ips, or put in a NAT. Sometimes security teams get grumpy (for good reason) when you open up machines that don't need to be to the internet, even if its just outbound.

8. Set up ELB logging, and pay attention to CloudTrail.

9. We use Cloudwatch Logs, it has its warts (and its a bit expensive), but it's better than a lot of the infrastructure you see out there (we don't generally index our logs, we just need them to be able to be viewed in a browser and exported for grep). It's also easy to get started with, just make sure your date formats are correct.

10. By default, stripe yourself across AZs if possible (and its almost always possible). Don't leave it for later, take the pain up front, you'll be happy about it later.

11. Don't try and be multi-region if you can at first, just replicate your infrastructure into different regions (other than users / accounts etc.). People get hung up on being able to flip back and forth between regions, and its usually not necessary.

edit: Track everything in cloudwatch, everything.

cachemiss··on Why the Economic Fates of America’s Cities Diverged
Fair enough, I won't try and convince you, FWIW I hope you are right, it's my home city and I want it to be a great place again.
cachemiss··on Why the Economic Fates of America’s Cities Diverged
So I can speak from experience here, on both ends. I was recruited back to upstate NY after moving away for about 200k to run an engineering team, I make a bit north of that on the west coast. I turned it down without even thinking about it too much, one I would have to uproot my family, and if that company went sour, I wouldn't be able to provide for them the same way and we'd have to move again. I also couldn't take the winters anymore.

If you are recruiting top end college grads, forget it, they'll get snapped up by the bigger companies, even if you are offering them the same, it's about the name and moving out west. If you are recruiting senior engineers, none of them will move from the bigger cities, and there aren't really that many local if at all that can solve certain problems, they all move away.

cachemiss··on Why the Economic Fates of America’s Cities Diverged
It's not, the issue is that the market is so small, that even if you are paying really well, no one else is. So if things go south, you're going to have to move.
cachemiss··on Why the Economic Fates of America’s Cities Diverged
The issue is the taxes and the brain drain. It's impossible to keep people there (I'm a native of upstate NY and have recruited developers). We kept losing people to NYC and West Coast based companies before we could even offer. Anyone we did get who wa sworthwhile moved west as well (I did too).

Upstate has a lot of advantages (its a great place to raise a family), but has some massive disadvantages as well (the weather is one of them). There isn't enough tech to keep developers, because they know if your company tanks or fires them, the market isn't very good.

cachemiss··on How to Start Closing Silicon Valley’s Age Gap
I think that's pretty dismissive of people with experience. Learning the latest stacks isn't particularly hard, and it's really not that valuable (especially if it's in webdev)

It depends on what you are doing, tech is hard to learn when its in a difficult discipline that takes time to do well. It also depends if you need a lead, or not. If you need someone to sling javascript, maybe you get away with someone younger.

If you need a lead to help you build a database kernel, or build a globally distributed system that needs strict SLAs and extreme performance, you are probably going to need someone with experience. Putting someone with 2 years of experience into that role is asking for a disaster.

Most startups aren't doing anything super difficult from a technical point of view, and don't necessarily need their systems to last, so hiring younger people, paying them less, giving them equity that isn't worth anything and burning them out is a winning strategy.

cachemiss··on Why I Left the .NET Framework (2013)
Yeah, he's comparing one of the best optimized ring buffers on the JVM to actual transactions, that's one of the worst comparisons I've ever seen. I've shipped large scale systems on .NET and on the JVM, the issues you run into with .NET are that windows isn't as easy to administer IMO as linux variants, and your ecosystem isn't as strong, that's about it.

.NET doesn't have a runtime profiler that will deoptimize / recompile etc., which can cause slowdowns in certain use cases, it does have a better unsafe story, and value types, and reified generics which can make things faster for you.

← PreviousPage 2 of 2