http://allthingsd.com/20110204/chinas-tencent-buys-riot-game...
357 karma · joined February 25, 2011
http://allthingsd.com/20110204/chinas-tencent-buys-riot-game...
We often discuss competitor products in the interviews as well -- I consider it a strong positive signal if they can already meaningfully benchmark against competitors. But the reason I don't make that the core case is because we have so much more insight into how our product works and why it is the way it is, so it's easier to validate their instincts/thought processes against reality.
After phone screen, I have the recruiter send them a case study that asks them to use my product (a social game, this works better for consumer internet, obviously) and identify the most important features for monetization and to suggest improvements to the game.
1) Because the prompt is somewhat open ended and they generally have at least a week to work on it, how much effort they put into it is a great indicator of how much they want to work at my company, which to me is one of the most important things I'm interviewing for once basic intellectual horsepower is sufficient. Responses typically range from "typed some stuff into the Word doc" to "50 page PDF with screenshots, charts and graphs."
2) That also gives a concrete basis for discussing product ideas. In addition to the obvious questions of "why do you think we implemented this feature" etc, I also often take suggestions they have for the product, and ask why they think we HAVEN'T implemented it, particularly if it's a idea we've considered in the past.
Even if you work 9-9 three or four weekdays a week, that still leaves at least one weeknight for meeting up with someone for drinks/dinner/social activity, plus weekends. I find that's more than enough time -- I generally have drinks with my coworkers Friday after work, have a social activity with friends on Saturday, and brunch or something on Sunday, and leave Sunday evening for chores.
I feel like I have a pretty full and balanced life, but of course ymmv.
Consider a team working on a live browser/Facebook-based free to play game (social game as a service). There are many releases a week.
- Engineering typically requires art assets to begin/complete work. Ie. building models, unit models, 2D unit images, item images, etc.
- Releases don't end when they are pushed up to prod -- most releases have some kind of "go to market" plan that must be executed. For example, if a new Dragon Armor is released, we need to message all players to inform them of the new functionality and educate them on how it works, there would typically be a simultaneous tournament or contest that awards the new Armor as a prize, and possibly some other sale or promotion on a related item.
- It's a time for product to provide updates on where different specs are and on key game metrics.
- Customer Support/Community Management also provide updates on any emerging issues in the game, which improves everyone's situational awareness.
As a result, they are typically the closest to really knowing what has worked and what hasn't and why, which makes them uniquely positioned to decide what to do next/prioritize upcoming features, make tradeoffs. It doesn't mean the decision making is perfect, but I like to think that if you ask our engineers and designers whether our PMs are adding value, they would agree. Our 12 month engineer retention and internal survey data indicate that that's at least directionally accurate.
For people with PM experience, I'll ask them about the things they considered and documented in their last spec. If they think a spec is a list of bullet points, that's instant fail. If they considered all user flows in and out of the features, edge cases, user education, analytics/success metrics, release plan and time vs cost tradeoffs that's a win.
For date, it typically does not truncate seconds, so not sure what's happening there.
Setup: http://sphotos-b.xx.fbcdn.net/hphotos-prn1/31539_10101310184... Setup 2: http://sphotos-b.xx.fbcdn.net/hphotos-snc6/184455_1010131018...
1) We generally have pretty tight revenue targets, so one thing I and my teams do now is forecast out expected revenue on a daily basis for the next 30 days, as revenue is highly dependent on a stream of new content, features and what we call live ops.
Live ops are basically a combination of promotions, contests, events and tournaments. We have decent tools, so we can generally spin up a live op in a matter of minutes to a couple hours without doing an engineering push. We also have content we can release with a few hours of work, so if today is not going to go well, it gives us enough time to prep something for tomorrow.
It's highly useful to know if a Live Op or new feature/content release is "working" or not. With the real time dashboard and experience, I can know with high confidence whether it's working within 30 minutes of release, which gives me more than enough time to spin something else up so that we don't lose the day.
2) On many days, we actually do multiple releases or live ops in a single game -- the pace of releases is fast even compared to other web companies. As unscientific as this is, it's often easier for me to evaluate/get a feel for the impact of a release/live op based on how much it moves the projection in the first hour after release, and before the next thing comes along.
Obviously with stuff like that, you validate after the fact against more full data, but I've found after looking at this lot for over a year, I have a strong "feel" for what's working and what's not.
3) Related to 1) and 2), but I've found that using the tool gives me a much tighter feedback loop. When I make a feature release or live op and I'm sitting on the dashboard watching the release move or not move the projection, I have a much tighter feel on the impact. Most of my peers who don't have something like this often are not as rigorous in understanding and evaluating the different levers they have to impact revenue.
1) You have to provide context for everything. Current real time revenue is presented right next to the 14 day average revenue up to that point in time, and also how many standard deviations the delta between the two is. Ie: Current revenue is $100 at 10am, vs. 14 day average of $90, which is 0.2 standard deviations of revenue at that time.
2) Hourly revenue is presented the same way, right next to the 14 day average revenue for that hour and the SD delta.
3) Look at it a lot. I've been looking at this sheet regularly for over a year now, and I have a really good feel/instinct for what a normal revenue swing is, and an even better feel for the impact of different features/content/events/promotions on our revenue.
4) This approach also works better when the impact of your releases is high. A big release typically spikes revenue 2-3 SD above baseline, and causes an immediate and highly visible effect. So while I'm not strictly testing for statistical significance, it's one of those things where it's pretty obvious.
5) It also works better if you use it in conjunction with other metrics. We validate insights/intuitions gained from looking at realtime data against weekly cohorted metrics for the last several months of cohorts.
I've done this with two lead PMs and an engineering lead so far, and it's been met with a positive reception.
1) He was referred to my startup by someone in our network, so he had at least a basic understanding of how our company worked.
2) He offered to come in as an intern just to get his foot in the door, and was extremely humble about his value to the company; basically recognized that he wouldn't that productive at first. (We brought him in as an analyst and he made because a lead PM in about 7 months).)
There is a lot to like about people with that background: extremely smart, hardworking, with hard analytical skills. If the two basic criteria from above are met (basic understanding of the business, strong desire to come in and learn), then I would strongly recommend snagging an MBA if you can.