1,483 karma · joined April 15, 2009
Are your customers actually writing SQL for executing against a database or is your app generating SQL? If instead the app's purpose is to provide a "power interface" to access data (read only?) and generate SQL, there are better choices. Use https query parameters to map to parameterized queries (https://blog.codinghorror.com/give-me-parameterized-sql-or-g...) which are a much better design choice.
Given that online programming competitions tend to fall into distinct classes (dynamic programming, algorithmic challenges, graph problems, string problems), this seems maybe more "solvable" with a non-AI implementation?
Imagine you have a framework that can spit out sub-pieces of a solution that worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected components in each graph | spit answer of graph with lowest number of SCC).
Then you need to grind out some type of expert system to replicate the competitive programmer who currently chunks together those framework pieces.
Of course, this probably doesn't work at all for something like kaggle or stockfighter.io, where parsing the instructions and observing a dynamic system are key parts of the hacking. I am more thinking of SPOJ or similar . . .
My institution was heavily recruited by big corps, government labs, and east coast companies.
The "best" students, by GPA, were in high-demand for all of the above. Many were heavily recruited into management tracks for non-IT companies. A large number of government institutions and defense contractors were also eager to land new grads from our school. The "best" students, by hacking skills were (maybe stereo-typically for hackers) less interested in classes that didn't involve slinging code, but also all landed programming gigs. Less committed students, from either metric, seemed to still be getting jobs but I can't generalize as to the job type.
I think it is fair to say that my undergrad course-work was not as demanding as (guessing a bit here) Stanford, MIT, or CMU. But my GPA and GRE scores landed me multiple job and graduate school offers.
One aspect of hiring from (or at least my) HBCUs is that there is very strong network effect - alumni come back to the school and recruit interns and fulltime positions for their companies, help prep students for the process, and students looked to those alumni as trusted sources.
If Silicon Valley really wants to hire from HBCUs, that is the path I would recommend. Hire a few alums from the HBCUs and make recruiting and grooming candidates a priority for those alums.
"Given that ∼20% of subjects fail to improve aerobic capacity with intense endurance training, while ∼30% do not enhance their insulin sensitivity (12), it is clear that we must develop genotype-tailored lifestyle interventions."[1]
[1] Using molecular classification to predict gains in maximal aerobic capacity following endurance exercise training in humans
James A. Timmons, Steen Knudsen, Tuomo Rankinen, Lauren G. Koch, Mark Sarzynski, Thomas Jensen, Pernille Keller, Camilla Scheele, Niels B. J. Vollaard, Søren Nielsen, Thorbjörn Åkerström, Ormond A. MacDougald, Eva Jansson, Paul L. Greenhaff, Mark A. Tarnopolsky, Luc J. C. van Loon, Bente K. Pedersen, Carl Johan Sundberg, Claes Wahlestedt, Steven L. Britton, Claude Bouchard Journal of Applied Physiology Published 1 June 2010 Vol. 108 no. 6, 1487-1496 DOI: 10.1152/japplphysiol.01295.2009
RTP historically has a bunch of big IT in it - IBM, SAS, and Nortel were the big three, but Nortel melted away, not sure what IBM is doing on the hiring front there, SAS is still chugging along. Currently, I think Fidelity, Deutsche Bank, and MetLife are out in RTP. Fidelity seems to be recruiting this year consistently and the other two as well. These big places all pay pretty normal salary ranges. I'm out of the corporate dev world, but anecdotal info would seem to peg salaries in the $90K to $110K at those places for experienced engineers. Friends and I have heard of $125K salaries plus bonus, but none of us can actually point to a verified example of that base salary as a direct hire. I think even these big enterprise corps want to hire a little lower level and promote up. For contracting at the financial corps, all bets are off. In 2008, I knew contractors in test groups doing $100-$150 an hour, which was insane given the skill sets I observed on some of those contractors (minimal). I wouldn't count on that today. I tend to put the Triangle (shorthand for Raleigh, Durham, CH, and RTP) into a "third tier" of tech cities, with SFB/NYC tier 1 and Chicago/Austin as tier 2.
I spend more time kind of looking at Durham for startups and small consulting firms. Adzerk, iContact, and back out in RTP, MaxPoint are startups at various points of existence. I have no idea what pay is like at startups. You might want to check out http://bigtop.it/jobs - they have both a job listing and host semi-regular mixer/job fair events. I went to one and didn't really pursue anything. It may have been a little entry-level leaning.
Oh, don't forget the local university tech scene. NC State is there in Raleigh (and RedHat is on Centennial Campus), Duke is Durham, and UNC-CH in Chapel Hill. If you are interested in being a staff coder in academia, there are opportunities in those places. Duke and UNC-CH also have big hospital orgs that hire buckets of IT staff - if you can shoehorn the word EPIC onto your resume, you can land something in one of those two quickly.
Weather is lovely, but hotter than the Bay area. Humidity and 99F is . . . nasty. And I'm a native NC'er. Good mix of sun and seasons during the year. Excellent beaches (and the Outer Banks) two to four hours driving. Same drive time West puts you at 4000 to 5000ft in the Appalachian mountains of NC.
Cost of living is ok. Housing cost varies too much to generalize. Cheaper, sometimes much, outside the cities. Good food given the college town status in all three. Reasonably active tech meetup scene.
If I was moving to the area and didn't have a partner or family, I would probably lean towards downtown Durham. I feel like it might be a little more funky and interesting than the other places. If I had a family or partner who wanted to start one soon, I might shop around for a school district I would be happy with and try to land there.
Oh, last thing - the Triangle has good political diversity, but I think once you leave it, the state is generally conservative with pockets with more progressive presence.
Don't you think it is fair to say that high throughput data (whole genome sequencing with variant calling) is still in a state of being evaluated to measure its effectiveness in aiding the treatment decision process but that early results seems to lean towards it becoming part of the standard diagnostic approach?
Genomic sequencing and patient outcomes is a thornier question. My non-practitioner take is that it is too early to tell scientifically, but that there will probably be some benefit to early identification of specific cancer types and choosing treatment. But I think many people would have made a similar statement about mammography and early detection, and absolute mortality appears to not be reduced by adding mammography to the diagnostic procedures, right?
The research value of genomic sequencing seems high enough to make it worthwhile. At least, when I sit in on molecular tumor board reviews (the oncologists at a table looking at called variant results for a specific patient), I hear them commenting about possibly new and unknown variants being of research value.
I am really looking forward to your reply - Internet message boards in general have to be almost the worst way to discuss medicine, but having participation from researchers and practioners like you is tremendously illuminating!
You could also skim
http://www.nature.com/nature/journal/v406/n6797/full/406747a...
for more information about genomic classification of breast cancer.
From a treatment prespective, I would say that just glancing at http://ww5.komen.org/BreastCancer/SubtypesofBreastCancer.htm...
would provide information on treatment decisions generally made by finding appropriate subtype classifications.
I think that it is pretty clear that genomic sequencing of patient normal and tumor tissue to find mutations is going to be standard-of-care sooner rather than later, but it is fair to point out that genomic sequencing is not currently standard-of-care. However, I know of studies currently underway that look at variant calls and the possibility of taking action on those calls in ways the involve specifically adding those results back into the patient medical record.
I am struggling a bit with how to phrase this, but I don't think you can argue against (1) different subtypes of breast cancer are separate diseases and can be classified by genomic sequencing and (2) treatments for these separate diseases are different and have different efficacies.
http://www.nature.com/nature/journal/v490/n7418/full/nature1...
See http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6026a6.htm?s_cid=...
I vaguely remember that the effect disappears for second generation births. Don't have a citation handy for that though.
(Edit: pdf link since the above link seemed truncated - http://www.cdc.gov/nchs/data/nvsr/nvsr59/nvsr59_06.pdf )
I would have expected at least some experienced candidates to drop into that "top tier" of people. What's up with this?
The next level up seems to be problems that require composing algorithms. This type of problem takes creativity, some insight into how to combine, for example, graph algorithms with dynamic programming to come up with "optimal" solutions. These are quite hard for me, especially dynamic programming - it seems to be a favored component in many problems.
Maybe consider downloading 5 or 10 years worth of ACM programming contest problems. Take your time, work through each one, and add to a notebook (or even something like Anki) for regular review.
There is a book on competitive programming at https://sites.google.com/site/stevenhalim/ The first edition is freely available there as a pdf.
Stepping away from your question, you mention that your puzzle solving speed is holding you back from getting your desired job. This is a tough situation - it sounds like the job you want is at a company where these puzzle interviews are prized. If it is possible, maybe you could try to create some working sample code that addresses the job requirements. A portfolio you could put forward and talk about with interviewers instead?
I sometimes try to think from the perspective of how would I setup a course to teach someone to be good at some specific subject. Occasionally I even manage to do that and turn around and make myself the student in that course. You've got a PhD already, so determination isn't a problem for you. Keep grinding!
I almost decided to go with a Thinkpad X230 - I have used a X200 running Xubuntu for almost 6 years happily. The X200 just started to fall apart and I was bumping up against the 4GB memory in it frequently enough in the last year to make a change. The X230 would have been way cheaper and supported the 16GB of memory I was after. But work was paying, I try to maximize laptop ROI by not upgrading every year, and the screen on the retina is quite lovely. The trackpad, not as much. I miss the thinkpad red nub.
But, debian Jessie installed very easily and so far, I really like the experience. I'm using XMonad, with very little customization, and in general it seems to be a solid machine. Check back with me in four years. If I'm still using this machine, the switch will have been worth it.
I'm quite sad about how Thinkpads seem to be transitioning from serious, hacker-friendly tools to somewhat generic consumer-grade gear.
Its interesting that you focus hard on "no SQL. So no SQL parsing. No query plan preparation, . . . ". Have you found SQL parsing and query plan prep to be actual bottlenecks in PostGres? This would be surprising to me.
Personally, I have found that occasionally doing simple things like memory mapping multi-gigabyte pre-sorted data and using simple binary search from within Java (well, Clojure to be truthful) can be quite performant. But that case involved static data and well-known search patterns. I have also flirted with column-stores for analytical workloads and they are pretty awesome for that.
There is such an explosion of options available these days in both relational and the NoSQL family. As much as I like hacking, I do try hard to ask myself "why can't one of these solutions do this hard work for me?"
I suspect a 10x improvement will elude you (no such thing as a free lunch), but I geek out hard reading what other people do in these cases, so if you have a business case (or personal side-interest) to tackle your own data store, have at it. Post us what you find sometime. Good luck.
I've found a number of people who point at the Github api as a good example of a RESTful api that includes hypermedia links for clients to follow rather than constructing URI requests manually. It looks good to me.
Just didn't ever know that entrepreneurship was an actual option for people. Coming from a risk-averse and somewhat fatalistic family/peer group wrt work/career control hasn't helped.
One of the best parts of HN is actually reading contributions from people like you and Patrick that break down what are frankly self-limiting thought patterns.
Hackers should pay attention to:
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...
I have also found collection+json and hypermedia application language (HAL) to be useful.
Plus, I have spent a bunch of time reading the Restful Web APIs book from O'Reilly. Useful for me, as a relative newbie, in providing some logical foundation to start from . . .
And this is coming from someone who remembers reading stuff from you on Joel's Business of Software forums, so it's not like I haven't seen the full path from BCC to consulting to AppointmentReminder . . .
Personally, I fall into the trap of thinking, oh, I couldn't do that unless I master <insert obscure tech at the Knuth level here>. Which is clearly just not the differentiating skill here . . .
The talent shortage has a strong geographic component. In SV or NYC, job search sites (a poor way to gauge actual demand, but whatever) seem to list 1,000s of available Java/C# gigs, with substantial numbers of listing that include javascript. But I would probably lump those three languages together as placeholder flags to indicate "standard enterprise developer" jobs. Most of these job listing seem to fall on the entry-level, with a steep fall-off in mid-level offerings, and a few (mythical?) mega-$$ listings. These types of jobs feel more "credential sensitive" to me - if your resume (not CV) doesn't say Comp Sci or maybe MIS, you probably get binned immediately.
Leave those regions, job listings seem to drop. If we can trust these listings to reflect some level of region-comparable demand, we might guess that in Seattle, WA or Chicago, IL, there is about 1/2 the demand of SV and NYC. Cut it down even more in places like Austin, TX or RTP, NC, maybe 1/6 or 1/8 the demand of SV and NYC? I'm sure we could include a bunch of other cities in those tiers, but maybe those are loosely representative.
I often wonder how much job demand looks inflated on these job boards though. A quick glance at Java in NYC, NY on indeed.com shows a few big staffing companies with hundreds of listings. I usually assume there are bunch of duplicate listings from staffing companies for a single position.
I think a lot of lower-level gigs out there are filled by kind-of-passable candidates not making anything like the SV salary numbers that get thrown out on HN frequently. Among strong engineers I personally know in my region, several are frequently contacted by recruiters for jobs that are, at best, side-ways moves for them (eg doing the same gig at some different place for roughly the same money). A PhD (and maybe even a MS) might scare employers at this level?
The other part of this, especially regarding the original article, is that deep, strong statistical/machine learning/magic fairy dust hacking gigs are much less numerous that the standard enterprise dev jobs and that those gigs are even more restrictive in recruiting nature.
Meaning, if you are really recruiting for a top-notch "data science" person, you are probably(?) looking for intensive credentials or portfolios. So MS or PhD in Applied Math or Statistics with easily demonstrated programming skills or a set of completed projects cast in the "data science" space. And if you're not recruiting at that level, the "data science" label means "query some data sources to generate a tabular report, maybe with some rollups."
https://jakevdp.github.io/blog/2014/08/22/hacking-academia/
The tl;dr summary from this second article referencing the first:
a quick summary is this: scientific research in many
disciplines is becoming more and more dependent on the
careful analysis of large datasets. This analysis requires
a skill-set as broad as it is deep: scientists must be
experts not only in their own domain, but in statistics,
computing, algorithm building, and software design as
well. Many researchers are working hard to attain these
skills; the problem is that academia's reward structure is
not well-poised to reward the value of this type of work.
In short, time spent developing high-quality reusable
software tools translates to less time writing and
publishing, which under the current system translates to
little hope for academic career advancement.
the skills required to be a successful scientific
researcher are increasingly indistinguishable from the
skills required to be successful in industry
the number of PhDs granted each year far exceeds the
number of academic positions available, so it is simply
impossible for every graduate to remain in academia.
The eye-opening and amazing part is the unrelenting practice, preparation, and study time Ricky Jay (and others in the documentary) have committed to the art.
Beautiful and inspiring to see that effort from people.
I am not a fan of the mathematical challenges on Hackerrank, mainly because I haven't done much interesting math in years and if I get all ambitious on math stuff, I'm going to dive into functional analysis and stats modeling and probably not from a coding viewpoint. Algorithmic challenges are much more engaging to me.
Hackerrank has a very interesting option for hiring companies to sponsor a challenge or set of challenges. Based on what I read here, if I was a hiring company I would be doubling down on this approach (eg coding challenges as a feed for potential hires like you guys were doing as well). Heck, companies would probably pay five to six figures just for a curated list of candidates from these sites, right?
One specific gripe - sometimes I had the vague feeling that the programmatic challenges encourage non-idiomatic language usage. I did a very fun challenge that involved range-minimum-queries there and to ultimately to have runtime under the JVM cutoff and pass all the testcases, I kind of optimized away from the way Clojure normally looks like. I think my point is code that works well for a programming challenge may not look much like code you would want in production.
I can probably spout off more opinions but this is probably already more than you were looking for for someone who didn't use the site for actually finding a job.
I'd love to hear if other HN readers are playing with algorithmic trading. I occasionally think quantitative trading would be fun because of the intersection of math/stats and CS. Of course, that's an "in the abstract" statement. Depending on it for my mortgage would be . . . different.
I actually found this article from a non-technical recruiter informative and do respect this effort from the author. It's nice to read from a recruiter who has taken, at first glance, a deeper look at functional programming in the marketplace and shares some insights. There's some hyperbole in the article, but it seems to be coming from a honest perspective.
But (and I say this after having read both the Lewis book and Dark Pools), I'm not entirely sure what to think about HFT (or more broadly, electronic trading in general) and market volatility.
It kind of seems like there is going to be some popularly accepted government regulation of electronic trading that my gut tells me will be more acceptable for the "other, more brazen pickpockets" than the new, despite the fact that we think choice (b) is better for investors.
Some years later when I returned to undergrad education in CS, I took advantage of several summer internships and coops. The money I earned through those basically covered the completion of my undergrad CS degree.
Maybe you could do the same? Does this USC program support coops? I would guess that you could make a large portion of the costs through that type of work.
I will say that my first suggestion echoes many other comments on this page. Don't incur a bunch of debt for undergrad education. Go to the cheap school and work hard to do really well. Definitely try to get internships and work experience. I might go so far to suggest that you look at the institutions you are considering and see which ones have particularly well-formed, visible internship programs in place.
My CS undergrad program was incredibly focused on the transition from school to work. It made getting internship and coop opportunities much easier. My (top 25) CS grad school was almost entirely focused on turning out PhD candidates into academia and had almost no real industry network to speak of. As a matter of fact, the several job offers I had in hand when finishing my MS in CS were with companies I had interned with during my undergrad days. My grad school professor network resulted in not a single job lead.
For using the 32-bit version (from Limits):
22 Limits
Each database runs in memory and/or disk map-on-demand -- possibly partitioned. There is no limit on the size of a partitioned database but on 32-bit systems the main memory OLTP portion of a database is limited to about 1GB of raw data, i.e. 1/4 of the address space. The raw data of a main memory 64bit process should be limited to about 1/2 of available RAM.
I have attended the conj in the past and taken a bit of training in Clojure-based tools (Cascalog). These days, I am using Clojure for online programming challenges and various learning activities.
The Clojure community is like others - some people are very humble and welcoming. Some less so. But it feels like an open community in most cases and I say that as a true introvert.
My take on the idea of "top 1%" devs in Clojure is that there is a clear set of top Clojure hackers who are also invested in the craft of coding. It doesn't seem to be particularly ego-driven, but there are some incredibly hard-working devs in that group. I have observed a few, well-known names in that community invest a ton of time in reading academic papers, trying ideas out with Clojure implementations, and then quite publicly sharing the results. I think this was more what was being shared rather than chest-thumping of "oh, I'm at the Conj and you're not" perspective.
The links section at the bottom alone are a tremendous resource. I've stumbled across most of these over a period of years and if you started with just skimming over this article and the links, you would save yourself many hard lessons.