Introducing U-SQL, a language for big data processing

Product looks good.. apart from the most important point from a dev point of view - the syntax!

They created a new query language but didn't change the syntax at all from SQL?? Why does every database product seem to think SELECT FROM is the be all end all? Ok they bolted on C# Linq, and you can reference other assemblies, but these are OLD ideas, and the combination/love-child of these ideas looks horrible.

For example writing SELECT before FROM is a massive productivity problem - mainly because the computer doesn't know what you are doing until you write the FROM table part, before it can finally auto-suggest column names for you.

Don't write SELECT col1, col2 FROM {table}, but how about FROM {table} Select col1, col2, etc.

Second problem with the syntax is it isn't modular: combine 4 tables together and get data from each and you have something like this:

  SELECT a.col1, a.col2, b.col5, b.col1, a.col3, c.col5
  FROM a 
  JOIN b on etc
  JOIN c on etc

What if I want to remove the join to table B, I have to modify the syntax quite extensively. (First comment out parts of the select, then get rid of the from - but this breaks the 'join c', so I need to repeat this process again..).

What if each from-select was it's own piece of code - revolutionary I know.

  FROM a
  SELECT col1, col2, col3
  JOIN
      (FROM b 
      SELECT [3]col5, col1)
      ON etc
      JOIN
          (FROM c
          SELECT col5)
          ON etc

Indenting is used to quickly show which table joined to which. If I want to just run table A, then I highlight the top 2 lines of code and run. Or for table B the 4th and 5th etc, you get the idea. The [3] syntax is used to tell SQL not to append the columns to table A, but insert starting from column 3.

This is what I thought of in 10 minutes, I'm sure there's some holes but already I like it better than the crappy SQL syntax I have to read every day at work :(

So, it's a glorified awk over .csv files?

I think that we as Open Source community can do better than that. We just have to make a product out of sort, join, awk, grep - not meaning making a wrapper over them but rather taking inspiration.

I did a great amount of medium-to-big data crunching in command line, and it's usually okay. There are still some pain points:

- Selecting by list of ids specified in one of columns.

- Joining two files together by field (you want them to be sorted on it obviously)

Interesting extensions.

The nice part of the syntax is that: 1. it is clear when you are going external. 2. procedural code can be written in C# -- big win there.

But it misses the two big issues in ETL that I always run into:

    1. You have a bunch files that are 95% the same but have 2-5 columns that are different.
    2. Someone sticks the wrong type of data a field.

For those issues, you'd be better off mapping a file (or a group of files) to a virtual table so you can specify the schema up front.

Interesting to see ETL extensions to (T-)SQL. I wonder if we will see anybody else offer some? (Come on Postgres!)

Man, does this look a lot like Pig. Why not just use Pig?