Hacker News Reader

Comment by gavinray | Hacker News Reader

This is pretty interesting.

Are you familiar with the "Morel" dialect of SML?

It's written by Julian Hyde, who has authored several RDBMS and is the lead maintainer of the Apache Calcite relational algebra framework. Your tool reminds me a bit of the work in Morel, where LINQ-like operators have been integrated into the core of the language:

https://github.com/julianhyde/morel

View on HN

I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time.

  let path = "/path/to/files"
  let fname = "quotes.csv"

  let quotes = load(path + "/" + fname)

Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation.

As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider.

Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing.

Empirical is the only system I know of that can infer types while staying static.

> Empirical is the only system I know of that can infer types while staying static.

F#'s Type Providers can do that too.

https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...

F# is how I learned about type providers:

http://fsprojects.github.io/FSharp.Data/library/CsvProvider....

But look at that syntax:

  CsvProvider<"file.csv">.Load("file.csv")

The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string.

F# has Literals that mitigate some of this, but my understanding is that they are pretty limited:

https://github.com/fsharp/fslang-suggestions/issues/539

On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc.

  func filename(n):
    let value = String(n + 1)
    return "/path/to/" + value + ".csv"
  end

  load(filename(7))

I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.

Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.

Reply on news.ycombinator.com