Comment by chrisaycock | Hacker News Reader

Comment by chrisaycock | Hacker News Reader

I created Empirical specifically to have statically typed Dataframes (tables).

  let trades = load("trades.csv")

  from trades select where symbol == "AAPL" and size > 1000

Empirical uses type providers and automatic compile-time function evaluation to sample the CSV file and infer a type ahead of time. The above can be run in a REPL or in a stand-alone script and the result is statically typed! (There is no "gradual" or "optional" types; it is fully static.)

Empirical has generic types, which is a syntactic sugar for templates. Here is an example of a five-minute volume-weighted average price (VWAP):

  func wavg(ws, vs) = sum(ws * vs) / sum(ws)

  from trades select vwap = wavg(size, price) by symbol, bar(timestamp, 5m)

I created Empirical for time-series analysis (I work in finance) and began with the question, "What if q/kdb+ were more like Haskell?" The result is very different from either language, but I wanted to make table actions feel dynamic even though they are fully static.

To run a script that takes a command-line argument for the location of the CSV file, just supply the type ahead of time since there isn't a compile-time path to the file.

  data Trade:
    symbol: String,
    timestamp: Timestamp,
    price: Float64,
    size: Int64
  end

  let trades = csv_load{Trade}(argv[1])

Feel free to try it out.

https://www.empirical-soft.com

This is pretty interesting.

Are you familiar with the "Morel" dialect of SML?

It's written by Julian Hyde, who has authored several RDBMS and is the lead maintainer of the Apache Calcite relational algebra framework. Your tool reminds me a bit of the work in Morel, where LINQ-like operators have been integrated into the core of the language:

https://github.com/julianhyde/morel

I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time.

  let path = "/path/to/files"
  let fname = "quotes.csv"

  let quotes = load(path + "/" + fname)

Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation.

As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider.

Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing.

Empirical is the only system I know of that can infer types while staying static.

> Empirical is the only system I know of that can infer types while staying static.

F#'s Type Providers can do that too.

https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...

F# is how I learned about type providers:

http://fsprojects.github.io/FSharp.Data/library/CsvProvider....

But look at that syntax:

  CsvProvider<"file.csv">.Load("file.csv")

The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string.

F# has Literals that mitigate some of this, but my understanding is that they are pretty limited:

https://github.com/fsharp/fslang-suggestions/issues/539

On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc.

  func filename(n):
    let value = String(n + 1)
    return "/path/to/" + value + ".csv"
  end

  load(filename(7))

I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.

Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.

Reply on news.ycombinator.com