Show HN: SpreadServe – Cloud automation for Excel spreadsheets | Hacker News Reader

Show HN: SpreadServe – Cloud automation for Excel spreadsheets

spreadserve.com

55 pointsosullivj28 comments

Hi, long time trading systems hacker and originator of SpreadServe [1] here with some background: I was inspired to build it after finding myself in a meeting in a London Investment Bank in 2014 hearing a manager say "the traders have defined the accrual calcs in a spreadsheet, now we just need a couple of C# devs for a few months to build a real server to do the calcs". I realised I'd seen that pattern repeat time and again over the ten years I'd spent working on London trading floors. By hacking together Excel Addins in C++ myself I'd enabled an options trader to build a program trading system in Excel by plugging his pricing sheets into Liffe's ETO limit order books. I put an exotics trader on Bloomberg by connecting his huge 30 tab sheet to my etrading gateways, then gradually replaced the spreadsheet with C++ pricing servers. My own XLL Addin was running on hundreds of trading desktops, and I'd helped any number of traders lash together ad hoc pricing and risk spreadsheets with XLLs connecting them to market data, prices and quotes [2]. The same cycle kept repeating: trader whips up spreadsheet, sheet becomes business critical, trader tires of manual keying of data, hitting F9, and the flakiness of huge Excel sheets and demands the dev team replace the sheet with a scalable service. Then the dev team takes months to replicate the spreadsheet functionality in a C++, C# or Java server, and the trader gets frustrated by lengthy change cycles.

I realised that we can escape that frustrating cycle by building a server runtime for spreadsheets. Just as node.js took JavaScript to the server side, SpreadServe is "node.xls": an asynchronous scalable server side runtime for Excel spreadsheets that supports XLLs, RTD & VBA as well as a REST API, RDBMS integration and websockets subscribers. As Felienne says [3] "spreadsheets are code". But that code is trapped in a desktop IDE for grid based visual functional programming: Excel! And that's not the right environment for reliable automated operation. So SpreadServe's aim is to separate the development and rumtime environments for calc heavy spreadsheets: Excel for rapid change cycles by tech savvy power users like traders, then SpreadServe for production scalability, resilience and audit logging.

[1] http://spreadserve.com

[2] http://babbingtonslade.com/industries/

[3] https://etrading.wordpress.com/2015/08/13/spreadsheets-are-c...

This is awesome. Excel is such a valuable tool for its ease in prototyping and testing calculations, but the "classic" challenge is when/how to operationalize it.

The problem is that prototypes usually end up being the production system.

And that's exactly the problem that SpreadServe address. A prototype spreadheet will operate far better in production as a scalable resilient SpreadServe server deployment than as a manually operated desktop spreadsheet.

I really like the concept of excel calc automation. Just one humble suggestion: add some eye candy to make me feel I am driving the batmobile.

Sincerely, your landing page gives you a very good impression about what spreadserve does and how sophisticated it its - but once you land on the server page it doesn't look as impressive. It works really good but I don't have the feeling I am given some superpower/I'm driving the batmobile.

You're right - the GUI is a very simple Angular 1.5 implementation. All the smart stuff is happening on the server side. Eye candy is definitely on the roadmap post beta.

I had a summer job in 1997 at a large Insurers (kill me), and people on my team would spend at least half an hour a day waiting for networked, dependent Excel files to calculate after opening them.

Since then, I have always suspected it was still a problem in large financial institutions, although the few times I mentioned it to people they looked at me blankly. Now I feel vindicated!

Out of interest, does the Financial Industry use any analogues to source control and unit testing for spreadsheets?

There are several vendors out there selling version control for spreadsheets like pathio, Finsbury Solutions. IMHO the problem is that devs know the value of version control, but it's a hard sell to the busy end users building sheets who see only barriers and controls. I believe unit testing for spreadsheets could eliminate the kind of errors that we saw in the London Whale spreadsheet [1]. SpreadServe's Python API enables you to build spreadsheet unit tests. I really should get round to posting some sample sheet unit tests on github.

[1] https://baselinescenario.com/2013/02/09/the-importance-of-ex...

I wrote a blog post about unit testing Excel workbooks using Python's built-in unittest module: https://www.zoomeranalytics.com/blog/unittests-for-microsoft...

When you say VBA support, does that mean that Spreadserve will calculate VBA UDF like any other function? (One of the main limitations of spreadsheetgear which seems fairly similar).

Yes. Support for VBA UDFs, XLL UDFs in C++ or C# and RTD is what differentiates SpreadServe.

I'm impressed. And it works with ExcelDNA UDF I presume?

Also one more question. Is it meant for server only or can it be used in a desktop app also? And if so does it require an install on the machine or can it be used just by copying the assemblies in the app folder?

Thanks! SpreadServe's own SSAddin [1] is Excel-DNA based, uses RTD, and works perfectly in both SpreadServe and Excel. The youtube channel has a video demo [2].

[1] https://github.com/SpreadServe/SSAddin

[2] https://www.youtube.com/watch?v=MmPPMsOCmqA&t=290s

To answer the second question: you can run SpreadServe on a Windows desktop OS just as well as Windows Server 2008, 2012 or 2016. The install is designed to have a minimal footprint so it can work with corporate IT packaging systems. So a SpreadServe install doesn't touch the registry, and sets just two env vars. Which means you can zip the directory tree, drop it on another box, set the two env vars, and off you go.

I did several Excel to App implementation for the reasons you mention and more.

I'd like to better understand the architecture of your proposal to consider it for my future projects. Does it run real Excel instance under the hood? Or is there a full emulation/interception going on?

Can you cope with several linked excel spreadsheets at the same time ? Can you handle custom DLL loading/extending? Or perhaps, are you willing to share a whitepaper/architectural outline?

I'm using the Open Office Calc engine under the hood for Excel formula emulation. I extend that with my own implementation of the Excel4 API's xlcall32.dll, so that XLLs can invoke the Excel4 API. I also emulate Excel's built in RTD client, so that XLLs that supply an RTD server can push real time data into a sheet. Feel free to download the install kit [1], or email me on ssbeta@spreadserve.com for more detailed discussion. Thanks for the interest!

[1] http://spreadserve.com/s3/downloads.html

And yes, SpreadServe can cope with linked spreadsheets by means of websockets. Which means sheets can be daisy chained into a real time calc graph. This youtube video [1] demonstrates.

[1] https://www.youtube.com/watch?v=MmPPMsOCmqA&t=10s

Knowing John's work at various investment banks, I would venture to guess that Spreadserve will be a top-notch product. Scaling Excel is something that has been needed for a long time. How many times have we, in IT, gotten complaints from traders and quants about Excel crashing with a large number of calcs?

How does it compare with HPC Services for Excel? https://msdn.microsoft.com/en-us/library/ff877824(v=ws.10).a...

HPC requires you to refactor your spreadsheets to distribute the workload across the nodes. XLL C++ code needs to be tweaked to make them parallelizable. And the resulting ensemble still needs to be driven from a desktop Excel. SpreadServe takes unmodified spreadsheets and runs them on a server, completely automated and decoupled from any GUI.

Really nice work! As someone who has automated several spreadsheets by essentially reimplementing them in a "proper" language like C# or Python for several financial institutions, I see the immediate need for a product like this across all business units. Not only does your product help save time and resources, but your focus on proper testing can easily help reduce very costly errors (which happen more often than one would like to think).

One question, how does your product integrate with 3rd party APIs like Bloomberg or SNL?

Thanks! SpreadServe has 3 APIs: REST/websockets, C++ and Python. A SpreadServe deployment is a cluster of servers. So if you wanted to push Bloomberg market data into a spreadsheet you'd code a new server component in C++ and/or Python against the BBG API.

What is the licensing model? Completely open source, free for small time use with license fees for heavier use, charge for all commercial use, undecided yet?

I ask out of interest rather than because I am thinking of licensing it. I can see it will be highly useful though to those that need it.

Free for public instances of SpreadServe that phone home spreadsheet details to spreadserve.com's live page [1]. Chargeable for offline instances that don't phone home as they have a paid license key. Some of the codebase is OSS [2]. More details on the license model here [3].

[1] http://spreadserve.com/cws/cdashboard.html

[2] https://github.com/SpreadServe/SSAddin

[3] http://spreadserve.readthedocs.io/en/latest/license.html

Thanks. Making the add-in open source makes sense from a business perspective, lets people inspect the code they are running on their machine.

I worked and help launch something similar a couple years ago : http://calcfusion.com/ .

This is really good.

What is your sales plan?

Free public licensed [1] SpreadServe deploys phone home to publish spreadsheet info on the spreadserve.com live page [2]. That's the way I aim to build adoption. Offline license keys are chargeable, and stop SpreadServe phoning home. Small hedge funds and asset mgrs are my first target market for paid license keys. I'm not going after big banks early as the enterprise sales cycle is so lengthy and torturous is can kill small vendors. I've seen that first hand, so those customers are for much later.

[1] http://spreadserve.readthedocs.io/en/latest/license.html

[2] http://spreadserve.com/cws/cdashboard.html

Reply on news.ycombinator.com