Show HN: A simple lazy Python calculation engine, with spreadsheet demo | Hacker News Reader

Show HN: A simple lazy Python calculation engine, with spreadsheet demo

github.com

123 pointsbsdz31 comments

This is something I built a little while ago. It's a proof-of-concept and inspired by what I was doing in my day job.

To play with the demo spreadsheet you'll need to clone the repo and run the demo script, ie:

  git clone git@github.com:bsdz/calcengine.git
  cd calcengine
  python3 -mvenv --prompt calceng .venv
  . ./.venv/bin/activate
  poetry install -E demo
  python demo/spreadsheet/main.py

There's an inline gif showing the spreadsheet in use.

Hope it's interesting :)

Why create a venv and then run poetry install? Poetry already manages virtual environments.

True. Just an old habit.

This is a core principle in a lot of the dependency-graph-based frameworks seen in banks (secdb, Athena, quartz) etc. Inspired by those?

Indeed. Actually with-in the demo samples folder you can find a simple QuantLib example (quantlib.json). Although you'll need to install the QuantLib package separately as I didn't include it in the dependencies.

You may also be interested in: https://github.com/timkpaine/tributary/blob/main/docs/exampl...

Yes I am. Thanks for sharing.

The Dag/graph is one of the best parts of those tool chains. That is one library I wish had a good open source equivalent. Partly because it would be nice to use in any project / environment, but also because there would be a lot more documentation and examples. Take Pandas for example, if it stayed internal then it wouldn’t be the powerhouse that it is today.

Most dependency graph examples are based around calculations but it’s a versatile pattern and not limited to just math. I find the JS community has really taken hold of the dependency tracking and “reactive framework” idea to build UIs.

Why QT and not tkinter?

No reason in particular. Qt has some easy to use widgets and is well documented. I've also some experience with Qt.

This is really interesting because I was working recently on a way to have complete spreadsheets using only CSV files. The project is in my Github (https://github.com/yassirnajmaoui/runcsv and https://github.com/yassirnajmaoui/runcsv-gui) The only (major) thing is that the cells are run in top-bottom left-right instead of running a graph like this repo suggests.

This is cool! Also gives me a great excuse to post one of my favorite papers, "Build Systems à la Carte", about how make and Excel are solving the same problem in different ways: https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

> make and Excel are solving the same problem

Well, that's a sentence I didn't expect to read today. Count me intrigued.

This is pretty cool.

Spreadsheets are so powerful and really great and producing one-off solutions, but when you want to automate them you always need to redo most of the work in a "serious" programming language and wrap all of it in an API etc.

I always wondered how one could bridge the gap between creating a PoC computation in a spreadsheet and getting the same code to run in production without the rewrite. This could be a nice first step for a python server application designed in a spreadsheet.

There is a tool called wings iirc that lets you pull out DataFrames from an xlsx and vice versa

It’s called xlwings. Not a bad tool. https://docs.xlwings.org/en/stable/quickstart.html

Looks very interesting.

Can we plug numpy into this? Or is it not necessary with the assumptions that most steps will be cached and don't have to be recalculated.

Yes, you can use any existing python modules. In the demo you'll see I am using pandas dataframes in individual cells. Pandas is built on numpy.

Nice Python + Qt demo. Gonna learn from it.

Thank you. Hope there's something useful in there for you :-)

    you can use pip install -r requirements.txt

I don't see a requirements.txt file.

Good spot. You can actually just run the following if you don't want to use poetry.

  pip install git+https://github.com/bsdz/calcengine.git#master

Unfortunately, this will only install the core library without the demo spreadsheet files. Even if you were to add the extras modifier "[demo]", it would only install additional dependencies and not the actual demo code. To use the demo you'll need to clone and follow instructions in my other comment.

This outputs many pages of debug information and then dies. The error seems to be this:

    This package requires Rust >=1.41.0.

Hold it! (Pretend I'm Ace Attorney: https://www.youtube.com/watch?v=2lQgZ52iONU&ab_channel=DHC)

The solution here is, your python environment is borked. I've seen this many times.

Step one, is to install a fresh Python. You can do that using pyenv: https://github.com/pyenv/pyenv#installation

  curl https://pyenv.run | bash

Once you can do `pyenv versions`, you should see `system` is your only python.

You can run `pyenv install 3.9` to get the latest.

Now that you're running the latest 3.9, the goal is to install each of the dependencies of this project, one by one.

Step one: try to run the project. It will fail with an error.

Step two: install the library that it says is missing.

Step three: Go to step one.

I've followed this algorithm so many times that I can't count anymore how often it's saved me. Forget poetry, and forget a requirements.txt file. This is the only way I do it on my local laptop. (And I usually use the regular system python, but, yours seems to be in a suspect state.)

If after all of this you still get errors, then I'm truly sorry; god be with you, you're not alone, but I'm not sure anyone can help. It's definitely not the fault of this project that pillow can't be installed. :)

> your python environment is borked. I've seen this many times

Your solution looks simple and easy to follow. There's a flow chart to simplify this process here: https://xkcd.com/1987/

That's very strange. The core library uses pure python and demo uses pyqt5, pandas, matplotlib and pillow. I wasn't aware that any of those projects (or sub projects) depend on Rust. I'm running on Linux / Ubuntu 20.04 with Python 3.8. How about you?

(The pip command I just gave should only install the core library and so only depends on base python).

Here is how to reproduce it in a fresh docker container, so we have the same environment:

    docker run -it --rm debian:10-slim /bin/bash

    apt update
    apt install -y git python3-pip --no-install-recommends
    pip3 install git+https://github.com/bsdz/calcengine.git#master

Result: "This package requires Rust >=1.41.0."

Yes you're right! There was something wrong with poetry build system. When I initialised the poetry project roughly a year ago it placed an unpinned dependency (for the build part) for cryptography which recently switched to rust. I'm not sure why poetry hadn't vendorized that package or, at least, pinned the version. In any case, I updated the repo's pyproject.toml to use poetry's new build settings. Then, to check the install works in docker I switched to python's base images and did the following:

  docker run -it --rm python:latest /bin/bash
  curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
  cd /tmp
  git clone https://github.com/bsdz/calcengine.git
  cd calcengine
  python3 -mvenv --prompt calceng .venv
  . ./.venv/bin/activate
  pip install "cppy>=1.1.0"
  ~/.poetry/bin/poetry install -E demo

Note that I had to force cppy to install using pip. At some point I'll refresh the poetry lock file to fix these glitches that seem to appear as an untouched project ages.

Does it depend on the cryptography library? https://github.com/pyca/cryptography/issues/5771

It doesn't look like it:

  $ grep name poetry.lock | cut -d" " -f3 | tr -d '"' | tr '\n' ', '
  appdirs,attrs,black,click,cycler,flake8,kiwisolver,matplotlib,mccabe,
  mypy,mypy-extensions,numpy,pandas,pathspec,pillow,pycodestyle,pyflakes,
  pyparsing,pyqt5,pyqt5-sip,python-dateutil,pytz,regex,six,toml,typed-ast,
  typing-extensions

> I'm running on Linux / Ubuntu 20.04 with Python 3.8. How about you?

Indeed, on Python 3.9 it doesn't install, as it requires Python 3.8.

I'd like to try this, how to proceed?

Reply on news.ycombinator.com