Ask HN: What should go in an Excel-to-Python equivalent of a couch-to-5k? | Hacker News Reader

Ask HN: What should go in an Excel-to-Python equivalent of a couch-to-5k?

10 pointsetothepii7 comments

Yesterday, my co-founder published a blog about her experiences Ditching Excel for Python in her job as a Reinsurance Analyst [0].

One of the responses on reddit [1] asked what they should do, "Step 1 day 1," if having read Amy's post they were convinced to try and begin the long journey from tangled Excel/Access spaghetti.

My (flippant) reaction to a friend that brought the comment to my attention was unhelpful; "Step 1 day 1, quit." So he has challenged me to write eight helpful blog posts during the remainder of my Garden Leave.

What should go in them?

[0] https://amypeniston.com/ditching-excel-for-python/

[1] https://www.reddit.com/r/Python/comments/knbv5t/ditching_excel_for_python_lessons_learned_from_a/ghm559c/?utm_source=reddit&utm_medium=web2x&context=3

View on HN

How to write functions in JS / VB script and call them from a cell expression.

How to name variables something other than AB3.

How to use physical units and datatypes. (How to specify XSD datatype URIs that map to native primitives in an additional frozen header row. e.g. py-moneyed and Pint & NumPy ufuncs)

How transitive sort works (is there a tsort to determine what to calculate first (and whether there are cycles) on every modification event?)

Which Jupyter platforms do and don't support interactive charts with e.g. ipywidgets?

pandas.df.plot(kind=) (matplotlib), seaborne (what are the calculated parameters of this chart?), holoviews, plotly, altair

Reproducibility w/ repo2docker / BinderHub:

   pip freeze > constraints.txt
   cp constraints.txt requirements.txt
   conda env export --from-history

Interesting, your suggestion is that the first thing to do is to learn how to make functions and variable names within excel?

Yeah if you port it to functions and verify that you haven't broken anything, you could then easily port to Python functions that you could call from Excel with an add-on that everyone that opens the sheet needs to have installed; but everyone that opens a sheet that calls Python must have that same extension (and all python package dependencies) installed, too

Also,

When is it better to have code in a notebook instead of in a module?

How to export notebook cells to a module with nbdev

How to write tests to assert the quality of the code and the model: @pytest.mark.parametrize, pytest-notebook, jupyter-pytest-2, pytest-jupyter

When is it appropriate to parametrized a notebook with e.g. papermill?

How to handle concurrency: dask.distributed + dask-labextension, ipyparallel

>My (flippant) reaction to a friend that brought the comment to my attention was unhelpful; "Step 1 day 1, quit." So he has challenged me to write eight helpful blog posts during the remainder of my Garden Leave.

One way to do it would be to do impedance matching: maximizing usefulness by bringing Python to the universe the person who asked the question lives in: Excel, Word, PDF, email.

This means "Automate the Boring Stuff"[0] to ease into programming with something specific to do that is relevant and useful right off the bat for manipulating files, spreadsheets, PDF documents, etc.

Openpyxl[1] for playing more with Excel files, and basically searching "Excel Python" on the internet for more use cases.

Python-docx[2] for playing with Word documents, extracting content from tables, etc.

PdfPlumber[3] for playing with PDFs. Sometimes you have data in .pptx files, and you can use LibreOffice to convert all of them to PDFs because you hate the pptx format:

  libreoffice --headless --invisible --convert-to pdf *.pptx

Also, if you both are using notebooks, take a look at what we're building at https://iko.ai. I posted about it a bit earlier[4]. It has no-setup real-time Jupyter notebooks, and all the jazz. Focuses on solving machine learning problems since we built it for ourselves to slash projects' time.

- [0]: https://nostarch.com/automatestuff2

- [1]: https://openpyxl.readthedocs.io/en/stable/

- [2]: https://python-docx.readthedocs.io/en/latest/

- [3]: https://github.com/jsvine/pdfplumber

- [4]: https://news.ycombinator.com/item?id=25608098

Using openpyxl to extract data from exist excel workbooks and other documents would seem like a good place to start.

I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks.

> I suspect many companies use excel workbooks as "forms" with lots of data at the same cell in multiple workbooks.

Downstream data quality costs can be minimized with data normalized schema and data collection process controls like forms-based data validation.

There are established UI/UX design patterns for data validation of user-supplied data: accessible [web] forms with tab-ordered input fields and specific per-input feedback with accessible HTML5 and ARIA. IIUC, Firefox now supports PDF forms, too?

Why would we move from a spreadsheet to an actual database?

Data integrity:

Referential integrity (making sure that record keys actually point to something when creating, updating, or deleting),

Columnar datatypes (float, decimal, USD, complex fraction),

Access controls (auth(z): authentication and authorization),

Auditing (what was the value before and after that) and Disaster Recovery,

Organizationally-unified schema development and corresponding validation.

Repeatability / Reproducibility: can you replay the steps needed to build the whole sheet? What parameters were entered and how to we script that par so that we can easily assess the relations between the terms of the argument presented?

Reply on news.ycombinator.com