rwlincoln

MATPOWER and OpenDSS test cases are also available in CSV format:

https://github.com/casecsv

https://github.com/cktcsv

L2RPN is a series of competitions, with the aim of training an Artificial Intelligence agent to operate a power network. The Delft-AI-Energy Lab is hosting a remake of the L2RPN challenge in the Netherlands. The L2RPN Delft 2023 competition offers an exciting opportunity for students and researchers in the Netherlands to test their AI solutions against others in a realistic and challenging environment.

SEEKING WORK | Berlin | Power Engineer

I specialize in combining power system simulation libraries with web technologies. I have experience with power flow simulation, optimization and transient simulation. I also work on solvers for sparse systems of linear equations and sparse matrix ordering algorithms. For web development I typically use Vue with Web Components and have experience with WebAssembly.

Sector: Electrical Power Engineering

Nationality: British

Language: English

Experience: 10+ years

Qualifications: MEng, PhD

Languages: Rust, Python, TypeScript...

Source: https://github.com/rwl

Flagship: https://matpower.app

Contracts: EU or US (Remote)

Rate: 120 $/hr (min. 20 hrs)

Comms: Asynchronous

Email: HN username @gmail.com

Armadillo is particularly notable in that has a Python interface (based on pybind11). Although, sparse matrix support still needs to be added. Also the PyArmadillo API resembles that of MATLAB, which can be convenient when translating.

https://pyarma.sourceforge.io/

I am surprised not to see any mention of the OSQP (Operator Splitting Quadratic Program) solver. It is the most impressive open source solver of this type that I have seen published in recent years. It appears to have been developed as a collaboration between Princeton, ETH Zurich, Oxford, Stanford and some other prestigious names. The benchmarks show that it compares favorably with leading proprietary solvers:

https://github.com/osqp/osqp_benchmarks

The problem described seems to be an ideal use-case for Machine Learning. The MATPOWER Optimal Scheduling Toolkit (MOST) can already solve:

"a stochastic, security-constrained, combined unit-commitment and multiperiod optimal power flow problem with locational contingency and load-following reserves, ramping costs and constraints, deferrable demands, lossy storage resources and uncertain renewable generation."

Much more and it becomes a global optimization problem where you can never really be sure you are not just stuck in a local optimum. The L2RPN (Learning to Run a Power Network) challenge, from RTE-France, is the most interesting effort I have seen applying Machine Learning to energy system management.

https://l2rpn.chalearn.org/

https://github.com/rte-france/l2rpn-baselines

The competition has been renewed for 2022 and has been accepted for the IEEE World Congress on Computational Intelligence in July.

https://wcci2022.org

The file is written to the in-memory virtual file system provided by Emscripten. It can be accessed from the console:

  pyodide._module.FS.readFile("/pjm5bus_out.txt", { encoding: 'utf8' })

I published this just to demonstrate that Time Domain Simulation was working to Hantao Cui, the author of Andes. Small Signal Stability Analysis will require SciPy and an ARPACK package. I would like to create a PWA that runs Andes in a web worker, like I did for MATPOWER:

https://matpower.app/

Recently, I was able to get Andes v1.3.5 (a Python package for power system transient dynamics simulation using SymPy) to run in WebAssembly using Pyodide v0.16.1.

https://andesapp.github.io/

Andes, a Python package for power system simulation, uses SymPy now. In previous versions, computing the derivatives for dynamic models was done using complex equations written out manually. SymPy greatly simplifies the process of adding custom device models, which power companies often wish to do.

https://andes.readthedocs.io/

The GNU Octave interpreter and its dependencies are all in the matpower.wasm file. 19Mb is still quite large for a website, but it is loaded asynchronously and WASM get compiled as it streams in. The interpreter runs in a WebWorker using a Promise based interface so the UI doesn't get held up. Many of the dependencies (e.g. FFTW, CHOLMOD, ARPACK, Qt, HDF5) are disabled as they are not needed to run MATPOWER. There was no particular reason for using v4.4.1. The core functionality of GNU Octave doesn't change much and that version was stable and sufficient for my needs.

Satisfying the BLAS/LAPACK dependency was one of the biggest challenges. I ended up passing LAPACK 3.4.2 through f2c before compiling the result with emcc. PCRE and some SuiteSparse libraries were the only additional dependencies needed for my project and they compiled with Emscripten without too much difficulty. The rest was just hacking GNU Octave's Autotools build system, which gleans a lot of information from the system environment, into working with emconfigure and emmake. My changes to the GNU Octave source code are available here:

https://github.com/rwl/octave-4.4.1/tree/web-assembly

I was able to compile GNU Octave 4.4.1 into Web Assembly for my MATPOWER PWA. When Mozilla was still at full force OctIodide might have been an interesting project for them.

https://matpower.app

https://github.com/iodide-project/iodide

The whole application runs on the client (in the browser). The server just serves up static resources. These are cached by the browser so the application will still work if you are offline. The result is much like the desktop applications you mentioned, but without anything needing to be installed.

The JavaScript application presents the UI and controls editing of a JSON representation of the MATPOWER case structure. This data can be seen using the JSON option on the Export page. The WebAssembly program contains GNU Octave and its dependencies. It is run in a WebWorker (background thread) and the JSON data is passed to and fro. The program that converts the JSON into Octave data structures and back again is available here:

https://github.com/rwl/octpower

Octave includes support for sparse matrices and MATPOWER makes use of them by default. It doesn't require any additional effort from me. In general, Octave can quite match MATLAB for performance, but it is typically not far off. Running it as WebAssembly also comes with a small performance penalty. I haven't run any benchmarks yet, but this can be expected to improve as browser vendors optimize their runtimes.

Octave is using UMFPACK to solve sparse systems of linear equations. There may be some performance to be gained by using KLU with AMD preordering.

I am interested to know how best to compile BLAS and LAPACK to WebAssembly. Traditionally, implementations optimized for a particular machine architecture are used to extract maximum performance. However, WebAssembly targets a stack-based conceptual machine. At present, I use LAPACK v3.4.2 and convert it to C with f2c before compiling to WebAssembly. It would be interesting to perform some benchmark tests against other implementations and compare across browsers.

My view is that the open source projects for power system simulation available now are approaching a stage where they can perform all of the main functions of commercial tools. In some cases, they even offer functionality that expensive proprietary applications do not have. However, they are typically provided as libraries or with text-based interfaces.

WebAssembly creates an exciting opportunity to combine these libraries with convenient, reactive user interfaces that help to manage the their complexity. The web development community is so large and the frameworks and tools available are of such high quality that an individual, or small team of developers, can quickly create complex applications that would have previously taken much longer. Ultimately, this should accelerate the process of bringing the latest research to industry.

There are some test cases included in MATPOWER that are around that size and may be imported. I only embedded small test cases in the application to keep the download size down.

I imagine this app being quite useful to lecturers teaching Power Engineering at Universities. Being able to just send students a link and not requiring them to install complex libraries, deal with licenses or restricted trial versions and them not having to use a particular OS or CPU architecture should help get them started quickly.

https://github.com/MATPOWER/matpower/tree/master/data

Shouldn't this be a WebAssembly (WASM) only feature? It seems a shame to be introducing the potential for race conditions in JavaScript. Compiling languages that support multiple threads into JS seems to be what is motivating this work.

Currently only the server-side of Vaadin has been translated and Muntjac relies on the GWT compiled Java from Vaadin for the client code. The plan is to translate the rest and use Pyjamas to produce the JS. That way new components could be defined in Python.

Thanks. I'm not sure how it compares to Vaadin in terms of speed, but it does run on PyPy (and IronPython), which might help.

Thanks. I'm new to HN and didn't realise what this would do to my traffic. It is currently on EC2, but I had it working on GAE. Perhaps that would have helped with the scaling.

I don't know much about Django sadly. I can tell you that the ApplicationServlet is just a WSGI app and can be stacked accordingly.

One issue is that the session gets unpickled/pickled with each request. When each of the samples has been run the pickled session files are ~2Mb.

I just switched to a High-CPU EC2 instance. I hope it helps. Otherwise:

$ easy_install muntjac

$ muntjac

to run the demos locally.