rpep

Generally for scientific libraries where this is common, the expectation is from many users that you'll make it available via Conda Forge/Spack/EasyBuild for easier distribution, and that anyone wanting anything else will either get a manylinux wheel targeting a lowest common denominator machine, or will have to build from source.

I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get.

If you test a single array operation with NumPy then it will compare favourably to a low level implementation, but where it generally will lose out is when you’re chaining together multiple operations which a compiler will vectorise more efficiently. For e.g. doing things like add two arrays then multiply them by another array and subtract a constant. If I put that in a C or Rust function I’d expect that to be auto-vectorized using fewer operations than what NumPy will do, and you of course also remove the overhead of the foreign function interface and control returning to the interpreter. On top of that, it’s usually trivial to drop in thread level parallelism in a lower level language, and NumPy doesn’t operate with core level parallelism out of the box for most operations, you have to use something like numexpr to achieve it.

I've personally used OpenMP + SIMD intrinsics or auto-vectorisation to great effect in scientific software and got performance >100x that of NumPy, but it's certainly not a free lunch. The question you have to ask yourself is normally “is it worth it”, and that’s something that can only be answered by profiling, looking at the overhead of maintaining the architecture, and understanding whether the increased friction with debugging and building is worth the hassle. Additionally, if you're doing linear algebra then generally if you're dropping to a lower level language you still will want to be using the same libraries that NumPy etc. uses unless you're exploiting some property of the matrix structure that can make operations more efficient in memory or compute. I worked on one codebase for e.g. where there was a block matrix structure and a hand written implementation of a solver was written because much more efficient operations could be performed with knowledge of that structure.

I spent a lot of time looking at this at work, and it was not smooth sailing for a complex workflow.

The main issue is that B2B supports it pretty well, but if you need to say, do something like add an extra attribute dynamically to the JWT tokens, or allow automatic migration from a legacy IdP on first logon, you are forced to use B2C and Custom Policies which are effectively a programming language defined in XML and which are quite nasty to use. In addition to that, B2C doesn’t fully implement all of OIDC so things like client credential flow are still in “Preview” at the moment. Deployment of custom policies is also only by use of a “Preview” API too. It didn’t feel mature enough to me.

Typical storage for a string string in C is just an array of “char” type data of some fixed length. The string is often shorter than the char buffer it’s stored in, so there needs to be some way of terminating it and for historical reasons/convention that is “\0”.

You can replicate the same sort of storage of the string length alongside the buffer itself that other languages have by using structs quite easily.

There’s a lot of variance between codes that run on a HPC system like this. You’d expect to see molecular dynamics, fluid dynamics, weather forecasting codes, density functional theory etc. etc., and the first two especially have a lot of variance in the codes and methods used to do those simulations.

I can’t remember the figures off of the top of my head but I read a report those use cases dominated ARCHER (UK national supercomputer) compute time a few years ago.

There are a few things I subscribe to, but they’re pretty few and far between now as I deliberately try and keep it to a minimum. For e.g. I use Google Photos and Gmail so I pay to upgrade my storage to 100gb, and I have a Spotify family subscription, but I think that’s it for ones I plan to keep.

I have a LastPass subscription but I’m planning to migrate off of that when the next renewal comes around as I don’t think the cost is justified compared to OSS options and I’m concerned at the recent issues they’ve had.

I used to subscribe to Evernote, but when I left academia I lost the need for that, and I was getting fed up with it getting slower and slower too.

Countries in the EU switched from coal to gas mainly because it’s about 50% of the carbon emissions per kWh:

https://www.forestresearch.gov.uk/tools-and-resources/fthr/b...

I would say that the Sellafield plant has a huge amount of nuclear waste that the UK has to deal with from nuclear weapons development and early power station decommissioning, and that modern nuclear power stations do produce waste but at considerably lower quantities than we already need to deal with, to the point that it’s basically a rounding error.

Construction on Hinckley Point C didn’t start until 2017. Until that point it was going through political and legal challenges and planning.

It’s questionable whether the price can stay as low as it currently is though. The three major manufacturers of wind turbines are basically operating at a loss on the turbines themselves. The price can’t go much lower.

In the early 2010s, a lobbying campaign against new nuclear in the U.K. basically started, including people like the deputy prime minister [1] complaining that it was too slow and renewables would be faster and it wasn’t necessary.

Fast forward to 2022 and we still don’t have enough renewables, the UK is not very windy at the moment so there’s not enough base load capacity, and we only have 3 active coal fired power stations left. Oh, and we’re in an energy crisis and France, which is normally a net exporter to the U.K., is needing to import energy, making the crisis even closer. It would have been pretty useful to have had a new nuclear power station online right about now.

[1] https://www.newstatesman.com/politics/2011/03/nuclear-power-...

I did similar, I asked it to compute the median on some filtered time series data, but as I added more requirements to the filtering it couldn’t seem to adapt the program.

The version support for Boost is still so impressive. GCC 4.6.3 is still one of the test compilers and was released in 2012!

> You want to signup for a digital service from the government, your bank, your insurance company,.. . Well you need to send in a copy of a bill and your passport.

This isn't generally the case; for the most part, a UK resident just answers some questions on application and won't be asked for ID to take out financial products. It's only normally if you fail to verify that you'd be asked that, which might happen if you've got no address/financial history here for e.g.

For online government services all tax/benefits are online and pretty seamless. The things that aren't seamless are much less common - for e.g. renewing driving license I remember being quite painful.

The account number/sort code combination alone is not usable by a fraudster to withdraw money in any way in the U.K., so perhaps this is where the disconnect is. You’d need to physically go to a branch and present the bank card and enter the PIN number, or present photo ID with matching details to the account.

There are “direct debits” that allow companies to draw money from an account, but they can only be set up by businesses, there are special guarantees, etc. etc. that mean the bank is liable to reimburse you immediately under the “Direct Debit Guarantee” if you report a fraudulent transaction.

I really don’t think people in the US realise quite how far behind it is in banking services.

In the U.K. we’ve had online banking for 20 years now, and telephone banking before that. With most banks you can send an amount up to £25k to any account in the country and the money will be there pretty much instantly. All you need is the name of the person/business and their 6-digit sort code and 8 digit account number. Since about 18 months ago, it automatically warns you if the persons name doesn’t match on the name on the account. We’ve had chip cards and PIN rather than signature in Europe since the early 2000s, and contactless cards started being launched in 2007.

It's not that bad:

  jobs:
    unit-tests:
      runs-on: python:3.8.15-slim-buster
      - name: "Run tests"
          working-directory: ./bin
          run: unit-tests.sh

I’m not sure the niggles here are platform things - they’re hard lessons that come from experience in using CI for the most part.

Generally you want to aim for writing things in scripts that your CI system calls, so that they can be run locally trivially and so that migrating away from one CI provider (for whatever reason - cost, employer decides to shift, etc.) is not a huge piece of work.

I’m also trying to do this and haven’t got very far yet.

The thing I’m trying to do is write things I would have liked to read myself, because it would have saved me time in my day to day work, or because it’s interesting to me.

Have you tried out Spack? I used this quite a bit in academia to install the scientific sorts of libraries many Python packages depends on without needing to rely on the OS versions which were often out of date.

I’m generally of the opinion that you either go all in for OS distributed Python packages, or you manage all versions yourself.

When I first learnt Python about 10 years ago, I found it much more common to install things from package managers because at the time, wheels had only just been proposed with PEP427 and so there was a lot more compiling from source if you installed things from PyPi. For similar reasons, Conda was popular.

These days, as long as you’re on a reasonably well supported platform (x86_64, ARM, and not Alpine Linux) and recent-ish version of Python (>3.7), then you can probably get a new binary version of most dependencies very quickly.

Predictive maintenance software for renewables assets is a fairly large area of growth.

It’s trivial but not useful when good solutions already exist

I'm a strong advocate (after migration between various cloud CI providers free tiers when I worked in academia) of making your build process as agnostic as possible to the CI provider. That generally means putting things into shell scripts and calling them, which can be a bit painful, and you'll never get 100% there, but it also has the added benefit of being able to debug things locally too if something goes wrong with the pipeline.

The first time I used GH Actions I thought there was a strong vendor lock-in element to it which I wasn't hugely comfortable with. I'm a big fan of using GitLab CI these days which seems to be a good trade off between various considerations.

I think the big point of interest will come with Python 3.12 where it's proposed that as part of the faster CPython project, the GIL-per-process will be removed.

My favourite resources for FEM were the book and courses that Hans Petter Langtangen (RIP) wrote at Simula. FEniCs made FEM so easy, it’s truly an excellent software project that is unfortunately not as well known as it should be within the community.

http://hplgit.github.io/num-methods-for-PDEs/doc/web/index.h...

http://hplgit.github.io/num-methods-for-PDEs/doc/pub/index.h...

I worked in C++ in the UK for a number of years and interviewed for a couple of positions like these. The main reason I didn't want to work for a financial company is that they insist on high number of working hours and being located in-person in London.

When I studied Physics, I did 2 years (4 semesters) of C++. Nowadays, nobody would do that, you'd learn Python by default.

The Gates of Europe: A History of Ukraine, by Serhii Plokhy.

He also wrote an excellent book about Chernobyl which I read last year.

On top of that, you almost certainly do not want to instantiate an N x N matrix, because sparsity is an important factor in most real problems.

There’re many people doing larger matrices, but of course they’re mostly sparse and so this isn’t relevant to them.