Each card (not core) is running an embedded version of Linux. Binaries are almost all linked to busybox. You can mount NFS shares to share data.

The embedded systems need specially compiled binaries. With the Intel compiler, this is usually achieved by turning on a flag at compile time. So yes, you can install and run your own software on the nodes.