ingenieroariel

I wrote: "we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.)" April, 2023

We are both late and early.

https://news.ycombinator.com/item?id=35527692

With Apple devices you get very fast predictions once it gets going but it is inferior to nvidia precisely during prefetch (processing prompt/context) before it really gets going.

For our code assistant use cases the local inference on Macs will tend to favor workflows where there is a lot of generation and little reading and this is the opposite of how many of use use Claude Code.

Source: I started getting Mac Studios with max ram as soon as the first llama model was released.

I like Nix as well, you can use this one liner in OSX or Linux to try out arcan/durden/cat9, as it matures you can expect arcan applications to be made available the way html pages/apps and this kind of nix derivation would let you run the "browser":

nix run --impure 'git+https://codeberg.org/ingenieroariel/arcan?ref=nix-flake-buil...'

To add to lproven's point.

An article called "A Spreadsheet and a Debugger walk into a Shell" [0] by Bjorn (letoram) is a good showcase of an alternative to cells in a Jupyter notebook (Excel like cells!). Another alternative a bit more similar to Jupyter that also runs on Arcan is Pipeworld.

[0] https://arcan-fe.com/2024/09/16/a-spreadsheet-and-a-debugger... [1] https://arcan-fe.com/2021/04/12/introducing-pipeworld/

PS: I hang out at Arcan's Discord Server, you are welcome to join https://discord.com/invite/sdNzrgXMn7

The scipy/numpy to matlab is a good example. In my opinion it is on its way but in many places the timing is more like 2010-2013 where a lot of people knew python was the future but universities still used only Matlab.

I think the answer depends on the country: In places where the government uses QGIS it is like Blender. In places where ESRI has a stronghold it is like LibreOffice.

Mark Knol is a great generative coding artist: https://github.com/markknol/

Chris Randall is pretty awesome too: https://www.instagram.com/chris.randall.art/

Hey AJ, this is almost on topic, do you know of a more up to date version of the dataset you used on the blog post release for H3 v4.0.0 [1]? They stopped updating in Oct 2023. Thanks! [1] https://data.humdata.org/dataset/kontur-population-dataset

It is a gui framework that allows you create terminals where you can detach any running process into another terminal.

Since it is a complete toolkit, you can have detachable applications where you send both code and state to a server and retrieve it from another device (like Apple's continuity).

In the end it is just a bunch of lua scripts talking to other components via /dev/shm and to other computers using a new protocol called a12://

I did not believe you and just typed it on OSX, half a minute later the app was ready for me to use.

nix run nixpkgs#pyspread [0/1 built, 3/113/132 copied (1311.8/1721.6 MiB), 280.4/300.7 MiB DL] fetching llvm-16.0.6 from https://cache.nixos.org

https://pasteboard.co/P1eh7B7W8C9R.png

I'll let Kyle chime in but I tested it a few months ago with millions of polygons on an M2 16GB of RAM laptop and it worked very well.

There is a library by the same author called lonboard that provides the JS bits inside JupyterLab. https://github.com/developmentseed/lonboard

<speculation>I think it is based on the Kepler.gl / Deck.gl data loaders that go straight to GPU from network.</speculation>

When you queried their 'Open Data' datasets and linked with your own it was absurdly cheap for some time. Granted we used our hacking skills to make sure the really big queries ran in the free tier and only smaller datasets got in the private tables.

I kept getting emails about small changes and the bills got bigger all over the place including BigQuery and how they dealt with queries on public datasets. Bill got higher.

There is a non zero chance I conflated things. But from my point of view: I created a system and let it running for years - afterwards bills got higher out of the blue and I moved out.

One way to wrap your head around it is apt/systemd in a pip-like config file.

Check out devenv.sh from the Cachix people, it allows you to list stuff like pip for any language, including services / postgresql extensions:

   { pkgs, ... }: {
   services.postgres = {
    enable = true;
    package = pkgs.postgresql_15;
    initialDatabases = [{ name = "mydb"; }];
    extensions = extensions: [
      extensions.postgis
      extensions.timescaledb
    ];
    settings.shared_preload_libraries = "timescaledb";
    initialScript = "CREATE EXTENSION IF NOT EXISTS timescaledb;";
  };
   }

Imagine if we had said the same about Nginx. Let's evaluate things on technical merits, specially if they are using open source licenses we understand. In this case GPL v3.

I recently learnt about this: https://nlnet.nl/project/libnix/

A project funded by the EU to bring Nix to Windows.

(edit: typo and clarity)

I went through a similar phase with a process that started with global OSM and Whosonfirst to process a pipeline. Google costs kept going up (7k a month with airflow + bigquery) and I was able to replace it with a one time $7k hardware purchase. We were able to do it since the process was using H3 indices early on and the resulting intermediate datasets all fit on ram.

System is a Mac Studio with 128GB + Asahi Linux + mmapped parquet files and DuckDB, it also runs airflow for us and with Nix can be used to accelerate developer builds and run the airflow tasks for the data team.

GCP is nice when it is free/cheap but they keep tabs on what you are doing and may surprise you at any point in time with ever higher bills without higher usage.

The use case was a client creating electrification plans based on structures from satellite data. We were able to get rid of gdal/pandas/networkx and several other dependencies and ended up with a fast python based process that could be given to clients for them to reproduce on their own machines (windows workstations).

In my use case, the niche is not having WSL/Docker available and letting end users repeat studies or re-run configuration scripts.

Thanks for updating the blog post too with the datasette screen recording, the speed difference is quite noticeable.

Adding it here since a few people were wondering about it in the comments, but feel free to check the original article for the 2024 update:

https://ahgamut.github.io/images/ape-datasette.gif

There are separate static builds for aarch64 and x86_64, those are put into a "fat binary" that runs on current generation hardware/software combos (osx with aarch64, win with x86_64 and linux with both). Just building for two things is better than figuring out the whole matrix of OS vs architectures which was the other option.

For other architectures like Power/armv7/i686 the software can run using the Blink project [0] [0] https://github.com/jart/blink

The reality is a bit different, the work on Python 3.6 was checked into the Cosmopolitan repo and I have been able to use it for production workloads that are in pure python. [0]

As Cosmopolitan Libc has evolved, it has been possible to compile more software without modifications, and that includes latest Python through a project called superconfigure[1].

Last person who tried to reproduce it from scratch did it last week (granted it too them a few days of solid work) but in the end they ended with a portable binary with Python 3.11.9, brotli, ssl and asyncio for their work related project.[2]

[0] https://github.com/jart/cosmopolitan/tree/master/third_party... [1] https://github.com/ahgamut/superconfigure/ [2] https://github.com/croqaz/cpython/

My interpretation:

- Worst case: as good as 3.5 - Common case: way better than 3.5 - Best case: as good as 4.0

TLDR: A model that could be described as "3.8 level" that is good at math and openly available with a custom license.

It is as fast as 34B model, but uses as much memory as a 132B model. A mixture of 16 experts, activates 4 at a time, so has more chances to get the combo just right than Mixtral (8 with 2 active).

For my personal use case (a top of the line Mac Studio) it looks like the perfect size to replace GPT-4 turbo for programming tasks. What we should look out for is people using them for real world programming tasks (instead of benchmarks) and reporting back.

I just did:

./mixtral-8x7b-instruct-v0.1.Q8_0.llamafile --cli -t 16 -n 200 -p "In terms of Lasso"

I got 15 tokens per second for prompt evaluation and 8 tokens per second for regular eval.

The same hardware can run things much faster on OSX, or if you use more quantization but I prefer to run things at Q8 or f16 even if they are slow. In the future I how to use GPU, ANE and the crazy 1.58 or 0.68 bit quantization but for now this does the trick handsomely.

I have.

On a Mac Studio with NixOS based Asahi Linux and 128Gb of RAM, mixtral 8x7b uses 49GB of RAM. At the same time I load airflow tasks that deal with world wide datasets (using ~60GB on 16 parallel streams with the performance cores) format is parquet and also mmaped.

Computer still has 8 efficiency cores and the whole GPU for visualizing the maps using lonboard / browsing / etc.

The computer uses 8-10W when idle, ~100W when running jobs or actively using the LLM and around ~200W when really using the GPU.

This makes it very efficient energy wise in my book compared to the beast of keeping a modern CPU and nvidia GPU on when idle. My electricity bill is unaffected.

Guix uses Nix under the hood (learned this when trying to compile it). Perhaps learning Nix first will let the grandparent be more effective in Guix later.

Since this uses cosmopolitan and the build script already downloads portable binaries from https://cosmo.zip has there any thought been given to wrap other portable binaries in scrapscript / download them?

Small, pure, functional, content-addressable and network-first sounds a lot like a mini Nix+ca-derivations [1]

[1] https://www.tweag.io/blog/2021-12-02-nix-cas-4/

Apple having so much cash perhaps the business model is to lease them?

The list of dependencies seems very short, apart from tornado it does not seem like the other ones pull in a lot of other deps.

Congrats, this looks very useful and awesome.

  dependencies = [
    # cli
    "click>=8.0,<9",
    # python 3.8 compatibility
    "importlib_resources>=5.10.2; python_version < \"3.9\"",
    # code completion
    "jedi>=0.18.0",
    # compile markdown to html
    "markdown>=3.4,<4",
    # add features to markdown
    "pymdown-extensions>=9.0,<11",
    # syntax highlighting of code in markdown
    "pygments>=2.13,<3",
    # for reading, writing configs
    "tomlkit>= 0.12.0",
    # web server
    "tornado>=6.1,<7",
    # python <=3.9 compatibility
    "typing_extensions>=4.4.0; python_version < \"3.10\"",
    # for cell formatting; if user version is not compatible, no-op
    "black",
  ]

I tried it last month and loved having Alacritty there. It built easily and ran well on both OSX and Fedora Asahi Linux.

That said, it does not seem yet ready to be a daily driver, small rough edges in the terminal behavior or the editor when using Vim mode were too distracting to do a long programming session.

Kudos to the authors, looking forward to giving it a spin again the future.