ingenieroariel

To your point: If you have your data as parquet files and load it into memory using DuckDB before passing the arrow object to Pandas you can run in arbitrary large datasets as long as you aggregate them or filter them somehow.

In my case I run analysis on an Apple M1 with 16GB of RAM and my files on disk are thousands of parquet files that add up to hundreds of gigs.

Apart from that: Being able to go from duckdb to pandas very quickly to make operations that make more sense on the other end and come back while not having to change the format is super powerful. The author talks about this with Polars as an example.

Great work Graham and team, I'll be switching to it on OSX.

I wonder if you took a look at some of the modifications done by portable-nix (https://github.com/DavHau/nix-portable), most important ones being:

  a) Allowing user to choose the location of the nix folder (for example $HOME/.nix) by using bwrap or proot
  b) Same binary for Win/Linux/OSX on x86_64 (Also see https://ahgamut.github.io/2022/07/27/ape-rust-example/)

And then, somewhat unrelated:

  c) Can it be used within NixOS as nix.package = ?

That's the latest one. It is the same as on the website.

If you need any help getting started, please join us on the Redbean Discord: https://discord.gg/eHtVmUEB

If you like Flask (I do too), you can do what we used to do before Flask existed (./djngo.com -p project runserver) where .python/project.py in the zip is:

  import os
  from django.conf import settings
  from django.core import management
  from django.conf.urls import url
  from django.http import HttpResponse

  # Django 2.2 Release Notes
  # https://docs.djangoproject.com/en/4.1/releases/2.2/
  # Documentation
  # https://docs.djangoproject.com/en/2.2/

  __version__ = 0.1

  DEBUG = True
  ROOT_URLCONF = 'project'
  DATABASES = {'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': 'project.db' }}
  SECRET_KEY = os.getenv('SECRET_KEY', 'Please set a SECRET_KEY as an env var.')

  if not settings.configured:
    settings.configure(**locals())

  def index(request):
    return HttpResponse('Hello Django!')

  urlpatterns = [
    url(r'^$', index),
  ]

  if __name__ == '__main__':  # pragma: no cover
    management.execute_from_command_line()

Please try again, I just re-ran the deployment and tested on my M1 (OSX):

  x@m1n1n0 ~> chmod +x djngo.com   
  x@m1n1n0 ~> sha256sum djngo.com   
  6205984f572575c407ce65502361b554af17584f076076bec7440613459d  1dff  djngo.com
  x@m1n1n0 ~> ./djngo.com 
  extracting blink-darwin-arm64 to /var/folders  /fn/4cdpvwcx1wnc53rzl2m0pn_m0000gn/T//.ape-blink-0.9.2
  Python 3.6.14+ (Actually Portable Python) [GCC 9.2.0] on cosmo
  Type "help", "copyright", "credits" or "license" for more   information.
  >>:

The technical reason for choosing Python 3.6 no longer applies due to how quickly cosmo has advanced in the past months (in particular thread support). jart and ahgamut were the ones who made it happen, I am only a python.com user.

But if you want Django 4 and Python 3.11, here is some info copied from the website:

3. Newer Python and newer Django? It is possible to have a djngo.com with Django 4 and Python 3.11. Solved by aghgamut who was the first to port Python to Cosmopolitan. Please note that this Python and the one that djngo.com use are different, the later has been heavily optimized by jart to run well on cosmo, the former is as close as upstream as you can get. Choose your destiny.

curl -O https://djngo.com/djngo4-py311.com

Author here. In 2007 when Django was starting to get popular there was a website called instantdjango.com where you could go and download a zip that would let you run Django and Python on Windows, with everything configured nicely.

I only used that version because it is the official one supported by cosmopolitan, but it is also possible to use newer versions with the same approach, for example Python 3.11 and Django 4: https://github.com/ahgamut/cpython/tree/cosmo-djngo (Website has been updated to point at that one).

Is the author stuffing the book contents in SQLite on load time or fetching them from a file?

I have tried to teach everyone in my family to use Google with special keywords instead of natural language, for maybe 15 years.

Much like how people who learned using books or articles diss on the people that learn using Youtube tutorials, LLMs via chat prompts and the iterative process may be better for the 90% of non pro users / developers out there even if we don't get it.

On number 2, even if you are John Carmack you may have trouble getting the right people on the phone.

https://twitter.com/id_aa_carmack/status/1305967411749892098...

Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...

I do run nix on the macos side, for graphics apps like a browser I would not recommend. But for command line apps like interpreters, compilers and vim-like editors it is really useful imho.

I have been using this repo as my daily driver for a month:

* The author is quite responsive to personal messages or issues / pull requests. * The people who fork it and change a thing or two are champions, they play with adding rust compilation to the kernel, suspend, gpu drivers all before it is added to this repo. * Once you get to just NixOS, and you tweak your editor, battery monitor, etc. It is pretty hard to switch back. Nix is hard to get right, but when you get it right, it stays that way. * I still boot back to OSX every two days when I have to share screen on Microsoft Teams or do a work thing. Takes two minutes to switch to the dark side and then a few days to remember to go back to Linux. Super polished and compatible dev machine vs fast linux on M1 with sane development environment is a great combo.

There is a point between AI (chatgpt, stable diffusion) and regular software based controllers (PID on an 8bit microcontroller).

This point is Machine Learning and for literature it may be indistinguishable from AI but can be just a Linear Regression / Random Forest / Basic Neural Network / Transformer implemented in 1000 lines of C or Python. The main difference in my opinion is the options we have to train those models [0] and be confident we can trim it enough to fit on the target device / get upgrades.

I think what the author should refer to is this standardized way of collecting data, running the models and shipping the weights for low power inference.

[0] https://cloud.google.com/bigquery-ml/docs/reference/standard...

Here is an example of a single file Django project I started around eight years ago that does what you mention:

https://github.com/planetfederal/registry/blob/149a2b958dd05...

Back in 2004? they would ship CD's to Colombia, even if you put like 20 in the form you would get them a few months later and could share with friends.

Those were the days, we laughed and laughed.

Just that it is built in I guess, and that cosmo libc places special attention to allowing you to use gdb/strace/memzoom and other debugging tools.

Another benefit of cosmo compiled apps is debugging. I won't get into pledge/unveil but check this out:

On my compiled python.com I try to open an app with a socket already in use:

./o/third_party/python/python.com run.py --strace

  SYS  15995        443'293'703 read(3, [u"egister(self, selectors.EVENT_READ)◙◙     "...], 8'192) → 8'192
  SYS  15995        443'440'589 close(3) → 0
    self.socket.bind(self.server_address)
  SYS  15995        443'471'588 write(2, u"      self.socket.bind(self.server_address"..., 42) → 42
  OSError: [Errno 98] EADDRINUSE/98/Address already in use
  SYS  15995        443'523'517 write(2, u"OSError: [Errno 98] EADDRINUSE/98  /Addres"..., 57) → 57
  SYS  15995        443'533'391 getenv("PYTHONINSPECT") → NULL

It was mentioned in the discord that instead of doing `curl something | sh` you could do `curl something | ./pledge.com sh` that looks like an interesting way to see what all those install scripts really want to get access to

Reading your writings since the times where you had to manually merge newadmin and geodjango svn branches. Lots of love from south america.

For anyone confused, this is not an "Open Source Cloud" replacement, but and "Open Source" "Cloud Replacement". A way to have the vacuum robot become cloud-free.

I saw a project get themselves in that position by using a lot of AWS services. At some point it seems they had a dev environment but it quickly faded.

Their solution was to have an anonymized copy of prod to a few (like 3 for 20 engineers) environment where features could be tested. Engineers usually paired in order use them concurrently for unrelated features and they were very expensive.

I did not live enough to see the on-demand environments arrive - devops was always busy with operations tickets to continue building those.

"Battling CodeWarrior": back when N95 was exciting to mention.

Vlad, I was using openlayers when leaflet was released and have been following your career. Big fan, thanks for always pushing the limits.

Lots of love from Colombia, to you and your family.

My question is: How have you approached communicating to your kids the current situation? I have been trying to explain to mine what is going on and it is really hard to maintain a balance between educating them and scaring them.

I worked on a J2ME app that needed to access location on phones without GPS, I was in touch with the operator and got a list of cell ids with lat, lon.

With that I created a daemon in Symbian that would query the cell id and open up a socket server to give it to the J2ME app.

With that we developed an app to request a taxi service, but none of the taxi companies wanted it. Some of the complaints were:

1. GPRS data plans were too expensive for end users 2. No one would put an expensive Sony Ericsson touch phone on a windshield, the windshield would be broken by thieves to steal it. 3. Looking at a screen on the windshield while driving was never going to be approved by authorities.

and number one in weirdo NFT marketplaces like Hic et Nunc

sel4 has a Haskell version that they write before the C (granted, not sure how a ghc generated binary would be verified) but would the problem be tractable in that case?

Last time I checked, they have a stripped down Android that would run a custom Termux and a single app with UI and different threads. Some of the services in Python, some of them using C, all communicated using capnp over zeromq

This is a big deal, congratulations on the milestone. Really looking forward to trying out Reflex FRP / Obelisk from IPFS, in particular the focus on making the source available because fetching Nix packages from semi-public caches and not being able to rebuild them is something that has bothered me for a while.

La olimpiada iberoamericana de matematicas en Caracas en el anio 2000 tenia una botella de Klein como logo.

Este post me trajo muy buenos recuerdos.

This is true. I am a NixOS fan, but when a piece of software asks me to use a Nix binary cache it can mask problems like a private git repo or other issues that would prevent a full build without me knowing. Worse yet, a Hydra box eating it's own dogfood ...