RedCrowbar

Looks like it's going to be available in Firefox 58 behind a pref: https://bugzilla.mozilla.org/show_bug.cgi?id=1415481

Łukasz Langa recently gave a PyCon talk [1] on the subject.

[1] https://www.youtube.com/watch?v=7m5JA3XaZ4k

Toronto is facing the same problem. The signal upgrades to our (puny in comparison) subway have been in the works for the last decade with no end in sight.

A ~20% improvement in throughput and latency while using 50% less memory (which could allow more workers per-box) is not a "marginal" improvement in my book.

I installed the 64-bit version, you might be using 32-bit.

My figures exclude shared memory and are calculated as VmRss - Shared from /proc/<pid>/statm.

If you are using Windows 10, your missing Atom processes will be under Background Processes in Task Manager. For the sake of the argument, I just did a fresh install of Atom and this is what I see on the first run: https://i.imgur.com/0ZRSumF.png. ~220MiB (no files open, zero extensions).

Each Chrome (and Electron) app instance is a group of processes. You are probably just looking at the main process, while the bulk of RAM/CPU use comes from the renderer processes.

Here are stats on my (Linux) box:

* atom - ~500MiB (one window)

* slack - ~816MiB

* chrome - ~935MiB (two tabs + hangouts)

* google music electron app - ~500MiB

There's​ an excellent analysis of the Uber's post here: http://use-the-index-luke.com/blog/2016-07-29/on-ubers-choic...

If you are porting from 2.7 and don't require compatibility with earlier Python versions, then "future" [1] is a better, cleaner alternative to six.

[1] https://pypi.python.org/pypi/future

Yes, async comprehensions are concurrent like all other async mechanisms in Python 3 are. But you are likely thinking about parallelism, not concurrency, in which case the answer is also yes, if you use thread or process executors for tasks.

While the claim of selection bias in JetBrains' survey may be argued, it is still a valid data source, which is what the gp was asking for.

Also, your anecdotal data is arguably biased as well.

My take is that many sources, including the ones linked to in the tweet's replies point to solid growth in Python 3 adoption. Python3 might not have overtaken Python 2 overall, but it's very far from being "dead".

> `OrderedDict` is now just a thin wrapper around `dict`.

It is not. odictobject.c has not been touched at all yet. Also, OrderedDict defines several methods that ordinary dict does not (and unlikely ever will).

No, it's on by default in v1.5.

Worth mentioning that in this release TextMate Python bundle was replaced with MagicPython [1] for much improved Python code tokenization. [2]

[1] https://github.com/MagicStack/MagicPython [2] https://github.com/Microsoft/vscode/commit/f4967722fa02471da...

A pure-Python implementation would probably never reach this level of performance, even under PyPy. Cython really makes it possible to basically drop down to C and write very efficient code.

That said, once PyPy supports 3.5, we will consider making a CFFI binding.

In the 1M benchmark roughly half is spent decoding rows (in [1]). The rest is spread over interpreter overhead, GC in particular.

[1] https://github.com/MagicStack/asyncpg/blob/master/asyncpg/pr...

12 in the particular case of 1M rows/s. The details are available in the benchmark report: http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python...

Haven't tried, but, AFAIK, redshift speaks the standard PostgreSQL protocol, so asyncpg should work.

> I guess I am frustrated that you folks put all this work into a great driver and then locked it in the largely useless dungeon of asyncio, where all the libraries are totally useless to those of us who don't think node.js is the programming model for all cases

I see your point. Nothing is really "locked away". The protocol parser and data codecs in asyncpg are independent from the I/O model and loop semantics. With effort, it's possible to build a sync interface on top. It's just not a priority, as our cases are async.

> that's great, you've written a very fast driver. My benchmarks are comparing psycopg2 to itself,

OK, I ran the asyncio suite (with uvloop event loop):

Python3.5.2 asyncio+psycopg+uvloop -- avg 8356.84 recs/sec

Really not that slower. There is also an important detail that you forget to mention: the threaded benchmark uses notably more CPU time (about 30% more).

> Everytime I talk about my blog post, I am super careful to qualify: in this case, "usually has little to do w/ speed" and other qualifications. The blog post as well.

I created an asyncpg suite for your bigdata benchmark (https://bitbucket.org/zzzeek/bigdata). Here are the results obtained from running it on my laptop against local PostgreSQL 9.5.

300 connections:

    Python3.5.2 threads -- avg 9798.16 recs/sec
    Python3.5.2 gevent -- avg 9677.84 recs/sec
    Python3.5.2 uvloop/asyncpg -- avg 11558.24 r/sec

The modified suite is here: https://github.com/elprans/bigdata

> I have no intention to stifle anything, I'm only raising the point that this seems like a really new driver and it's likely that it would take a while before it approaches the stability of psycopg2. That is, the stability of psycogp2 is not to be so lightly thrown in the trash.

We in no way question the merits and the stability if psycopg2. We created asyncpg to solve the challenges we face in the development of EdgeDB, it's not just a toy project for fun.

> Does your driver seamlessly handle reading and writing of three-dimensional arrays of JSONB structures, and arrays of custom enumerated types ?

Yes it does.

> edit:

> > but efficient implementation of the binary type I/O and the protocol itself.

> that's great too! you don't need async for that either.

> I guess I am frustrated that you folks put all this work into a great driver and then locked it in the largely useless dungeon of asyncio, where all the libraries are totally useless to those of us who don't think node.js is the programming model for all cases.

I'm sorry to hear that. Nobody forces you to use asyncio or asyncpg. But please don't try to claim that something is a "useless dungeon" simply because you disagree.

> the 1M row/sec part is easy to explain:

Prepared statements were used in all benchmarks where supported. Please don't dismiss the results so easily. The source of performance increases here is not the use of prepared statements, but efficient implementation of the binary type I/O and the protocol itself.

> I'd be very wary of a new PG driver especially one that implements the wire protocol itself. > pg8000 does this, and also does prepared statements, and while it doesn't have the speed approach here, it also has serious problems supporting all the features and datatypes that psycopg2 does.

asyncpg supports 100% of pg_catalog types (we have a test for that.) Additionally, psycopg2 doesn't _really_ support all the features and datatypes and simply returns unparsed text in many cases. You have to write lots of typecasters to convince it otherwise.

> It's been around for years and reliably breaks on many Postgresql features (and Postgresql introduces new features like a firehose). Writing and maintaining a driver for Postgresql is not an easy task.

Please. Simply because something has been around is not the reason to stiffle new development and innovation.

Handmade with d3.js

There appears to be an adapter for Core already: https://github.com/CanopyTax/asyncpgsa

The protocol is documented in the official PostgreSQL documentation: https://www.postgresql.org/docs/current/static/protocol.html

SQLAlchemy ORM is not async, so there is no way currently to use any async dialect for it. However, it is possible to write an SQLAlchemy Core adapter for asyncpg.

> would be cool to also make jmh tests against jdbc and async pg on java.

Pull requests to https://github.com/MagicStack/pgbench are welcome :-)

We intentionally benchmarked simple queries to measure the overhead of the driver implementation. So, the 2x speedup really means 50% reduction in overhead.