Here's a writeup on a few approaches: http://slides.com/shirleywu/deck#/
278 karma · joined July 27, 2012
The CLI is really really good for a CLI. Wagon is also good, my coworkers have been using it to pull data and like it a lot.
Overall love postgres, it's been 6 months (since starting a new job) and I'll never go back.
The language for building workflows is pretty half-baked, and there have been times when I've wondered whether RQ or raw message passing would be better in my situation. But it's better than any of the alternatives that I've had direct experience with.
The main benefit as I see is when data sources are external, with ill-defined or ambiguous schemas. Often when you fit data into an ETL pipeline, you find out issues at the output of the pipeline, but the fixes need to happen way upstream. Often this involves rebooting the entire process entirely and renormalizing all your data somehow.
If you delay interpretation and normalization to later in the processing pipeline (i.e. in the system I worked on, we did it lazily at interpretation time), then doing smarter things with the data is a matter of changing code -- and it's a lot easier to ship fixes to code than to ship fixes to data!
Was surprised when I read that. He also ascribed some of the misperception to a young urban journalist echo chamber effect.
From what I can remember, a lot of the slowness came from GPG, and specifically files being compressed before encryption. Disabling compression speeds things up, but trades off disk space (and security).
I wonder what it'd take to reengineer the thing to take advantage of multicore -- being CPU-bound on a single core is I think what makes it slow.
I once had to extend it for a distributed setting (think mapreduce) where you can cheaply iterate all records but not sequentially. Instead you have multiple reservoirs, and you can't resample those uniformly since each saw a different number of records.
The unbiased way to aggregate 2 reservoirs is to draw a random number from a hypergeometric distribution. You aggregate a larger number by combining 2 at a time.
Fun story, an interviewer at Google once asked me to derive reservoir sampling a long time ago, which I completely wiffed.
So lately, for my own projects, it's gotta be dpkg-based with an active upstream repo.
What I like about this article is it's talking about what the global optimum is -- what's the best approach for the entire ecosystem? Not just for one individual or one org. What's Pareto optimal? (Or maybe it's a little less general -- what's the best approach for those who depend on FOSS, and don't pay someone like RedHat for long-term support and CYA insurance).
And basically I agree. The world's upgrade cycles could stand to be a whole lot shorter than they are. Every time I think about all the hardware in this world that'll spend its whole lifetime vulnerable to heartbleed, or shellshock, or a gazillion smaller vulns, my skin crawls.
It was on discussed on HN, at the time noone pointed out anything glaring about the advice given.
It looks like process-private storage is one of its features.
It's been too long since I used it to remember the details, but I believe process-private keys are one of this API's features.
There's a big issue with the unordered_map.erase(const_iterator) method -- STL mandates that .erase() returns an iterator to the next item in the container. There's no simple implementation of that for hashtables other than to walk the hashtable, which is quite slow.
This was once a serious performance issue with unordered_map (discussion here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41975). I haven't followed up in years, but is the issue?
I started working on a software PLL recently -- not analog like this one, digital. Specifically, I was trying to work out the math on a fully event driven type-II PLL -- i.e. instead of updating at a fixed sampling rate, it'd wake only when an edge happened on the input or VCO. I was starting to get curious whether there's a solution for that somewhere on the web.
This is a good intro to (hardware) PLLs in general: https://www.youtube.com/watch?v=0jzLDe950AY followed by http://www.ece.tamu.edu/~spalermo/ecen620.html
CPython's behavior is nice, but it seems to me it came about by accident. Big heavy resources use refcounting because everything uses refcounting. Plus, if CPython had true concurrency, across-the-board refcounting probably wouldn't have lasted nearly as long (reason being: multithreaded refcounting requires atomic ops which make them much more expensive).
But in the general language design world, IMHO, refcounting is actually a pretty good compromise for resource management. Because:
1) The efficiency lost compared to Rust's approach is probably immaterial (since the Big Expensive Object that you're mananaging dwarfs the refcounting -- syscalls and file descriptors are expensive). So I don't think zero overhead is critical.
2) Cycles are pretty easy to avoid when you're only refcounting Big Expensive Objects (FDs, mmapped buffers, etc. etc.) For a general runtime, refcounting is tricky. But if it's a special part of the language that's small, simple, and only for resource management, it's pretty convenient. I think it's worlds better than the approach of mixing GC and resource management -- language designers should admit that's a terrible idea.
The laws were always on the books but rarely enforced. Only once the UBS thing broke did the IRS start cracking down. Then the OVDI happened, which presumes guilt (ie intentionally evading taxes) from the get go. So based on the timing, I always thought things got to the point they are because of swiss bank accounts and tax evaders. And I thought penalties were structured the way they were so the IRS could throw the book at those guys when they were caught.
The bank account held five figures. The cost to come into compliance was low five figures. The potential penalties were easily high six figures. It was much larger than the either (a) my total earnings from 4 years abroad or (b) the maximum amount the account had ever held. I believe my tax guy was on the cheap end too.
I'm still angry about it; I feel like I was guilty of nothing more than ignorance. IMHO this is case of strict policies that were intended for rich tax evaders. And they're being retargetted at millions of expats & foreign nationals.
Even if the CGI script is in some other language, think of how many .sh wrapper scripts there are out there.
EDIT: http://lcamtuf.blogspot.com/2014/09/quick-notes-about-bash-b... explains it better
Though Amazon might've already locked that market up. Why rent when I can buy for $0.01 + $3.99 shipping?
One of the key activities in programming is reasoning about time and space cost, and this is a space optimization that's nondeterministic. It kicks in sometimes, or sometimes not at all, and happens behind the scenes at garbage collection time when it's nearly invisible. If you're sloppy, your program may have a huge asymptotic space usage, and this may paper over it. But the impl has heuristics, it may not work all the time -- even their example program needed Thread.sleep() calls! Unpredictable semantics help nobody. So I always liked explicit string interning (whatup, Lisp!).
All the same, faster is better and I'm sure this makes things faster.
Oh and can we talk about how broken it was that Java 6 and under had a fixed size pool for .intern()'d strings?