Sparse vs. Dense Arrays in JavaScript
dmitripavlutin.com
dmitripavlutin.com
- Sparse: Most of the data is "missing"
- Dense: Most of the data is not "missing"
The key words being "most" rather than "any" or "none."
At any rate, I suppose it depends on the language and how you view data, but to my mind that's just how arrays work.
This is an issue because JavaScript Arrays are not like arrays in other languages - they are dictionaries where the keys happen to be numeric. So it is possible for a numeric key not to be defined, which is different from it being defined but having "undefined" as a value.
sparse arrays are usually called associative arrays, or maps
That sure seems like a feature, not a bug. I can remove entries from an array, and still iterate the array without having to recreate it - Sure, being aware that .length is incorrect is these cases is good. Likewise, it is good to know that if the nulls are meaningful, you need to use another technique. But that is all the more reason to appreciate that forEach, etc. skips the nulls, while a for loop on .length will process them. Choose whichever is appropriate.
[1,,3].length
// 3
[1,,3].forEach(v => console.log(v))
// 1
// 3
[1,undefined,null,3].forEach(v => console.log(v))
// 1
// undefined
// null
// 3 t0 = new Date();
a = Array(100000000).fill(0);
console.log("time elapsed: ", new Date() - t0); // 22912
a[100000] = 1;
a[10000000] = 1;
a.forEach(e => e == 1 && console.log(e));
console.log("time elapsed: ", new Date() - t0); // 28371
t1 = new Date();
a = Array(100000000);
a[100000] = 1;
a[10000000] = 1;
a.forEach(e => e == 1 && console.log(e));
console.log("time elapsed: ", new Date() - t1); // 2787
Upon doing a couple not-so-scientific runs of this, it looks like the latter isn't really faster than the former.Sorry, I don't quite understand: running through a sparse array - running through the "undefined" - is much faster than running through a dense array. What have I missed?
So what this code demonstrates is that if you have to look up a billion keys, that is going to be slower than if you have to look up two keys for an object.
Additional reading for this: https://v8.dev/blog/fast-properties From this article:
const sparseArray = []; sparseArray[9999] = 'foo'; // Creates an array with dictionary elements. In this example, allocating a full array with 10k entries would be rather wasteful. What happens instead is that V8 creates a dictionary where we store a key-value-descriptor triplets. The key in this case would be '9999' and the value 'foo' and the default descriptor is used. Given that we don't have a way to store descriptor details on the HiddenClass, V8 resorts to slow elements whenever you define an indexed properties with a custom descriptor:
You can show that as follows:
t0 = new Date();
a = Array(100000000).fill(0);
a[100000] = 1;
a[10000000] = 1;
counter = 0;
a.forEach(e => e == 1 ? console.log(e) : counter++);
console.log("elapsed: ", new Date() - t0); // 19415
console.log("visited but not logged: ", counter); // 99999998
t1 = new Date();
a = Array(100000000);
a[100000] = 1;
a[10000000] = 1;
counter = 0;
a.forEach(e => e == 1 ? console.log(e) : counter++);
console.log("elapsed: ", new Date() - t1); // 1936
console.log("visited but not logged: ", counter); // 0Please note this observation only applies to access during execution.
During the late 70s a developer named Paul Heckel discovered that hash maps (what JavaScript objects are) were randomly accessed in memory until the specified key was found. That random access was faster than the access of a specified index from an array, because array indexes are accessed sequentially. Because arrays are ordered sequentially, even in memory, they are substantially faster to iterate over, though.
http://documents.scribd.com/docs/10ro9oowpo1h81pgh1as.pdf
Interesting (and unexpected): I tried searching for "Paul Heckel" in Google and only found stuff about a jazz musician. Then I searched for "Paul Heckel's algorithm" and my writing about implementing the algoritm in JavaScript was one of the first Google results.
In my experience the common javascript engines are so accelerated that in practice, before anything happens to the array it is uninitialised and the engine uses the length parameter as a future hint. If the array is promptly filled contiguously it will perform as a dense array. The types which the array is filled with and subsequently maintains can have a greater impact on performance. An array that only contains integers performs faster than one containing doubles or strings, and once types are changed or mixed in the array it slows down, generally by a factor of 2 or 3.
Array(n).fill()
is a permanent deoptimization? And if so, is the new array returned by: Array(n).fill().map(x => x)
better optimized (after the, presumably, suboptimal iteration)?...also, from a reasoning approach - why would a competitively accelerated language engine create an array full of holes as soon as one is declared? Why would the engine not try to optimally structure the type as it is used? This is something Chromes engines and Firefoxes appear to have done, quite impressively since many years back.
Over 7 years as Node.js developer I have never needed to use them and I don't see use appealing use case.
An array becomes sparse when the datastructure doesn't try to manage things linearly, *not* when there's a placeholder "nothing" value stored in it.