cbkeller

Sure, but that's a caveat that applies to testing in general, not something somehow special to composability. Bugs that only appear in edge cases or give wrong answers are always a concern when writing tests, that's nothing new.

I don't actually even see any evidence here that these sorts of bugs are more common or more insidious than any other sort of bug, rather it looks to me that since dispatch-based composability is a bit of a new phenomenon, people haven't always been on the lookout for these bugs yet, or learned the patterns of where and what to test for to find them -- but once you've seen a few cases, patterns start appearing and it becomes more clear what you need to test, just like anything else.

The broader issue to me is that I think people often underestimate the degree to which Julia is a fundamentally different programming paradigm than any other language most people have used before (unless you've spent a lot of time in maybe CLOS or Dylan) -- there's not even an agreed-upon name for it, but I'd call it "dispatch-oriented programming". Often times people will come in to Julia expecting all their expertise and intuition from class-based OO or whatever other paradigm to just effortlessly transfer to Julia, and that's really not the case, which tends to lead to frustration.

The answer as far as I can see is to write integration tests when you want to use composability (and/or just check that the packages in question already have integration tests against each other -- increasingly many do). It's not especially hard or anything, but you need to know to do it.

Oh, nice!

For it to silently fail of course though, he would have had to explicitly used the OffsetArrays package and explicitly switched all `Array`s to `OffsetArray`s (which hopefully you would notice) -- and then you would have to go ahead and use those OffsetArrays in a package which doesn't support them; if you just go ahead use 0 as an index in plain Julia code it will error as you would expect.

Personally I'd probably categorize most unit tests as verifying correctness (but only for the scenarios tested); integration tests may be more useful for finding incidental issues that you wouldn't have thought to test for directly. I'm for sure on board with dedicating more resources to testing -- and in my case as an academic, this is something I only have really been exposed to as a result of interacting with the Julia community.

Matlab is pretty mature at this point, but I'm sure it's had its share of bugs over the years as well (especially if you also counted the file exchange, which is probably the closest thing they have to an open source package ecosystem); it would be interesting to compare the two at a similar level of maturity / development person-hours if quantitative data could be found.

I think it should be fine for performance AFAIU to use `eachindex` instead; at least I know `eachindex` plays nicely with LoopVectorization.jl with no performance costs there.

That said, I think you're exactly right that people may wonder just this and use the seemingly "lower-level" form out of concern with or without testing it.

I actually wouldn't be surprised if the total number of tests run in the Julia ecosystem wasn't too different (thousands of packages with typically hundreds to thousands of unit tests, run on every commit and PR) -- virtually every Julia package has CI set up (at least standalone unit tests, though many packages could use more integration tests). Of course, in neither Matlab nor Julia do tests guarantee correctness.

I don't see anything problematic in what Viral said here; I think it would be fair to say your initial take ("Julia has been the future of machine learning for 10 years and will stay as the future of machine learning for the next 10 years") is likely to be perceived as at least somewhat inflammatory, a defensive response is natural enough in that context.

This seems hard to evaluate without a quantitative comparison to the abundance of bugs in the package ecosystems of other languages at the same age. So, for instance, how many correctness bugs existed (or, alternatively, had been found and fixed) in the Python ecosystem when Python was ten years old? The author makes a subjective claim, but from the few other languages they mention it seems they are comparing primarily to older and more stable ecosystems.

For an array, you'd have to worry about row- vs column-major orientation if multidimensional, but for simple numeric vectors and base strings (which are just a collection of UInt8s in memory), it appears to be sufficient to merely pass the pointer to C:

  julia> a = "hello there!"
  "hello there!"

  julia> p = pointer(a)
  Ptr{UInt8} @0x000000010a431458

  julia> ccall(:puts, Int, (Ptr{UInt8},), p)
  hello there!
  10

For vectors of structs, then of course you'd need to know the layout of each struct when operating on them from the other language, but still doable enough in principle. Vectors of union types would be trickier though.

Yeah, not frowned upon at all. StaticCompiler.jl [1] has made huge strides in the past few weeks actually thanks to Mason and Valentin (swapping in GPUCompiler as the backend); you can even compile to a tiny standalone binary without linking to the runtime if you're willing to use some tricks e.g. [2] to avoid GC allocations (stack allocations and manual heap allocations are both fine):

    # This is all StaticCompiler-friendly
    using StaticTools

    function print_args(argc::Int, argv::Ptr{Ptr{UInt8}})
        # c"..." lets you construct statically-sized, stack allocated `StaticString`s
        # We also have m"..." and MallocString if you want the same thing but on the heap
        printf(c"Argument count is %d:\n", argc)
        for i=1:argc
            # iᵗʰ input argument string
            pᵢ = unsafe_load(argv, i) # Get pointer
            strᵢ = MallocString(pᵢ) # Can wrap to get high-level interface
            println(strᵢ)
            # No need to `free` since we didn't allocate this memory
        end
        println(c"That was fun, see you next time!")
        return 0
    end

    # Compile executable
    using StaticCompiler # `] add https://github.com/tshort/StaticCompiler.jl` to get latest master
    filepath = compile_executable(print_args, (Int64, Ptr{Ptr{UInt8}}), "./")

yielding:

    shell> ./print_args 1 2 3 4 5.0 foo
    Argument count is 7:
    ./print_args
    1
    2
    3
    4
    5.0
    foo
    That was fun, see you next time!

    shell> hyperfine './print_args hello there'
    Benchmark 1: ./print_args hello there
      Time (mean ± σ):       2.2 ms ±   0.5 ms    [User: 0.8 ms, System: 0.0 ms]
      Range (min … max):     1.5 ms …   5.5 ms    564 runs

      Warning: Command took less than 5 ms to complete. Results might be inaccurate.

    shell> ls -lh print_args
      -rwxr-xr-x  1 user  staff   8.5K Feb 10 02:36 print_args

GC allocations are allowed if you use instead the approach in this PR [3], but we haven't wrangled that approach to produce standalone binaries yet.

[1] https://github.com/tshort/StaticCompiler.jl

[2] https://github.com/brenhinkeller/StaticTools.jl

[3] https://github.com/tshort/StaticCompiler.jl/pull/58

And the "how" behind Octavian.jl is basically LoopVectorization.jl [1], which helps make optimal use of your CPU's SIMD instructions.

Currently there can some nontrivial compilation latency with this approach, but since LV ultimately emits custom LLVM it's actually perfectly compatible with StaticCompiler.jl [2] following Mason's rewrite, so stay tuned on that front.

[1] https://github.com/JuliaSIMD/LoopVectorization.jl

[2] https://github.com/tshort/StaticCompiler.jl

That sounds about like my experience too, only s/physics/geology/ :)

Hmm, I would be interested to see where exactly it is that these Julia advocates have claimed it to be "the fastest language"

Ah, so there are many things that it is hard to be absolutely certain of in geology, but changing Earth's orbit is at least very very hard; even the kinetic energy from things like the Chixulub impact are far too small to have a significant effect. The "moon-forming impact" in the most common model of the origin of the moon might be more on the right order of magnitude, but there don't seem to have been any of those more recently than about 4.51 Ga. An astronomer could say more, but solar luminosity is also relatively well understood from studying other main-sequence stars of various ages.

The most common solutions involve high concentrations of organic greenhouse gases like methane as well as high CO2, but it's always possible there are other possibilities that have not yet been considered.

Ah, good questions. Although we are missing a lot of rock, paleontologists have spent a lot of time looking through the non-missing parts. So far, they have found a lot of really interesting things, such as the macroscopic soft-bodied fossils of the Ediacaran biota [1], but even so there's really very little in the way of shelly fossils until the Cambrian, and their appearance really does seem to be quite sudden as far as we can tell!

[1] https://en.wikipedia.org/wiki/Ediacaran_biota

Erosion, and for that wet-based ice would be key, so thickness is quite important.

Yes! There are a number of estimates on this from icesheet modelling; check out for example Fig. 5 in Donnadieu et al. 2003 [1]

[1] https://doi.org/10.1016/S0012-821X(02)01152-4

I'm not aware of any time when that was a mainstream belief, but there might be a gap in my history-of-science knowledge.

In any case, while we're currently in an "interglacial" period with respect to the Pleistocene glacial cycles, we're actually still in an "icehouse" period by the big-picture definition, since there are continental icesheets on Greenland and Antarctica. By contrast, a lot of the rest of the Phanerozoic has been characterized by "hothouse" climates, where there are no major continental icesheets anywhere on Earth.

It's possible that anthropogenic CO2 emissions will prevent us from going back into a colder "glacial" period when we otherwise next would have ~80 kyr from now, but geologic time is long and there is a lot of it still in front of us.

I would love it, but no one has quite managed it yet! There are simulations of, e.g., plate motions, but even those get quite uncertain if you go back more than about a billion years.

Forward models of the flow of the atmosphere, oceans, or even the mantle all exist, but with harsh tradeoffs between resolution and how much geologic time you can simulate. For this reason, we have coupled atmosphere-ocean models, but not really atmosphere-ocean-mantle models (because the timescale of mantle convection is just too much slower than the other two). I would really love to see more integrated whole-earth-system modelling, and that's probably something I'll try to work on in the future, but we may have a long way to go!

That certainly doesn't help either! To some extent though, anthropogenic emissions are dangerous more for their rate than their absolute magnitude; in the long run, once we stop emitting, silicate weathering will take back over "soon enough" -- it's just that "soon enough" in this case means ~5 myr and probably a mass extinction later.

The other one I forgot to mention is that the sun is a bit brighter now than it was 700 Myr ago (by perhaps a few percent). Go back another two or three billion years to the Archean and the difference would have been bigger -- to the point that we have some trouble explaining why there weren't a lot more snowballs back then [1]

[1] https://en.wikipedia.org/wiki/Faint_young_Sun_paradox

Older ocean crust does indeed generally subduct at a steeper angle! Relatedly, if you have ocean crust that is relatively young and hot enough but is still colliding with continental crust, then you can get "flat-slab" subduction [1]

Once the crust gets deeper in the mantle, eventually metamorphic phase transitions become the most important factor. In some seismic images, it looks like subducted slabs make it all the way to the core-mantle boundary (where they may contribute to the source of new mantle plumes).

For sure, I would love to have that simulation!

[1] https://en.wikipedia.org/wiki/Flat_slab_subduction

That's great to hear, thank you!

We are still in an icehouse period, yes! However, a few things may make it a bit harder for that to develop into a full snowball, currently:

1) While having continents at the poles makes it easier to have icesheets at all, it also makes it harder for them to grow into a full snowball. This is because

a) covering the continents with ice shuts down silicate weathering (and silicate weathering consumes CO2, so that's a stabilizing negative feedback)

b) the difference in albedo between water and sea ice is greater than the difference in albedo between land and ice. So if you can get cold enough to start making sea ice at the poles, you should get a stronger positive feedback of cooling -> higher albedo -> more cooling

During the Neoproterozoic, most or all of the continents seem to have been near the equator, so silicate weathering could keep going until sea ice reached the "point of no return" of the sea ice-albedo feedback [e.g. 1]

2) The biosphere is pretty different today than it was last time we had a snowball, and there is some reason to think that evolutionary developments like land plants and pelagic calcifiers may make the climate system more stable than it was 700 million years ago.

None of that is to say it's impossible though! The solid earth acts slowly, but it's a big lever, so hypothetically if you could somehow crank silicate weathering up high enough and volcanic degassing down low enough, you could probably still in principle reach the tipping point again.

For your last question, you are probably thinking of Milankovitch cycles [2] -- those are definitely going strong as well, though generally not strong enough to get us into or out of a snowball state.

[1] https://doi.org/10.5194/cp-8-2079-2012

[2] https://en.wikipedia.org/wiki/Milankovitch_cycles

Yeah, the oldest ocean crust on Earth today is only about 180 million years old, and the older it gets the colder and denser (and thus more prone to subduct) it gets. And when ocean crust subducts it generally (with some caveats) takes the sediments that have accumulated on top with it.

So we still have ocean crust from when T. rex lived, for instance, but none from when trilobites lived, and definitely none from the time of these glaciations.

To start with the last part

> How much rock would have to get scraped and pulverized into future magma to wipe out all evidence of a civilization?

it would take a lot. Even in the snowballs, when we had ice on every continent, there are still plenty of basins (mostly at the continental margins) where syn-glacial sediments are preserved; that is in part how we know the glaciations happened. While we have maybe 1/5th as much sedimentary rock volume per unit time prior to the end of the unconformity, that still leaves a lot!

At the time of the Cryogenian, both the fossil record and DNA-based molecular clocks suggest we didn't have multicellular animal life until after at least the first (Sturtian) glaciation. And we're talking basically just sponges (porifera) at first.

Of course, it's not impossible we could have another snowball in the far future (probably unlikely for several reasons, but never say never), and the question of what that would do to the record of modern human civilization is an interesting one. The short answer is "I don't know", but I think it would be hard to erase all traces without something a good bit more severe than the erosion that produced the Great Unconformity.

Short answer: it's complicated, and not a binary, but perhaps a gradual process of "becoming less of a mystery". The history of the subject is actually pretty interesting, and like most things in science gets more nuanced the more time you spend on it.

Ah, so while there were ice sheets on every continent during these glaciations, there are actually quite a lot of places where sedimentary rocks survived the glaciations - perhaps as much as a fifth of the area of the continents (back-of-the-envelope, given that there is about a fifth as much preserved sedimentary rock per unit time prior to the end of the unconformity).

Generally a lot of these regions where rocks from the "missing" interval are well-preserved are at the margins of the (paleo)continents. This is for two reasons, as far as we can tell: (1) erosion by continental icesheets is more hit-or-miss near the margins; "hit" if you're in an outlet ice stream, but "miss" if you're not, since marginal ice tends to be "cold-based" and generally not very erosive at the margins outside of the outlet ice streams; (2) the paleo-continental margins are where all the tectonic activity was at the time -- and while tectonic uplift won't help any, tectonic subsidence can help a lot if it makes a basin subside below sea level (which will protect against erosion).

Ah yes, quite right on the first part! The catch is that to be preserved, those sediments have to be deposited somewhere else on the continental crust, say in epicratonic seas [1] or in subsiding basins [2]. If the sediments wash all the way off the continental shelf and onto the ocean crust, then they'll ultimately get subducted into the mantle!

The catch with glaciers is that putting a lot of big glacial ice sheets onto the continents takes that water out of the oceans, and lowers sea level -- meaning a bunch of places where you could normally preserve sediments on the continents will be above sea level during the glaciations (AKA above "base level"), and more of those sediments will be washed off the continents entirely. There still some places where you can find the fossilized glacial till from these Cryogenian glaciations (which is in part how we know they happened), but it's basically only in the tectonic basins at the margins of the continent that were subsiding fast enough to stay below base level and not get eroded away.

[1] https://en.wikipedia.org/wiki/Inland_sea_(geology)

[2] https://en.wikipedia.org/wiki/Sedimentary_basin

Ground up and washed into the ocean -- and then ultimately subducted and made into new magmas!

Great question -- and yes!

Pleistocene glaciation in the northern hemisphere has been a lot shorter (so far) than the Cryogenian glaciations, but it is probably not a coincidence that the outline of Canada's "Precambrian shield" basically matches the outline of the Laurentide ice sheet (check out Figure 5 of the 2019 paper [1]). We're probably only talking about scraping off no more than a couple hundred meters of sedimentary rock formerly covering the shield, but that's about what you'd expect.

More broadly, as one author noted long before us [2] it turns out that most of the places on Earth where there is a lot of Precambrian crystalline basement exposed at the surface today (i.e., where later sedimentary rocks have been scraped off, one way or another) were glaciated either recently or in the Late Paleozoic Ice Age [3], which hit much of Gondwana (see also Fig S16 here [4]).

More recently, another group of researchers using thermochronology in Antarctica [5] found evidence of several kilometers of exhumation during the Late Paleozoic Ice Age, as well as perhaps 1-2 km during the last ~35 Myr of Cenozoic glaciation (n.b., Antarctica has been glaciated for a good bit longer than we've been having ice ages in the northern hemisphere).

[1] https://www.pnas.org/content/116/4/1136/tab-figures-data and see also Fig S16 in [4]

[2] https://pubs.geoscienceworld.org/gsa/gsabulletin/article/83/...

[3] https://en.wikipedia.org/wiki/Late_Paleozoic_icehouse

[4] https://www.pnas.org/content/pnas/suppl/2018/12/26/180435011...

[5] https://doi.org/10.1016/j.epsl.2018.10.044 (see especially Fig. 7)