The Misfortunes of a Trio of Mathematicians Using Computer Algebra Systems [pdf]
ams.org
ams.org
I asked about x^n + 1 = z^n, which is a simpler special case with y=1. The system no longer recognized it to be False. So the theorem was programmed as thoughtless pattern matching. I think one day computers might become authentic "creative" tools for mathematicians (as opposed to "computational" tools), but Mathematica's philosophy seems to be a dead end in this regard.
(Sadly, I can offer no data point, as I probably wouldn't have recognized either case...)
Regardless, I don't think that changes my point.
With the help of Mathematica, one of us found some counterexamples to our conjectures. Fortunately, another one of us was using Maple and, when checking those supposed counterexamples, found that they were not counterexamples at all. After revising our algorithms from scratch, we concluded that either the computations performed with Mathematica or the computations performed with Maple had to be wrong. Things started to become clear when the colleague using Mathematica also found some “counterexamples” to the above-mentioned result of Karlin and Szego for the case in (1) and, even more dramatically, his algorithm yielded different outputs given the same inputs. Our conclusion was that Mathematica was computing incorrectly.
I think that the use of closed-source software in any public research or scientific undertaking should probably be verboten.
A short preliminary article:
http://angrystatistician.blogspot.com/2014/08/learn-this-one...
Here's the Sage code:
https://github.com/octonion/puzzles/tree/master/product-sum
I'd be curious to see the speed of the corresponding Mathematica or Maple code.
10^8 22s 65s
10^9 225s 635s
10^11 14h20 16h05
However, I found that Sage seems to calculate divisors faster than Mathematica for large numbers. The following are the times to sum the divisors of one million numbers starting at 10^8, 10^9, 10^10, 10^11:
Sage: 18s, 21s, 24s, 27s
Mathematica: 12s, 14s, 28s, 49s
Solutions[pSum_, pProd_] := With[{term = 1 + pSumpProd}, With[{div = Divisors[term]}, Map[Function[x, If[Mod[x + 1, pProd] == 0, (1 + {x, term/x})/pProd, ## &[]]], Take[div, Ceiling[Length[div]/2]]]]]
WithNonOneValues[n_, nval_] := Last[Reap[If[nval == 0, Sow /@ Solutions[n - 2, 1], Module[{val = Array[2 &, nval], idx = 1}, While[idx <= nval, With[{pSum = Plus @@ val + n - nval - 2, pProd = Times @@ val}, If[pSum + 2val[[1]] < pProd*val[[1]]^2, If[(++idx) <= nval, val[[Range[idx]]] = val[[idx]] + 1], Map[Function[x, If[Min[x] >= val[[1]], Sow[Flatten[{x, val}], ## &[]]]], Solutions[pSum, pProd]]; idx = 1; val[[1]] = val[[1]] + 1]]]]]]]
AllSolutions[n_] := Flatten[Map[Function[x, WithNonOneValues[n, x]], Range[0, Log2[n]]], 2]
Here's the public SageMathCloud worksheet: https://cloud.sagemath.com/projects/49a2531d-9d02-42c9-9db6-...
I wish they would have supplied those matrices they printed out in machine-readable format so we could easily put those in Sage too.
I wrote the det code in Sage that you're calling there (though in Sage-6.4 it'll likely be replaced by a call to FLINT). It computes det(A) in a very interesting way, which is asymptotically massively faster than Mathematica. To compute det(A), choose a random vector v and solve Ax = v using a p-adic lifting algorithm (the one in https://cs.uwaterloo.ca/~astorjoh/iml.html). One can prove --using Cramer's rule--that the lcm of the denominators of the entries of x will then be a divisor d of det(A), and with high probability one expects that det(A)/d is a tiny integer. One can then provably (using the Hadamard bound) find det(A) by working modulo a few additional primes and using the Chinese Remainder theorem.
But we know from experience that one can often work around bugs in hardware in software (and/or microcode). So I wouldn't say "just" using open source software on top of proprietary/opaque harware gains you nothing.
Also, it's been a long while since using an open source operating system was a controversial choice -- in fact, I think it'd be hard to find a proprietary operating system that both supports good CAS, and runs on at least three different architectures...
Because science and the scientific method proceeds via the necessity of verification and because a lot of science is mathematics and because a lot of mathematics is nowadays done on a computer we need to be able to inspect these mathematical black boxes.
If you don't mind me promoting my own work, here it is: https://jyx.jyu.fi/dspace/handle/123456789/38495#
http://sagemath.blogspot.it/2014/08/you-dont-really-think-th...
Sage is absolutely amazing, but it is very far behind the cutting edge CAS systems (MAGMA/Mathematica) - I think he is being a bit harsh in his assessment, but he has incredibly ambitious goals - and by those goals, he has fallen short.
[1] http://en.wikipedia.org/wiki/Sage_(mathematics_software)#Sof...
According to openhub.net [https://www.openhub.net/p/sage/analyses/latest/languages_sum...], Sage has 1.7 million lines of code. Maxima alone has 700K lines, Singular has 640K lines, SciPy has 650K lines, SymPy has 480K lines, etc.
/tmp/maxima-5.34.1$ sloccount src
...
Total Physical Source Lines of Code (SLOC) = 129,584It's not your fault that economic resources in human societies are hugely misallocated, and that social progress is always painfully slow.
If you have a look at the Wolfram MathWorld article »Uniquely k-Colorable Graph« [2] you see the same wrong graph with the following information.
It is implemented in Mathematica as GraphData["UniquelyThreeColorableGraph"] and Uniquely3ColorableGraph in the Mathematica package Combinatorica` , in the latter case replacing the incorrect graph (from Harary 1994, cover and p. 139) present in earlier version.
So this bugfix did not really fix the bug. I reported it using the feedback form and we will see if the third attempt will get it right.
[1] http://www.wolframalpha.com/input/?i=Uniquely3ColorableGraph
[2] http://mathworld.wolfram.com/Uniquelyk-ColorableGraph.html
Any symbolic system should be treated with care, but they can of course be extremely useful. My typical use case is to have it compute very challenging symbolic expressions for me. I then treat it as a "very plausible hypothesis" which I then prove.
This whole article is really just telling people to use the scientific method in computing too: if you don't know how to reproduce a result it's not science. Fast computation which a human can't possibly do is something that's new from the last 30 years and most scientists still don't know how to deal with it.
It is very sad to see that statisticians have switched to R, biologists to python, computer scientists to the gnu tool chain, yet physics and maths seem to have been colonized by mathematica when you have a whole set of open source tools which are superior in every way: pari pg, gnu arbitrary precision libraries, axiom, maxima and sage is you want everything under one roof with a unified interface.
One of the major driving forces in the design of computer proof systems is that they (a) output proofs along with assertions that are (b) trivially verifiable. Indeed, verification is a major philosophical and practical force behind the theory of any type theory.
Ultimately, this is a major component of the design of a proof checker because while lots of its pieces can be complex and scary and potentially buggy, the proof verifier must always be tiny and easy and impeccable because it bears the entire burden of safety.
A similar argument cannot really be made for CAS, I think. Even open source ones.
(There's a catchy name for this principle, but I forget it. "Somebody's Something")
For some use cases at least, Mathematica is clearly better than anything else I have tried.
I feel like something like ipython notebooks with the right combination of libraries might eventually get there, but that is unfortunately still years away.
This is shown no where better in the paper than when they try to calculate Integrate[Exp[-pt](Sinh[t])ˆ3, {t, 0, Infinity}]. In a good cas, such as maxima, you will get no result and a ton of errors. Which is what you should get without specifying what sort of variable p is, is it a matrix, polynomial, group of some sort? That it's implicitly assumed to be a real number that might turn out to be complex under some circumstances isn't a feature, it's a bug.
What I did mean, was that for my use cases, I found Mathematica was superior to the alternatives. You are welcome to think this was because I was lazy or misguided, but it was definitely not because I liked not having to specify domains for my variables (and I do remember having to write Assuming[p>0 && x \in Reals, ...] and the like often, it definitely does not assume everything is a positive real).
One thing I used Mathematica a lot for was numerical integration of ODEs (that had been derived using the CAS part of Mathematica and had some pretty nasty coefficients). NDSolve in my experience was just better than competition. You can definitely get nonsense out of it, but with a modicum of care it works incredibly well.
What? This is a software bug. Open source software has bugs just as closed source software has bugs.
The average user of sage is not going to go hunting for a bug in sage's source, whether it's available or not, which is just the same as with Mathematica. Examining an unfamiliar code base will be of no help when you want to know why a certain integral was evaluated incorrectly.
> open source tools which are superior in every way
They are not superior. At best they are equivalent. What you list is a number of disparate tools that may or may not cooperate together well. What Mathematica provides by way of competition is a set of reasonably polished packages, all in one place.
Mathematica also makes it possible to quickly write a one-liner that solves a problem and lets you move along. This experience is simply not there for tools like sage. Sage's plotting (matplotlib, IIRC) is just not comparable.
The function call is not the issue. The issue is that it is a large unfamiliar code base, and the bug, as in this case, would not be an incorrect None somewhere, but will be a mathematical bug somewhere, like an incorrectly written formula. Chasing that is far far more difficult.
Bearing in mind also that the average phd mathematician has at best cursory knowledge of programming and software engineering, enough to get on with mathematics, expecting them to dive into sage and fix things is just unrealistic.
> it's largely trivial compared to the everyday work you do
I think this is also a very unrealistic expectation of users, however skilled they may be as mathematicians.
It's an unrealistic expectation to expect it from every single user, but not for many of them.
These two programs are a prime example of how open source is just as inaccessible as closed source. It's simply too difficult to learn a large complex codebase. People have their own jobs to get on with.
Who is "you"? In fact, very few people can debug a modern symbolic mathematics system. They are fiendishly complex.
Astronomy is one of many sciences which can never reproduce its results in the way you're talking about. Do you not consider astronomy to be a science?
In any case, this paper was looking for counter-examples. If found, then they would be all the evidence that's needed. The method to compute the counter-examples is nice, but irrelevant.
In any case, tell me about how you can predict gamma-ray bursts. When will the next detection of extrasolar neutrinos occur? How do we recreate the Big Bang?
I am not confused. I regard historical sciences like astronomy, geology, paleontology and archaeology equally part of science even though there isn't the high level of reproducibility of, say, most chemistry.
I also regard nuclear bomb physics to be a science, even though by law it's impossible to reproduce those tests.
Yours seems to be a very medieval mindset. Just because we are ignorant of the initial conditions of a system doesn't mean we are ignorant of the equations by which it evolves. And the example of chemistry is just bizarre. If we couldn't reproduce the same reaction down to the atom time and time again our silicone based infrastructure would have filed a very long time ago. Similarly for nuclear bomb tests, if they were truly irreproducible then things like [1] should happen a lot more often than not.
You claim that I have a medieval mindset. It seems you espouse a 19th century view of science, of the clockwork universe.
Science doesn't require reproducibility, nor your weaker requirement of "how to reproduce."
Predictability is the key to science, not reproducibility, though of course those are inexorably tied when it's possible to reproduce something. We can predict that fossils of a certain type will only be found in a specific layer of geological strata. We can predict that hurricanes will be created by and affected by certain wind patterns. We can predict that radioactive atoms will spontaneously decay.
Even though we certainly cannot reproduce those.
Chemistry is a field where it's easy to set up very similar conditions to previous experiments ("reproduce") and where the expected confidence of predictability is quite high. Hence my use of it as an example. It's much harder to reproduce an observation of a supernova, but we have pretty high confidence that when we do see one it will follow certain patterns.
The resources might be beyond the capabilities of humanity to ever achieve, but that detracts nothing from the point made. "Chaotic systems" aren't "unsolvable systems". The idea that they aren't reproducible to any desired degree of accuracy is laughable.
To me it means that no matter how precise you measure everything, at some point even the unpredictability of a single atomic decay is enough to make a difference such that one of the planets may is ejected. As far as I know, that's also the generally accepted meaning.
"In 1989, Jacques Laskar of the Bureau des Longitudes in Paris published the results of his numerical integration of the Solar System over 200 million years. These were not the full equations of motion, but rather averaged equations along the lines of those used by Laplace. Laskar's work showed that the Earth's orbit (as well as the orbits of all the inner planets) is chaotic and that an error as small as 15 metres in measuring the position of the Earth today would make it impossible to predict where the Earth would be in its orbit in just over 100 million years' time."
So yes, that's exactly what I said.
The observation you quoted isn't using the full equations of motion. The next study on that page uses a change of 1 meter and finds that 1% of the 2501 cases Mercury goes into a dangerous orbit, including one where "a subsequent decrease in Mercury’s eccentricity induces a transfer of angular momentum from the giant planets that destabilizes all the terrestrial planets ~3.34 Gyr from now, with possible collisions of Mercury, Mars or Venus with the Earth."
You assert, seemingly as a matter of faith, that it is possible to measure all of the relevant factors such that a prediction can be made. We can't predict when an atom of uranium will decay, but we can make statistical predictions about the population. We can't predict when an air molecule out of a mole of molecule will hit the side of a bottle but we can make predictions about the pressure.
Do you think that we can ever do either of those two cases?
It's the same for the Solar System. As far as we can tell, it's not possible to have accurate enough information to predict the evolution of the Solar System. Even with numerical simulations, the presence of a space craft, or an extra-solar meteorite, might change things after 10 million years - things that can't be predicted.
A secondary reason with closed systems in particular is that it may reveal methods it uses which is a competitive advantage to other systems.
I doubt this is the case. Mathematica at least doesn't really contain any "magic" in the actual algorithms and uses pretty well-known methods that are for the most part documented, if not open-source. Most of their competitive advantage is in the design of the language itself, the interface and in their internal virtual machine. These details aren't really things you would get any insight into with a "show the work" mode.
Your first reason is totally right though. "Show the work" mode requires a lot of extra stuff, because the way a person reasons through solving some thing like an integral isn't much like how a computer solves it. To make it spit out something really comprehensible to a person, you have to have lots of heuristics and it isn't generally very reliable. Even automated theorem provers (which Mathematica has some very basic functionality for) spit out rather hard to follow or extremely long proofs. Those are much more rigorous and less reliant on heuristics than a CAS and still don't produce "pretty" output (AFAIK, I'm not super experienced using automated provers or proof assistants).
Showing steps has been done before, e.g. by Macsyma like OP mentioned and by Wolfram too (Alpha does it). It's just not that great for problems of any substantial complexity.
Pity it never compiles, and has been forked into two other projects.
"the determinant of bigMatrix is, approximately, 1.95124219131987·10^9762."
Approximately? How many significant digits in the exact answer? If Mathematica promises arbitrary precision, even to 10,000 digits, they have a case. If not, it shouldn't be any surprise.
As for getting different results on each run, I've seen this with iterative algorithms that use random starting values. For numerically unstable problems, that can lead to different results each time because it converges to rounding error.
That doesn't mean Mathematica is wrong any more than getting wrong results from doubles means its wrong. It just means it's poorly designed so the user doesn't know about this limitation.
I would certainly not expect this at all. These are not int64s here, but are instead "big integers" as bunches of languages have, see also GMP.
There are 9763 significant digits in that calculation. They wrote it in shorter form because the actual digits were irrelevant. The numbers they got from Mathematica had the wrong sign and were off six orders of magnitude. There's no reason to list more digits to show that's the case.
Mathematica promises arbitrary precision. Here's the documentation. http://reference.wolfram.com/language/ref/Det.html . It says "Use exact arithmetic to compute the determinant", for the construction given in the paper. The also submitted a bug report, and got the statement "It does appear there is a serious mistake on the determinant operation you mentioned."
Yes, of course random algorithms can end up with different answers. The determinate definition is not random, the documentation for Det doesn't say it uses a random implementation, non-random methods to compute it are well known. Again, the vendor says it's wrong - why are you disagreeing with practicing mathematicians and the vendor? Why do you think you know enough about the topic to be able to judge what "odd" means, in the context of this sort of field?
> To avoid the usual problems with floating point arithmetic (rounding, truncating, instability), we construct all our examples with integers.
I think this is complete overkill. I know floating-point arithmetic can be tricky, but it's not at all random. If you have a stable algorithm and a multiple precision floating-point library, you can use floating point numbers. It won't necessarily give you a proof that your result (unless you spend time to derive rigorous error bounds), but it will certainly be good enough.
I think people sometimes do slightly irrational things for fear of floating-point arithmetic.
Similarly, if I need to write an app that deals with real money (say, a banking application), you can bet your bottom dollar I'll be using integer arithmetic.
The inability to represent 0.1 properly in binary floating point is not a problem solved by increasing the precision your floating point library.
Also, there can be be surprising cases where single-precision might not be enough. For example, if you are going to represent exchange prices for, say, futures, you need to take into account that some of the price increments are in pennies, some in fractional dollars. You need the calculations to be reversible, and to have enough precision and accuracy to represent the whole range of price increments and expected volume.
One example of a lack of appropriate paranoia about floating point numbers was the i'm-not-very-good-at-writing-parsers' author of ASP and the result was http://www.exploringbinary.com/php-hangs-on-numeric-value-2-... denial of service to all unpatched PHP sites. Also bit java.
Note that interesting prime number work depends on integers and won't work with floating point. Including factoring large numbers in cryptography.
> you need to sort them first
Yes, but only if there are quite a lot of them and the partial sums are large relative to the result. Not really an issue usually.
> The inability to represent 0.1
Many mathematical conjectures apply for arbitrary real numbers. The nearest floating point number to 0.1 is a real number, and it's not really an issue for mathematics that it's not exactly 0.1.
> single-precision might not be enough
These are mathematicians working in Mathematica, which has an arbitrary-precision floating-point type. They don't have to come anywhere near single precision floats.
> if you are going to represent exchange prices
You can use integers. Floating-point arithmetic isn't really great here because it makes the wrong kinds of accuracy guarantees.
> prime number work
Primes numbers are all integers. It would be a bit silly to represent integers with floating-point numbers.
...I think you'll want to use a priority queue, with the smallest numbers on top, adding the first two, and then pushing the result back onto the priority queue. Repeat until you only have one item left.
That is precisely the point.