A port of Andrej Karpathy's llama2.c to pure Julia. Works on Llama2 and Andrej's tinyllamas models. REPL mode allows interaction from the Julia console
501 karma · joined December 10, 2013
[ my public key: https://keybase.io/jiahao; my proof: https://keybase.io/jiahao/sigs/EAK5furhvrbEq6fik1OISVRRHRjIdOOx30J0DN5sVpU ]
TL;DR: Researcher A finds things he wants improved in Magma (closed source) but can't. Researcher B tries to write improved FOSS implementation, but lost his job, likely in part by spending too much time writing said code and not doing other things like writing papers. Researcher A moves on and has a successful academic career. Moral: writing FOSS can cost you your academic job; it's safer to find something else to do.
Yes, there are at least two separate stages of historical development here. The first is when people realized it was useful to repeat the same operation on different data in memory and viewed the collection of data as a variable in its own right. The earliest term I can find for this concept is "subscripted variable" (many examples prior to 1954, e.g. Rutishauser, 1952 in the "for" loop paper; Laning and Zierler, 1954)
but the idea appears to go all the way back to Burk, Goldstine and Von Neumann in 1946. Quoting p. 9, paras. 3.3-4:
"In transferring information from the arithmetic organ back into the memory there are two types we must distinguish: Transfers of numbers as such and transfers of numbers which are parts of orders. The first case is quite obvious and needs no further explication. The second case is more subtle and serves to illustrate the generality and simplicity of the system. Consider, by way of illustration, the problem of interpolation in the system. Let us suppose that we have formulated the necessary instructions for performing an interpolation of order n in a sequence of data. The exact location in the memory of the (n + 1) quantities that bracket the desired functional value is, of course, a function of the argument. This argument probably is found as the result of a computation in the machine. We thus need an order which can substitute a number into a given order-in the case of interpolation the location of the argument or the group of arguments that is nearest in our table to the desired value. By means of such an order the results of a computation can be introduced into the instructions governing that or a different computation. This makes it possible for a sequence of instructions to be used with different sets of numbers located in different parts of the memory.
"To summarize, transfers into the memory will be of two sorts:
"Total substitutions, whereby the quantity previously stored is cleared out and replaced by a new number. Partial substitutions in which that part of an order containing a _memory location-number_-we assume the various positions in the memory are enumerated serially by memory location-numbers-is replaced by a new _memory location-number_.
"3.4. It is clear that one must be able to get numbers from any part of the memory at any time. The treatment in the case of orders can, however, b more methodical since one can at least partially arrange the control instructions in a linear sequence. Consequently the control will be so constructed that it will normally proceed from place n in the memory to place (n + 1) for its next instruction."
https://library.ias.edu/files/Prelim_Disc_Logical_Design.pdf
The language is of course archaic, but the idea described clearly is that of indexing in 3.3 and arrays in 3.4. They use the word "sequence" but arguably this usage is in its ordinary mathematical sense.
The written historical evidence, at least, would confirm your strong guess that the idea of arrays itself is older than index registers. There's a missing etymological link though: when did a sequence of data stored consecutively in memory become associated with the word "array"? Still, the earliest written reference I can find for this second stage of historical development is the 1954 preliminary report on FORTRAN.
Maybe the word "array" is somehow derived from the advent of RAM, which even in its earliest form in Williams tubes had memory locations arranged physically in two dimensions. So right from the start we have two dimensions physically, but only one dimension logically, since the earliest computer instructions only dealt with (one-dimensional) offsets, if at all. Furthermore, popular science accounts of magnetic core memory describe them in terms of arrays. To give one example, the June 1955 issue of the Scientific American (no 192, pp 92–100) writes about "magnetic core arrays".
http://www.nature.com/scientificamerican/journal/v192/n6/pdf...
I was surprised to discover recently that the word 'array' prior to 1950 was used exclusively to describe two dimensional tables of numbers that one might find in a matrix or determinant. But by the advent of FORTRAN I in 1957 and ALGOL 58, 'array' now referred exclusively to a one-dimensional entity, as compared with 'n-dimensional arrays'. I was interested in digging through John Backus's papers from this era to see if I could find any clues.
I was able to narrow down the near window to 1952-1954, since the FORTRAN preliminary report of 1954 uses the word 'array' casually in the modern one-dimensional sense as interchangeable with 'subscripted variables', the latter being the more common terminology at the time. By comparison, a virtually unknown paper by Rutishauser in 1952 describing the "for" loop did not use the word 'array' at all, only 'subscripted variables'. (Rutishauser was an accomplished mathematician and quite possibly the world's first computational scientist.) A paper by Laning and Zierler at MIT in 1954 describing a formula compiler also used only the term 'subscripted variables'.
Backus's papers also have evidence showing that FORTRAN I was clearly written specifically to take advantage of the IBM 704's machine capabilities. Not only was the IBM 704 the world's first commercially successful computer, it was also an improvement over the preceding IBM 701 in providing index registers (3 of them) and floating point instructions which were fast for its era. Backus's papers describe how providing hardware support for indexing and floating point was revolutionary, as all programs up to that time had to write in all these instructions by hand (and for many programs was pretty much all they did).
So it is clear to me now that the changeover in the implied dimensionality of the word 'array' must be related to how the array developed as a data structure abstracting away indexing operations. By the time IAL (pre-ALGOL) came on the scene in 1958, the idea of indexable homogeneous containers was already well established. But I still haven't found any strong smoking gun evidence introducing the one-dimensional sense of the word. I suspect further digging into the description of the IBM 704 may be necessary. The 704 was not the first to provide index registers, but it may have been the first to call them as such. (The Manchester Mark I computer of 1948 appears to be the first computer with an index register, but it was called a B line. The [patent](https://www.google.com/patents/US3012724) claiming to cover index registers uses the term "control instruction" - no arrays mentioned - but it very cutely describes numbers as residing in known locations or "addresses" in quotes.)
I've started a GitHub repo with a first draft of transcripts for all the letters.
https://github.com/jiahao/backus-dijkstra-letters-1979
Anyone else who is interested is welcome to help proofread and submit PRs.
Unfortunately, the Library of Congress does not allow scanners without prior approval, so this was the only way I could make my own copies. It did not help that all these letters were written or typed on very thin mimeograph paper.
- Both Julia Computing and the Julia Lab have grown sizably over the past two years. The Lab now houses ten full-time researchers (up from four last year), with five new students coming online over the summer and fall. We also maintain more active research collaborations with more research groups at MIT and off-campus.
- Julia is a grateful recipient of 12 Google Summer of Code slots this year, compared to 8 for 2015's Julia Summer of Code program (sponsored by the Moore Foundation) and 4 for GSoC 2014.
- JuliaCon grew from 72 attendees in 2014 to 225 in 2015 and we are on track to meet or exceed last year's ticket sales for 2016.
- New packages continue to be registered on the central METADATA repository at roughly the same rate since June 2014. http://pkg.julialang.org/pulse.html
By some measures we are still a relatively small project, but I don't see any serious evidence for the imminent heat death of the Julia universe.
Mostly false; see https://www.cs.berkeley.edu/~wkahan/Mindless.pdf
It is true that an arbitrary, independent sequence of n floating point operations can potentially lose up to 2n bits of precision. However, it is false that most scientific computing code has such structure. On the contrary, the loss of precision in numerical algorithms can be estimated quite well using standard techniques of error analysis, especially when there is underlying matrix structure to exploit. The growth of error in properly written numerical code should NOT grow exponentially with the number of floating point operations. It should be bounded by intrinsic properties of the underlying matrix, such as its condition number or spectral norm.
https://github.com/rpmuller/pyquante2/tree/master/julia
Let me know if you do want to do some quantum chemistry coding. It would be good for my soul to revisit some of my PhD roots...
http://www.acs.org/content/acs/en/about/aboutacs/financial.h...
The smaller American Physical Society made <$100,000 on $53.5m gross revenue in 2015:
http://www.aps.org/about/governance/annual-reports/
So yes, it is possible to have high quality publications produced on much smaller margins. The question is whether it is possible to provide similar services for similar margins in a for-profit context. Given that for-profit companies, by definition, seek to extract the highest possible profits on what they do, one should really question whether it makes any sense for for-profit publishers to have the stranglehold they have on academic intellectual property.
https://en.wikipedia.org/wiki/Elsevier
https://libraries.mit.edu/scholarly/mit-open-access/open-acc...
Happily, the present situation is more rosy than it has ever been. Funding agencies are beginning to realize that academic software development is an activity worth funding in its own right, not necessarily tied to research. Julia Computing has also been fortunate to receive funding from organizations such as the Moore Foundation to work on precisely the tools that users most need, but are difficult to justify development with research funds.
I have yet to come across any other linear algebra library for any other high level language that provides the depth of integration available in the Julia base library. Want all eigenvalues of a symmetric tridiagonal 10x10 matrix between 1.0 and 12.0? Simply call T=SymTridiagonal(randn(10), randn(9)); eigvals(T, 1.0, 12.0). Or if you want to work closer to LAPACK, simply call LAPACK.stein!. I don't see a wrapper in Breeze or SciPy for this function. Want an LU factorization on a matrix of high precision floats? lufact(big(randn(5,4))). And so on.
Julia may not have everything users want, but its base library really tries to make matrix computations easy and accessible.
See Misconception #2 of http://lipforge.ens-lyon.fr/www/crlibm/documents/cern.pdf.
Floating point numbers are _not_ intervals. If you read carefully any formal definition of floating point numbers (The IEEE standards, TAoCP Chapter 4, or Higham's _Accuracy and Stability of Numerical Algorithms_, to name just three possible references), you will see that floating point numbers by definition form an exact rational subset of the extended real line.
> All numbers in that range alias to the same double, yet differ wildly in their decimal digits beyond the fifteenth. That "long tail" of digits is a meaningless residue arising from the arbitrary difference between the chosen center-of-range point and the actual number it approximates.
See Misconceptions #1 and #2 on Kahan's list (https://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf).
You are not being clear about the set of real numbers a user may input and their ultimate representation as a floating point number. While it's true that many real numbers round to the same floating point number _f_, that's not the same as then saying that _f_ carries that interval around with it. The latter is false, since the intervals do _not_ propagate in floating point arithmetic. It's also false that the floating point number _f_ is the midpoint of the set of numbers that round to _f_; the precise set depends on the rounding mode and the granularity of the set of floating point numbers around _f_.
April 1 was a good day in the lab!
http://data.jobsintech.io/companies/massachusetts-institute-...
By the way, the paper was really an excuse to document this IJulia notebook.
http://nbviewer.ipython.org/urls/raw.github.com/jiahao/ijuli...
If you run it on your local machine, one of the cells has an interactive slider that lets you control the breadth of the parallel prefix tree you generate.