Getting Started with APL (2017)
sacrideo.us
sacrideo.us
Then there's the issue of typing in all the weird APL symbols. I've reconfigured my keyboard to type special symbols a long time ago, but I don't even remember how to do that now so it'd be nice if there was some straightforward documentation on how to do that for APL on a typical Linux system -- or, better yet, some easy plug-and-play way to do that. That would eliminate yet another major hurdle.
They've changed the licensing a few times. Originally the 32-bit version was free for commercial use, then it became non-commercial use only. Now you can get both the 32-bit and 64-bit versions free for non-commercial use, but you're not allowed to run the free 64-bit version "in the cloud".
I've first-hand seen the cost/proprietary argument thrown into the face of KDB, to then see companies proceed down a path of other technologies, with a 10x spend in hardware and dev effort.
KDB is insanely tuned.
I thought Gentoo people despise plug-and-play. This is what I have for xorg
Section "InputClass"
Identifier "KeyboardDefaults"
MatchIsKeyboard "on"
Option "XkbLayout" "us,apl"
Option "XkbModel" "pc104"
Option "XkbVariant" "dvp,dyalog"
Option "XkbOptions" "grp:shifts_toggle,caps:hyper"
EndSectionIf we did, would we have a package manager? Would Gentoo auto-identify hardware as it booted? Gentoo is mostly plug-and-play, like every other distro. We just have a lot of choice.
Thanks for the apl ebuild, by the way. That should make things easier, though it'd be much better if it was part of Gentoo's official portage repo.
Thank you also for the keyboard configuration. But how would I switch layouts from US to APL? Also, is there some graphic I can refer to to see which keys will output what APL symbols?
I like the APL symbols better though.
I don’t at all mean this to be heckling from the peanut gallery, I have great respect for the people, the tools, the company; but from the distant outside there seems an air of slightly directionless faded grandeur, if you will, like looking at an old-timey cinema.
If Dyalog APL is the future of APL, how do you imagine that future?
Before Dyalog was acquired in 2005 and Gitte and I came in as CEO and CTO, the team working on Dyalog APL consisted of 5 people. Today that number is heading towards 25, several of whom were hired over the past decade to be ready to take over the work of Scholes and Streeter.
Gitte's comment about "not enough" young people signing up needs some context. The problem was actually caused by significantly increased interest in APL, which made us realise that our "classical" registration process was putting potential new users off. It was often taking too long to get people a copy of Dyalog APL because of the manual processing that was required, people were confused by the licensing terms and afraid of using the software – and many young people are just allergic to providing any information about themselves online. Now, APL is available for Windows, Linux (including the Pi) and macOS with no questions asked – for non-commercial use.
Losing Jay as CTO was a bit of a blow, but I think we are on the way to recovery. Now that he has completed his PhD on the subject of a parallelizing compiler (research that Dyalog has funded for the last 5 years), Aaron Hsu will join young guns Marshall Lochbaum and Adam Brudzewsky at Dyalog, and we have one more C developer joining the team in the UK in January. We hired two fresh APL consultants in the USA in 2019, and will be looking to hire two more in 2020 to support both existing and new business. I've been in the APL business for 40 years now, and my opinion is that the "kids" at Dyalog are every bit as impressive as Scholes and Streeter were at the same age; the shoes will be filled. There is as much young talent at Dyalog as there has ever been working on any APL interpreter, throughout the 50+ year history of the language.
As the team grows, we are able to afford having one or two team members focusing most of their efforts on performance. These efforts have accelerated over time, as first Nic Delcros, then Roger Hui and now Marshall have spent significant time rewriting primitive functions to take advantage of vector instructions and new hashing, sorting and other algorithms that have been evolved over the last four decades. Hardware has evolved in ways which mean that many of the old algorithms are sub-optimal on modern chips where the relative speed of CPU cycles vs memory access is completely different from the Z8000. Yes, Marshall is very good indeed, but I can't see a problem there . Performance has improved consistently in most of the last ten versions of Dyalog APL, and I hope we may see even more spectacular improvements in the next decade, both in the interpreter and in GPU-based parallel compilers.
We have much work to do: In addition to language and performance work, we expect Aaron to work with Richard Park, another new recruit who was added to the team last year, to work on training materials and documentation – and with Marshall, Roger and myself on quote evangelism unquote; talks at Lambdaconf, Functionalconf and perhaps some new conferences in 2020 and beyond (invitations welcome!).
For decades, Dyalog was a company that made a very comfortable living providing a better APL interpreter to clients who converted to Dyalog APL after initially purchasing an APL system from one of the companies that did serious marketing of APL interpreters: companies like IBM, STSC and I.P.Sharp Associates. Thanks to our recent growth, and the fact that Dyalog APL now has decent tooling not only under Microsoft Windows, but Linux and macOS too, we are now starting to focus on developing new business and attracting the next generation of APL developers.
I'm sorry to hear that we appear "directionless" to you. I will admit that it is an interesting challenge to keep existing customers satisfied (which should always remain the first priority for any organisation – put on your oxygen mask before helping others) while also working on attracting completely new groups of users. The rate of development of new language features, development tools and libraries for Dyalog APL is as high as it has been for any APL interpreter in history, at the same time as we are making significant performance improvements. Is there anything in particular that you miss?
Aaron's work on using APL as a high-performance tool for manipulating tree structures demonstrates that APL is more relevant than ever before, as a programming language that has a natural mechanical sympathy with modern hardware, and allows relatively unskilled developers to write extremely efficient solutions without using complex tricks. I recently had the pleasure of sharing a podium with Aaron in Mumbai as part of the JioTalks series, where we covered some of the reasons why APL is a tool worth taking a look at in 2020, even though the core of the language is more than 50 years old: https://jiotalks.com/watch/204/home/Morten_Kromberg_&_Aaron_...
Thanks for asking!
I keep up with Dyalog's twitter and conference videos, but wish there was a little more in the way of beginner material. The recent video on using the CSV/HTML/XML/JSON calls was great btw.
Two questions I have revolve around performance, numerical computing, and program distribution. To me, having to use Python + Numpy, Matlab, or low level languages for scientific and numerical computing is a shame as APL is basically math notation and should excel here. I hate wasting time writing loops (I parse lots and lots of text files) and how it makes my code harder to view and keep up with (APL fixes this). However, will Dyalog ever invest in writing some libraries or code snippets or primitives for things like sparse matrices and matrix factorizations that I probably don't want to implement myself and that make more sense to be done by the language implementers in C (I guess similar to how Dyalog has native support for JSON & CSV)? I'm guessing you can call out to BLAS/LAPACK (like J) or use Python or R's libraries via APL's bridge libraries, but that defeats most of the reason why I would want to use APL to begin with and I would end up with a franken-system and would be better off just using Python.
I guess what I'd like to see as a user is a way for me to write APL code in a tacit manner (I really like the organization of Aaron's compiler) and the compile it as a binary that could be shared with other users at work. It would be nice to have some built-in scientific primitives. Honestly though, the licensing is the real problem last I checked. It seems like running in production would be a non-starter, so I could only use as a single user tool (probably not worth the fight). I know this might be counter to how y'all currently make money, so I'm not exactly expecting a response on this note, but felt you should know what the few deal-breakers are from someone who has considered using it professionally but has shied away.
Additionally, sparse matrices are on the roadmap for Co-dfns in the future, so there's hope for you there. If you have a specific problem that you wish would work, the best thing to do is to make it known to us so that we can actually prioritize it. I think myself and Dyalog often prioritize those things for which we know there are users. If you want to be able to do this stuff, it would be great to get some concrete programs that we can work off of.
Finally, licensing should, IMO, be a non-issue for most people in the vast majority of cases. Dyalog provides some of the best licensing terms that I've seen, and there are already models that scale well both to startups and to existing deployment in large-scale commercial enterprise situations. If something in the licensing terms doesn't seem to work, I would get in touch with Sales @ Dyalog and they will surely be able to work with you to figure out something that would be equitable.
Most people are surprised when they find out how easy it is to move forward with Dyalog on a commercial product, or how readily Dyalog is looking at making things work for customers or people who wish they could be customers.
> I'm sorry to hear that we appear "directionless" to you
I hoped for this to be quiet feedback in case you don't see it from inside, rather than criticism, I'm not a commercial user (yet?) and you have nothing to be apologetic about; I appreciate your detailed and comprehensive reply, am enthused about many of the things you've said, am glad Dyalog is thriving, and wish you all continued success!
> Is there anything in particular that you miss?
I miss the familiarity of an imperative/OOP language with intellisense completing the names of methods and ordering of parameters, but I'm sure that will pass with time. :)
Dyalog APL does seem to be nice.
I would encourage you to leverage this to write a Gentoo portage (or whatever they are using now) build for it yourself and send it out for the world. We have a number of Nix users that come to our APL workshops who have arranged for just such a system on Nix.
For typing the APL symbols, I use this: 'setxkbmap -layout us,apl -variant ,dyalog -option grp:lswitch'. I find this to be the most efficient for typing APL. The APL keyboard map comes default on modern Linux distributions.
Dyalog APL also has extensive documentation for configuring your APL system on your platform.
https://github.com/markus1189/dyalog-nixos
This should get you started, though some of the instructions are based on the less friendly download process of v17.0 instead of the streamlined process of v17.1.
As for the beginner's book, I've come to the general feeling right now that a online video course will be a better fit for APL beginners than a textbook. This is an active project of mine that I anticipate completing some time within the year 2020.
You may also want to see my presentations on design patterns for APL and the upcoming videos at FnConf 2019 this year. I am not sure if the APL Training Wheels session was recorded, unfortunately, but the content from that work will be going into the video course.
Other than generating emotional responses, I don't find video better than text at anything, and most especially not technical education.
Part of this is a question of what the stamina level is for the current computing student. With a lot of technical documentation that people are learning, they're essentially consuming either thorough reference materials or walkthrough/how-to materials.
We can characterize both of these materials as efficiently transferring a certain sort of information about how to do things by virtue of a very wide base of shared prior knowledge. This is what allows reference materials and walkthroughs to work.
But this shared base of knowledge does not exist for most people learning about APL. Target audiences include non-computing students who are just learning computing, programmers who want to learn APL, and technical professionals who have little or limited prior computer science knowledge (but perhaps cursory programming knowledge) who need to learn APL to accomplish the work they want to do in their domain. In the cases of the technical professional and the uneducated student, reference materials and most tutorial books are wholly inadequate.
For the programmer, these tutorials and references work because most of the time you are adding to your existing knowledge as a programmer and have already trained numerous soft skills that enable you to navigate the sea of potential directions to integrate the new knowledge.
I worked on my thesis to try to make it as accessible to the general programming community as possible to teach a set of algorithmic concepts that most programmers already have a vague notion about (but almost never use), namely, SIMD/data parallel programming, using an extremely simple and easy to explain syntax and vocabulary (APL). My thesis is already pretty detailed and illustrative, with worked examples for every single programming statement in the entire compiler going over examples and prose detailing everything that is going on, a sufficient background introduction not only to the syntax and semantics of APL, but also the design and stylistic questions, reference material, and the architectural education necessary to understand the structure of the compiler as a whole.
Even with all of this, presented to experts in the C.S. fields of PL and parallel computation, who have seen this material multiple times over, there was a general feeling that my thesis could have benefited from even more exposition.
My thesis is around 70,000+ words and around 250+ double-spaced pages. For 17 lines of APL code. People wanted more. Had I added the additional exposition desired, I imagine the thesis easily could have been 80 - 100k words.
A typical short novel is around the same size, and those often take about 8 - 12 hours to read through aloud. If you give me around that amount of time in classroom time, I could easily have almost any programmer fully understanding my compiler within that time, and probably with more thoroughness than if someone were to take the same amount of time to read the thesis.
There is something about the higher bandwidth and richness of full video content that contributes better to the transmission of data in some ways, particularly to the understanding of certain ideas and concepts.
I began to think about what it would take to put together an introduction to APL as a textbook. These have been done before and are out there, and they would generally work, but I'm not convinced anyone would read them.
You can teach APL the language to a sufficient level in a very short document (maybe a few pages) and in maybe 2 hours of classroom time. It's relatively trivial to teach the language, as it's very, very easy to understand the semantics and syntax. There's even a walkthrough of APL at http://tutorial.dyalog.com already.
The challenge comes in its application, and that's where the real value of a good education rests. But that part of the education is a lot harder to teach in text, because there is an element of style, aesthetic, and practice that goes into it. It's something that you gain by seeing, watching, and doing. You have to bring your mind into the space and struggle, go in and out of exploration mode and absorption mode.
How do you teach the exploratory process of APL? The design aesthetic? The set of idiomatic patterns? How do you get people to intern those concepts into their practice so that they are productive?
You could write a textbook to do this, no doubt. But thinking about it, I think such a textbook would be very large. It would be not very "exciting" emotionally. It would have very little of the emotional hooks that help a person stay engaged in things. And moreover, it would be extremely intimidating.
How many people effectively learn without any teacher in a totally "new" field from a textbook that is devoid of any human feedback? You can do it. I've done it before. But it's infinitely improved by the introduction of someone who can help to inject energy and dynamism into the space.
I just don't think people will want to pick up a 400 page "introductory" book and then have the motivation to stick with it when there are so many other shiny things in this world today. The truth is that the world is filled with distracted people who are very bad about putting the time into learning things. I want APL education to be engaging, accessible, and satisfying, to say nothing of effective. For most people today, a textbook is unlikely to be that.
Another big challenge with the "experienced programmer" target audience is that they are used to reference and tutorial materials. Very few programmers actually sit down and read such materials. They skim them to find what they think they want to know, and then plug that into their existing framework. That's great for efficiently extracting new information that is similar enough to their old practices to work. If something breaks, they can just go back and do a little extra skimming to clarify. That doesn't work really well when what you need to learn is fundamentally conflicting with or altering how you think about problems.
Getting good with APL at the beginning, before you've learned to intern it, is about undoing or unwinding previous habits and thought patterns that worked for you in past programming experiences, but that won't work for you in APL. After you have interned the new methods, then you can switch back and forth at your leisure, but until then, learning APL is as much about unlearning as it is about adding new knowledge. For programmers who are used to just skimming what they read to get what they want to know without actually listening to what the author is saying (and that is how most programmers read that I've met), this makes textbooks a recipe for disaster. Either the information has to be presented in such a dense way that most people will not be able to figure it out, or it has to be presented in such a measured, hand-holding way that most people will just ignore it, get bored with it, or skim it and still have the issues.
It does little good to have a full, thorough textbook introducing everything if no one reads it, though they think they've read it, and then they complain about how hard APL is to learn because they couldn't just take what they thought they wanted to know and leave the rest.
If you could get a programmer who was motivated, eager, and willing to read from cover to cover, stopping at each point where a hard question is asked, do the problems, quizzes, challenges, and exercises, in order, linearly, from start to finish, and actually pay attention to what was said when, and then take the points in the book that indicate one should think about this or that, I would agree with you that a textbook could have some potential benefits over the video format; but, I don't think that this is reality, based on my experience.
Your typical APL workshop for technical professionals is 4 days about about 24 hours of work with probably around 12 of those hours being expository. After that, people are quite solidly grounded in APL. But a lot of people have gone through that same amount of time on their own with plenty of the same resources available to them in terms of problems, documentation, and the like, and they haven't been as successful. Guided, in-person education is hard to beat, and video is the closest thing online.
I haven't given up on the hope of a great APL book, and I could be convinced by someone that I should do a book first, but at the moment, I feel that video courses are more likely to be more effective for more people and present a larger ROI in terms of the difficulty in producing and creating them compared to a book of similar effectiveness for the same range of people. If anyone has an alternative, please do speak up, I'm all ears.
I find Jupyter notebooks in Python and Wolfram's Mathematica to be nearly perfect for explaining complicated content where you can easily format text, images, and inline code snippets.
github.com/rikedyp/learnapl
It is currently very rough around the eges and also in the middle. Work will continue in the new year.
There is also now a set of introductory problem sets that have been used in a couple of schools on problems.tryapl.org
https://www.youtube.com/watch?v=v7Mt0GYHU9A&feature=youtu.be
You don't really need anything of this to solve a problem with Python and Pandas.
I always feel like I wish I'd solve it with APL (because it's beautiful) but end-up just getting the thing done this way :-]
Thank you for the lecture, perhaps it is going to inspire me to actually learn to solve real problems with APL.
I maintain that APL isn't fundamentally more difficult. If there is something that you have as an example that you think is just too "hard" to do in APL that is very "easy" in Python and Pandas, please do let me know via email (arcfide@sacrideo.us) as I would love to make sure APL has a good answer to such things (I want to make APL accessible).
Since you are talking about statistics and data analysis type things, you may find TamStat of interest:
Additionally, don't forget that Dyalog APL also has a full Chart/Graphics system (two, actually), and a large suite of idiomatic expressions for computing many classic analysis problems. There's a number of other integrations that people seem to be unaware of often enough.
I very seriously would like to receive from you one example of something you have done that you find too difficult in APL, because if it really is, then we can do better and I'd like to make sure that we do.
Python certainly requires one to learn iterative programming which is definitely not a walk in the park. I learned it over a decade ago, but vividly recall how non-intuitive it was at the time (global versus local scope had me scratching my head once upon a time). Then there is lists, tuples, dictionaries, pass by reference, iterators, file I/O...just soo much to learn.
One interesting thing is that I used OO methods in Python for ages before understanding how to write and use classes. To me, a = list.sort() was just a weird way to read a = sort(list). You can almost write Python entirely in an imperative manner and for short scripts I almost always do.
Learning all the OO aspects of Java/C# and all the fundamentals of FP (monads, currying... etc) is much harder for me than APL which is basically just learn what these symbols do and the very straightforward way to combine them and you're done. The only harder part of APL is somehow learning to replace imperative programming with array operations. Sure it is easy to do some things like sum up an array (+/ array), but that is also easy in Python (list.sum()). The idioms as APL people call them are not at all obvious to me, nor is the path of enlightenment outside of stop talking and go write some APL haha.