Eve: My Concerns
robmoff.at
robmoff.at
We've learned a ton over the past couple of years and we've finally settled on a path forward. That means we're starting to move out of research mode and starting to execute on all of this knowledge gathering. To that end, we're going to be doing a public alpha release in the next few weeks for people to play with. :) It ought to be interesting to the folks here to see how crazily different our work could be - the latest version of Eve is better likened to Medium's content editor or Dropbox's paper than it is to Visual Studio. It's been a long road (with around 30 versions of Eve at this point), but there are some really exciting times ahead.
Happy to answer any questions! Though I imagine there'll be plenty of discussion when we do the release :)
As a blind developer, I always get a bit nervous when the future of programming is discussed in terms I've read used to describe Eve. It seems like a very useful way of developing apps, but one that can easily exclude a subset of the population if a11y issues aren't thought through in advance.
Also, part of me is tempted to dismiss Eve because I suspect we'll always have the current way of developing software to fall back on. But when I consider it more, I don't find that line of thinking productive for me. For instance, lots of folks might be tempted to dismiss GUIs for blind users because graphics, but there are actually huge advantages to representing UIs as trees of objects that can be introspected and interacted with. I can write code that scans the accessibility hierarchy for a specific element and reads it, for instance. In a command line interface, that code would need to either scan for a non-specific set of characters that a curses app might use to draw a thing that looks like a button and make a guess at its contents, and even then that logic would be very brittle. So even if Eve and whatever comes after derives lots of its power from rendering concepts visually, the underlying primitives needed to do so might have beneficial a11y implications and shouldn't be discounted because the end goal happens to look nicer than a wall of text.
Anyway, would be nice to see that on the roadmap if it isn't already. Even if only 20% of programming is like this in 10 years, that's still a substantial subset that might exclude me by virtue of the accessibility story not being thought out first.
Many of the previous versions of Eve would be pretty completely inaccessible, so your dismissal would definitely be warranted. The version we've decided to run with though basically has three parts: a document editor, embedded spreadsheet cells, and a fancy plain english search. All three of which have a history of accessible solutions, so that makes me much more optimistic that we can provide something nice here. Along the way, another thing we've been very cognizant about is making sure no matter what form of input you prefer, Eve should be efficient and useful to you whether you're using a keyboard, a mouse, or touch. Many of the interaction issues we'd need to solve to better work with screen readers popup when trying to support complete keyboardability, but I have no delusion that there will be plenty more to solve there. Another interesting aspect of this version of Eve is it's really easy to imagine a voice interface to since virtually everything is done using plain english queries - it'd be kind of like having Jarvis in your google docs.
Admittedly, we don't know what we're doing when it comes to accessibility, but it is something we've been thinking about indirectly and it's something I think we can at least do more naturally than the code editors of the world.
Thanks for raising the issue and it's something I'd like to have a more concrete answer for in the future.
What's interesting is the problems you want to address I deal with every day programming games. When you're making games you have to simulate physics in your head and the setup to just get a project ready is a gigantic pain. How could a game programmer use this type of system to make a game? It seems like the complexity would just get out of hand.
The first is whether a bloom-like language such as Eve is semantically well-suited for game design [1]. Unfortunately, there's not a lot of precedent here yet (it feels like there hasn't been a lot of overlap in interest in game design, datalog, and free time yet). Having used various datalog implementations over the past year though, and I'm convinced that it lends itself very naturally to building a Component Entity System style game engine. datalog's twin benefits of provenance-based debuggability [2] and strong decoupling [3] really help in building and maintaining advanced logic systems. I'm concerned it will have a bit of a learning curve, since we are as a rule not very used to thinking about systems programming this way, but it's certainly felt more intuitive than any other technique I've used in the past once I picked up the mindset for it.
The second is whether a language that lives in a persistent (potentially remote) relational DB is fast enough to run a game in. This is the bigger question for me. We've been doing a lot of research in this field and I'm convinced that for Eve's specific needs we can architect a much faster DB than the general purpose alternatives [4]. That said, even with all of the clever tricks in the book, there's still going to be some amount of overhead compared to working in purpose-built data structures with raw assembly. In cases where we're just not fast enough (either because the optimizations and query planning aren't all there yet or because we've come up against a fundamental design bottleneck) we do have an escape hatch. Through a system called watchers (which enable Eve to I/O with the outside world), it's possible to wire in compute programs written in other languages to do the heavy lifting with a very low latency. The watcher system is probably best conceptualized as a very simple FFI. I think we can get Eve to the point where it handles typical 2D games with aplomb, and for the rest I plan to put a lot of love into the watcher interface with the hopes that it will be fast enough to keep the game logic in Eve and just delegate physics and potentially rendering work to external components.
Anyway, this is still a fair ways off, we're beginning with an MVP and iterating from there. Game development is certainly on my radar though, and I look forward to publishing news about it when Eve is ready to do so. :)
[1] In the current experimental version of Eve, the only exposed query language is our high level search language, which makes very expressive searches like `dishes with eggs and sugar that are savory` trivial, but someone limits the power and scope of a single query. This is built on (and indeed compiles to) a Turing complete sub-language that will look much more familiar to datalog users. We tentatively intend to expose this in the near future.
[2] Given that the underlying graph structure of the program is stored relationally as data (exactly like user data) in Eve, it's very simple to see which sources a view depends on or feeds into, and even to ask complex questions like "when did this value become 10?".
[3] One of my favorite benefits of Eve is that it strongly encourages data composition over functional composition. In effect, If I have a view `active users` over the list of all users `users` and a set of other views dependent on that, they depend only on the data `active users` produces, and not it's implementation whatsoever. `active users` could be replaced in any way that still provides data isomorphically without any of its dependents being aware of the change. Further, even if the data changes non-isomorphically, only it's direct dependents will need to change, rather than having to propagate these changes all the way to the leaves of the system.
[4] To be clear, I don't claim that ours would hold a candle against the 20+ years of experience that GP DBs bring to the table across their whole domain of use cases, just over the small handful that we require.
anyway, suggestion is to try and keep your voice at a more constant volume? :) you tend to trail off into a lower level (though not as bad to become inaudible) and then "yeelp" all the way back up at the start of the next sentence, which was occasionally a bit startling :) I was listening to it late at night on medium-large speakers so finding the right level was a bit of a gamble to not wake anyone :-P
either way just a suggestion, maybe there was a reason for that particular presentation. do with it what you will ;-)
and thanks for the interesting ideas on a very interesting topic! :)
Well, if you took away Excel, the banking system would collapse. It might be old and clunky, but it can do anything you need it to. It might not be "real programming", but near enough anyone is able to get a spreadsheet up and running much faster than a POC app that's been built in a proper programming language.
When one imports a CSV into Excel, Excel has a nasty tendency to actually change the data based on formatting rules. When you open up a CSV file in Excel, it silently corrupts the information. You need to back up your file before doing any work on it.
Excel refuses to do proper Unicode, or at least I haven't found the right way to reliably get a Unicode CSV out of it that won't break in third-party tools.
Excel does not do versioning natively, and you have to rely on a buggy mishmash of Microsoft-based solutions to do file-based version management. Excel files tend to get emailed around everywhere, and so there's never a single source of data. It has the worst properties of a decentralized architecture without any of the benefits.
Data belongs in a relational database. You can use a spreadsheet to manipulate it, but the manipulated data needs to go back into the database. The spreadsheets that do this are programs and they need to be managed as code, going into version control systems.
I think it's a failure of venture capital. What's funded is often what looks good rather than what is needed. That's my guess but I don't know.
Nobody wants Excel programming workflow management. But they definitely need it.
I can tell you why they don't want it. Most people do not think in terms of infrastructure. Their job is a series of one-off tasks and they see little commonality between these tasks. Constantly fighting fires, never thinking to design a fire-proof building or procedures for preventing fires.
You ask them why and the answer's always the same. No time. They're too busy. It's wrong, but they don't have the skill set they need to understand why it's wrong.
I've seen this for CRM, accounting, liquidity, ad-purchasing, security vulnerability data. People were thrilled to be rid of Excel and use something that took development out of their hands. The software ends up being more useful, too, when a dedicated team is building a tested, version-controlled application for hundreds of customers.
I've felt the exact same pain as you - having the non-technical people work in Excel and moving it back and forth to a database is a nightmare. I've started creating Flyway migrations for each import/export sequence just so we can keep track of what's live in our DB. If you were working in CSV you could even script an import as part of a build process or something.
In some cases better products even exist. For example LaTeX is a fantastic tool - you can create complex layouts (not just for documents, but also Illustrator-type tasks), automagically manage cross-document references/citations and generate tables-of-content and bibliographies, lay out images and tables and formulas, modularize documents (eg by chapter) and best of all the documents are a plain-text "source" that you can easily check into revision control or merge between multiple simultaneous users. Zero adoption outside the academic world and certain publishing niches (eg math textbooks).
Side note: one unfilled niche I've noticed is that there isn't a good word- or character-level diff tool that's appropriate for LaTeX. Ideally you would want to be able to type natural paragraphs and have the diff pick out changes by word instead of line (but preferably also identify command tokens more greedily up to a line). You can break a paragraph across several lines, but if you're making heavy revisions it just ends up being a solid block of line changes.
One more question: is there a "worksheet" or "notebook" concept in something like Mathematica that might be a better model for the type of work you're considering? I know Mathematica has a pretty complete feature set, it very well might have database hookups and an appropriate workflow for that type of thing.
However, very few programmers can successfully go into a project that is fundamentally a programming tool, and maintain the mind set of "how I make sure that my own way of looking at this remains completely unnecessary?"
If you find yourself having a bunch of conversations with finance types and analysts that all look like: "but if you just approach it this way, it's much more powerful because X" you've probably already failed.
btw, the last company I worked in used Excel as a disconnected editor', extraction, load, edit, save, pushback was ugly. You'd need a careful and solid bridge between the two, and hopefully a semantically adapted backend.
This was my reasoning behind a spreadsheet prototype backed by kdb+ and the q language (which is a superset of SQL): https://github.com/srpeck/kxl
There is value in this simple combination, but tons of work required to gain feature parity and market share from Excel. I think the latter is why research in this space is lacking.
Because now you've removed the primary thing that makes Excel so popular: its approachability. You don't have to learn anything special before you can start getting value out of Excel. You don't even have to understand how to write a simple formula! It's useful even as a simple, dumb rows-and-columns data store.
This is how Excel makes so many people into accidental programmers: they start using it in a naïve, low-knowledge way, and then eventually wonder if there's some easier way to do what they're doing. ("I'm tired of adding all these figures up by hand. Is there some way Excel can do that for me?") So they dip a toe in, learn a little bit, and become even more productive. Then they repeat this cycle over and over, and years later they have become Excel gurus without ever actually setting out to.
(Of course, the downside to all that is that they've now built a ton of stuff in Excel in hacky, incremental fashion, and are quite possibly using Excel to do things a real database would be much better at. But they don't know that -- and besides, if the only tool on offer was a real database they probably would never have gotten around to building anything at all.)
Anything with a "real programming language" is going to lose these people, because learning that language is a big chunk up-front work they have to do before ever seeing any benefit. Anything that wants to compete with Excel needs first and foremost to offer the same kind of immediate productivity benefit and gentle, incremental learning curve to reach the Excel audience.
Lovely comment, and so true. And you're right: for a POC it's unstoppable. I was trying to get to the nub of "why is it perfect for a POC, but not for running an enterprise application"? That's kind of my point, though I admit it was a bit clunkily-put.
At my firm, we are continuously in the process of formalizing excel sheets into auditable applications that don't give enough degrees of freedom to allow our investors to shoot themselves in the foot.
1. Your examples all seem to suffer from the same problem all of Bret Victors ideas do too; they don't scale. Sure I would like interactivity, no I don't want to re-eval my entire codebase on every keystroke, and no I don't want to recreate my vertex data every frame in the Draw method just so I can scrub a tree's bendiness.
2. The card wiki is too reliant on perfect structure. What if I need to change the structure at some point (refactor), then all my queries break. They'll break even worse too if my queries just says 'get the salaries per department', but then I modify the structure to have two sets of employees per dept (perhaps current and ex-employees), which changes the meaning of the query, yet the computer would plod along and now wrongly sum both groups I presume, if working in this automated way.
3. Performance. Sure it's easy to re-calculate an equation based on dependencies. Want to do 'a = b + c'? Then obviously re-calc 'b' and 'c' dependencies before performing the equation. But the hard part is when to do the re-calculation? Perhaps store an 'isDirty' bool? Or cache the value? Or just re-calc everytime it's needed? Human programmers make this decision based on how the result is used, how many times it is used, framerate, memory consumption and performance heuristics. There isn't a one size fits all, it's highly contextual.
4. You say 'we know Wikipedia works'. But have you tried scraping data from Wikipedia? Computers get stuck on all sorts of dumb stuff. Wikipedia is obviously meant for human consumption, where visual data and the English language rule, and the only structure we need is a 'top to bottom' layout. Programming is way more specific that this, so I can't see how combining the two ideas is any good.
5. Organisation. Perhaps you're still working on this, but you switched back and forth between a single card at a time, but given 20+ cards I'm not going to remember where stuff is. Code editors throw a bunch of stuff into one document, have tabs, and an explorer, and intellisense and search etc - and I still lose stuff all the time!
Hopefully when you do this release you'll include some complex samples, like you said you could build a compiler with these cards? So perhaps I'll be proved wrong.
This is a similar idea and there is a long elaboration https://www.youtube.com/watch?v=voG5-15aDu4
Use your awareness and ask yourself: Is this person primarily optimizing programming, or is she/he primarily optimizing their social status as a programmer? (Maybe doing the latter is more rational? Doing the latter likely results in higher pay and a more secure job.)
> The lesson from this was apparently, that just fixing the IDE wasn’t enough. Programming itself suffers from the problem that it is indirect and invisible, beleaguered with accidental complexity and unobservable.
These are the two deepest sentences about programming I've read in a while.
Here's the video in a nutshell:
1 - Chris Granger asks a deep question
2 - He observes people
3 - He tries some lateral paradigm
4 - It's interesting
5 - It's bootstrap implemented in only a few lines of code
6 - It's not the right answer, repeat from step 1
Btw, if you found those two lines interesting, I did an essay/talk about coming to that conclusion: http://www.chris-granger.com/2014/03/27/toward-a-better-prog...
1. Often presentations, reports, etc. rely on calculated data from e.g. an Excel spreadsheet. Whenever data gets updated it's really tricky to make sure everything that uses that data reflects the latest state. Too often have I heard the phrase in meetings "Oh that graph? Yeah, that's still based on last month's data."
2. I've had very smart colleagues who couldn't program, but could bend Excel to their will. Eventually they'd hit the limits of what's possible in Excel though. It always struck me that a) Excel is basically an FRP system and that therefore b) these people already have most of the skills to program if they had slightly different tools.
I always had an inkling that these two problems were somehow related, but I couldn't put my finger exactly on how they are related or what the solution would be. I'm really excited to see that EVE seems to go a long way to solving these things. I can't wait to see where they take it next. At the same time I also take some solace from the fact that, based on the number of iterations, people who are clearly much smarter than me also struggle with defining and solving this problem.
And then ideally it would be possible to state an invariant 'this graph must include this months data' and have the system automatically verify it for you.
The author claims this is a compliment, but generally Notes has a terrible reputation as one of the most over-architected and misused software solutions of the '90s.
You can check it out (in excruciating detail) at this Interface Hall of Shame article from 1999:
http://hallofshame.gp.co.at/index.php?file=lotus.htm&mode=or...
It starts off with: "We wish we found IBM's Lotus Notes a long time ago. This single application could have formed the basis for the entire site."
Maybe there was an elegant core to Notes that was hidden under all the insane GUIs built using the tools? Would be interesting to hear more.
It was the IBM buyout that started the big downfall.
> all the insane GUIs built using the tools?
Remember VB6 and all the shitty applications written in that? Uh, yeah. And don't get me started on the disasters that people perpetrated on us from the web. And how about the pile of dogshit that was Gtk v1? <shudder>
In addition, this was also still the time of "look and feel" lawsuits.
Everybody had insane GUI's in the 1990's, so singling out Lotus Notes is far from fair.
The best ideas from that document database live on in Apache CouchDB. http://couchdb.apache.org/
Yet a simpler model did prevail, and now we use so called toy languages like Javascript all the time. Eve's time will come, and in 10 years most of us who frequent hacker news today will be like the C++ developers of the 1990s in that we will be the minority. And languages like Eve and other similar projects will be considered normal programming languages. Java, Python and other languages will just be considered special purpose languages, much as C++ is considered "special purpose" today for high performance computing and operating systems
What will programming look like in 10 years? My guess is that it will look much the same as it does now. We'll have better languages for expressing temporal logic and consistency but it might not be radically far off from where we are now.
Text is still useful and certainly notations could improve but that's about it.
The idea of programming from tables using declarative or logic programming is certainly not new but very powerful. k comes to mind... which is probably at the extreme opposite end of usability. Never the less even COBOL had a sense of this.
Looking forward to where Eve goes next. My ideal environment is one where I can freely mix hand-drawn "symbols" to data and express relations using familiar mathematical notations in a single free-form canvas. I sort of do this on paper presently but I have to translate this artifact into a programming language to insert values and explore the various dimensions. It'd be nice to have a system that could recognize my notations and let me hook them up to data.
[0] https://www.wolfram.com/programming-lab/
update
Well... not quite like Wolfram Language; but the connection to knowledge, queries, and building new knowledge from queries. The interface is quite flexible too.
This is why I like it so much, even if it is commercial. After all I used to pay for my compilers.
https://github.com/witheve/Eve
And there's a blog post describing the features of version 0.
TypeScript 69.2%, Rust 9.9%, C# 8.9%, CSS 5.9%, HTML 3.8%, JavaScript 1.4%, Shell 0.9%
I'm in the arts n' crafts camp. Craftily building up abstractions and playing them off each other -over time- seems to be what we're doing here.
I think any program representation should fit your use case. If Excel or SQL fits for data simulations, use that to program, if C++ and Assembler fits for your systems coding, use that, etc...
I'm not a fan of CASE tools, BPEL, instarepl, naked objects, or any other visual or visual/textual modes of coding for general programming work. I agree with the scalability and productivity issues others have mentioned with visual programming systems. Visual probably has a relevant place somewhere in effective software development, but they're just not general purpose enough because the visual bits and bobs tend to muddle the abstractions and are generally too fiddly.
I intuitively agree/sympathize with the myth that less lines of code == less bugs. I think we should seek for languages and libraries that require less lines of code to get stuff done.
Before we can bravely board our starships and chat with our sentient computers in order to program them, we'll need people to write less buggy code with existing systems. For me, that requires:
* TDD which allows everyone to verify their work
* Employing Agile teams to encourage robust communication and to reduce YAGNI, and
* Establishing software architectures where tools and libraries are chosen -primarily- for their ability to promote developer productivity
The programming world would look much different (for the better) if Lisp Machines had succeed instead of UNIX workstations.
However, since these systems have been shown to be somewhat limited by the boundaries of efficient computation or even computability in general I'm hard to convince on this.
I think the only way they can truly convince people that this is feasible is by implementing the system in itself.
I don't think I have to summon Lisp to illustrate my point in a Hacker News Thread...
Likewise, iOS is one biggest changes in the last 10 years, but based on old technology.
For that to come, a paradigm shift would need to happen. Now what could cause that shift? In my opinion its blockchain technology.
I think one of the platforms which allow to run Code on blockchain will reach critical mass, here is a nice intro talk to one of them: https://www.youtube.com/watch?v=U_LK0t_qaPo
You may ask but what about my beloved javascript, c++, python or java? Well there is plenty of that used in the implementation of ethereum. And they even try to make their language look like javascript a lot ( look for solidity) to attract developers. But those are just aesthetics which come due to bootstrapping.
And bootstrapping is THE key issue here which eve doesn't even begin to deal with, which would deal with interesting questions like:
How many programs/functions/snippets would a blockchain need to become "liveable" by itself? 10000? 100000? more?
People today think but I do this great "app" and people cando such cool things with it. But the point is, very much as Alan Kay also says, apps are just the wrong thought. Appstores are a JOKE for the end user, you can buy what you want , genius, but can you combine two programms to make it do soemthing interesting? No you can't, you would need files and formats , but ahh.. that war has been lost way long ago. We need maximal flexible programs, code, snippets. Write something on your own, include something here, send a message there and so on. Something like a giant gloabal object or actor space.
Digital Identity, no user EVER wants to log in, remember? So just get that working, it can be solved with the crypto infrastructure in place. Scaling to millions of users? Scaling to billion of users?
Most of the solutions, which try something today, including eve, try to do something which is based on a webbrowser. But the point of these "future" things should be how to bootstrap OFF the current browser (technical AND content wise) mess we have today.
So no I'dont see eve giving anything of interest, I don't see it capable to grow into a network. If eve tries to "solve" programming I can't see it, I made some notes about other interesting approaches here: https://gist.github.com/AndreasS2501/30ffda6d10b1c7ac53b1
o/
I'd really like to know what you think about our alternative to to something like Eve, but less low-level.
Well suited to typical database apps and workflows.
I'd really appreciate any feedback.
We've currently positioned it as a "Build your own CRM" but you can use it build any process.
There's a partial user manual here https://taskputty.com/user-manual
Which will give you good idea of the flavor of TaskPutty.
It's not for "build any kind of app" (like Eve is trying), just a fast way to automate any typical business process, drag and drop your own custom forms, drag and drop some rule logic.
Could also be a really nice CRM or process automation tool for any startup.
I'm mostly curious, do you guys think we should persist and build it out, or do you think we should pivot?
At the moment, it is positioned as a "Zero-code" tool.
We've applied several times to YC with no luck. We get a couple of signups every month, and have a hand full of paying users.
What do you guys think: full steam ahead, kill it, or pivot??
Andries