I am afraid to inform you that you have built a compiler (2022)
rachit.pl
rachit.pl
Perhaps burned by experience, one time I implemented a mini-language for specifying some business logic that I knew—just knew—that our client would change his mind on a dozen times and not understand the half of the ramifications of his requests and would only arrive at the solution he really wanted by trial-and-error. Was the little language I made as complex as a "real" compiler? Goodness no. Was I happy to have a flexible language-based solution? True to my predictions, I did get many, many logic change request and handled them with ease. Yes, I was very happy after that.
In my experience, most developers-as-consultants aren’t good at that. So, when we’re (rightfully) late, we probably reduce our margins. When we do a great job, we don’t make the client pay for his own mistakes.
I'm not exactly sure what you're asking. Should you be charging clients for time that you didn't have to spend because you made some good decisions upfront? That strikes me as odd. If you agreed upfront on the cost for a certain end result, then sure, charge that full agreed-upon amount even if it took you less time than expected. But it seems odd to say "this change only took me an hour, but it would have taken 10 hours if I had made worse choices in designing this system, therefore I'm going to bill you 10 hours." After all, there's always a hypothetical situation where any given change could have taken arbitrarily more time if you had made worse decisions in the past.
Yes! Completing a job or task in 1 week is more valuable than completing it in 5. The lion's share of value we provide as developers and consultants is not output but battle-tested intuitions that let us navigate the giant solution space, so bad decisions we don't make is definitely hard-earned value servicing the client.
A man once interrupted Picasso at his evening meal. Pulling a napkin from his pocket, the man said,
“Could you sketch something for me? I’ll pay you for it. Name your price.”
Picasso took a charcoal pencil from his pocket made a rapid sketch of a goat. It took only a few strokes, yet was unmistakably a Picasso. The man reached out for the napkin, but Picasso did not hand it over. “You owe me $100,000,” he said.
The man was outraged. “$100,000? Why? That took you no more than 30 seconds to draw!”
Picasso crumpled up the napkin and stuffed it into his jacket pocket. “You are wrong,” he said, dismissing the man. “It took me 40 years.”
As a consultant, you are typically billing by the hour (or day) and you will most likely be giving the client some kind of estimate/quote as to how long the job will take, and your rate, and hence the cost to them.
You might very well be able to do the job in one week rather than 5, but it would be dishonest to tell the customer it will take 5, then complete it in 1, but still bill them for 5.
Instead you should tell the customer that it would normally take 5, but you are able to do it in 1, so you are charging 5x (or 3x or whatever) the going rate.
but the problem shows up when the customer doesn't believe you when you say you're 5x faster.
They compare you against another consultant, who might be cheaper, and say that you're just over charging.
Therefore, you should be charging the going rate, but do it 5x faster, and have free time to take on more projects. You could also deliver earlier, and then have a charge for change requests (which invariably come).
This is like saying you'll only pay your brain surgeon for four hours work because that is as long as the operation you require will take. But that four hour operation is built on a lifetime of training and skill.
The value that you provide the customer is the metric from which you should be deriving your billing. If someone else provides an equally effective solution but takes an order of magnitude more time to deliver it, then you are just that much more efficient.
In a lot of consulting gigs, you don't do time-and-material. You do fixed price. So, you invested _your_ time and you took the risk of creating such solution, even when the client sweared that some requirement would never change.
In most contracts, if you, as the contractor, make a mistake, it's up to you to fix it. If you trusted the client, you'd have to charge 10 at any CR instead of 1. So, it makes perfect sense to charge at least 5, or maybe even 10.
(ofc if you're working time&material or in an agile/flexbile fashion where there's trust between client and consultant, this doesn't apply)
The best solution is to have a good deployment process in place that makes it easy for you to make business logic changes using your language of choice, rather than building and maintaining a secondary language that is understood only by you.
No need to create a DSL or compiler or worry about maintenance when you're just using the same tooling as everything else you use.
Seen way too many times in my career developers who build the fun, interesting solution (DSL/compiler/parser) instead of the boring, but far more practical solution of fixing their deployment processes.
The client shouldn't have to know a "full" programming language, but a "sufficiently simple" DSL isn't that much more than a config file format, right? They can learn how to make small changes themselves if the language is easy enough, can't they?
I tried to convince management that this wouldn't work. If nothing else, because our DSL was invented in-house, there was no-where else the client could find answers on how to make the changes they wanted. There would be no hits on Google, no answers on StackOverflow, no random client employee who happened to have some relevant tech knowledge. If they had any questions at all, they'd need to come to us, and we'd end up writing the changes for them anyway. And, as you point out, we already had a perfectly good programming language - the one everything else (including the DSL) was written in.
I should have charged them more than I did.
That's the problem isn't it? Unless it's SaaS you don't want to solve a problem too well...
Suddenly the API looks oddly familiar…. OMG…
In fact I'd argue that it's much harder to not accidentally build something that you don't envision.
maybe our previous works can be externally ad-hoc incorporated into new works by others...
Many years ago I was writing a bunch of modular audio applications that were all interconnected using JACK. So of course I needed some convenient way of storing/restoring the graphs that connected the various components that were always in flux -- each audio experiment had it's own graph, but there were often re-occurring sub-graphs.
So, I started to build a small cli utility that could save the current graph and restore it later... of course because of the common sub-graphs it needed to be able to refer to other labeled graphs so i could easily reference and re-use them... and of course to (save myself time) it also needed to have arithmetic composability so that I could add/subtract existing graphs to create new ones... and of course it would be helpful if I could also annotate the graphs so that they could launch the associated programs...
So in the end I had a generic DSL that had recursive graph parsing, reification (labels), graph composability, arithmetic composability, and could associate arbitrary labels with executable code.
I eventually looked up from my adderal fueled hack-a-thon and realized I had created a half-baked LISP and decided it was time for to go to bed! :)
Long ago I wrote an inspection reporting system, in the days of Turbo Pascal and MS-DOS. I ended up building a domain specific language for the configuration of the system. The configuration files looked almost identical to Pascal source.
The only thing worse than creating a compiler is being unaware that one already exists and creating a new one on top of the existing one.
A year later, there was a team of 5 engineers maintaining a half-baked implementation of gRPC. Good for him. Bad for the company.
But multiply that by a few languages, and now you have to paper over error code and exceptions, signed and unsigned ints, unicode and bytes
Not to mention network errors, retries, throttling, etc.
It tends to grow without bound, mostly because the RPC system tries to do too much. You can't really abstract the network -- that's the #1 lesson
When you try to abstract the network, now you OWN a bunch of problems that you can't solve
In some ways I feel like my impact has been quite boring, in other ways quite vital. But it’s never made me friends with the kind of developers who look sideways at the idea that other peoples life’s work might be better than their 5 year old weekend project.
I still dearly love it. It makes operating a massive MySQL DB that has to deal with terrible queries from an ORM palatable.
Dear sir, you have built a compiler - https://news.ycombinator.com/item?id=29891428 - Jan 2022 (175 comments)
And yet few people think about using a Lisp for their DSL.
You start by just adding some kind of configuration for rules, maybe in JSON. Then you start wenting to make more complex rules so you allow some kids of recursive system in your JSON that can nest rules and combine them. Then you find yourself copypasting rules a lot and so you implement some kind of naming convention so you can reuse rules. Then you realize how disgusting your JSON is getting so you dust off a parsing library and make a basic DSL that compiles into that JSON, and then it dawns on you.
I would imagine that this is just an extension of the above idea about maths. To describe something in such a way that it operates formally, you eventually end up with stuff like LISP, or another equivalent language.
And somewhere else about accidentally building a real time chat service (or was it email? Don’t recall).
Maybe Zawinski's Law?[0]
> Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.
[0] https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law
Every program attempts to expand until it can render HTML. This usually results in it being able to read email.
It started as a few sed commands to merge TeX+code -> TeX for a book project. I ran these sed commands from a makefile. Life was easy.
But then there were complications, and I needed to make slightly more sophisticated substitutions. So the sed commands moved into an awk script, run by the makefile. This was better than maintaining a handful of little commands that were growing on a weekly basis. Life was good.
The transformations I needed kept growing a bunch of little variations, and the awk script became hard to maintain, so I rewrote it in go, with proper parsing and output. (And even unit tests, after the 2nd time I broke some output.) Designing it as almost-a-proper-compiler was 10x better than maintaining an ad hoc script. Life was great, even with the overhead of maintaining a separate processing tool.
Both the book and the system were heroically good.
More likely pl -> perl, I think.
I think interactions between features are very hard to think about.
I think constructed languages have the opportunity to think about potential interactions that would be useful and aim to support those ones.
But there's lots of permutations to features.
Just look at async functions in Rust and coloured functions. It's such as pain.
It also reminds me and brings up thoughts about "the expression problem" [0]
How do you think which combinations of features would be useful upfront? For example: there's interactions between memory management, garbage collection, async, multithreading, coroutines, closures, the stack, FFI. It's all very complicated.
"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
...which often happens when somebody creates a UDP-based protocol and then adds their own reliability and robustness on top of it ;)
And if you don't have that problem then you're probably also reinventing HTTP as well as TCP.
[0] without head-of-line blocking
[1] ideally with controls over how reliable we want our messages to be. Real-time usecases like videoconferencing or multiplayer games tend to fail horribly over TCP.
I've looked into some compiler-like tools (can't remember the specific ones, sorry), and from what I can tell their code generation phase looks very similar to mine in that they use string templates.
This library does exactly what you prescribe. Pretty sure under the hood it's using macros with string templates
- the value that's constructed has to be valid code - macro "hygiene" is maintained
I've only done anything like this once and not regretted it, and it's purely visual scripting, if this then that style. Anything that can't be handled by an event that triggers a list of actions, then stops when one returns False, I will hardcore a hack just for that feature.
Most actions and triggers are responses to specific use cases.