How an Engineering Company Chose to Migrate to D
dlang.org
dlang.org
The need for nested functions bugged me as well. That seems like an odd deal-breaker and hints at code smell, rather than a genuine technical need. If it's just for the sake of one-to-one transpilation, then I can see the point, but a refactoring exercise seems like a better effort to make than language migration. (Also, don't Rust and Go have closures? Surely that would satisfy this requirement.)
I think the author makes it clear that direct conversion from Pascal is the primary goal. I'm guessing that once the codebase is fully transpiled to D, refactoring would be the logical next step.
He mentioned an awful lot of code that would have to be refactored, and it could be that they are just too far down the rabbit hole to get themselves out.
I have never seen a reasoned argument why C++ vs Fortran is better for technical programming.
That solution is probably the lowest cost / safest course... Assuming the Extended Pascal compiler they have supports it.
But I suspect choosing D was a fait accompli.
Transpiling is normally a little risky. Transpiling with a custom transpiler over ~500KLOC is downright terrifying.
I'd be concerned that the cure is worse than the disease.
I understand the allure of this solution. Bringing 30ish years of code into the modern era with a single piece of code sounds like a huge win. But you owe it to your employer (who seems really gracious and trusting) to ask:
1. Are there edge cases? Programmers are a clever bunch, and you can never be too sure about what features/bugs they've used/abused!
2. How will success of the transpile be determined? I'd use unit tests, but that'd take a lot of unit tests!
3. What % of that code path is critical? Can you get away with ensuring the functionality of a subset of the codebase? If so, could you just manually port that?
4. "Blaming the compiler" will be a reasonable excuse for any problem over the next year or two after deployment. I guarantee you'll be spending time proving that other peoples' mistakes are not your fault. How much flak are you and your boss going to take for this?
5. Since the code can be linked, might a more popular language (that will be known by a wider community) with a slow porting plan be reasonable?
6. Python is commonly used with scientific computing. Why does this case preclude it? If you are doing mostly numeric processing, that gets directly passed to the hardware anyway!
You have been given a rare chance to pay down technical debt. From what I understand of your solution, I believe the bespoke transpiler and semi-known language will ultimately increase it. And I know you love D, but if I were brought in as a consultant, I'd be recommending the .dll/.so solution with Python. If something is really time sensitive, use Python binding to work with other languages (avoiding premature optimization).
Even if myself or my fellow posters made a point that resonated, it might be too late for such discussions. I can't weigh the political costs for changing strategy now. So I'll wish you good luck, and hope that Extended Python doesn't have too many sharp edges for you to contend with. I hope to see a postmortem a few years from now detailing the outcome of this plan.
The solution of using a HLL and dropping down to C for performance strikes me as overly optimistic: the only language where one can legitimately claim that it's pleasant to use C for extra performance is Objective-C. Anything else will be different degrees of painful.
I didn't get the solution of using a dynamic lib and calling/extending that from Python. One would end up with 500kloc of Pascal and a Python one liner. All the work still happens in Pascal.
Having binary compatibility with the old version would be a great safety net while transitioning to another language. The main weakness of the author's plan, as the article explains it, is that they will be dealing with a lot of newly generated code at once (as well as an unknown number of unknowns). The plan would be fine if it was only ~10kloc, but ~500kloc is a very different beast!
Come to think about it... It would be wise to have a "Plan B" still written in D, but linked to the binary rather than transpiled. That way if anything goes wrong after deployment, users can temporarily use that rather than revert to Extended Pascal.
There probably isn't an easy solution for this problem. But maybe the author has found a silver bullet.
I brought up Python because it sounds like SARC is adjacent to scientific computing, and it is likely they'd be able to pick up physics/math/engineering majors that are already familiar with Python. Plus, I bet they'd be able to use some really good Python libraries (possibly replacing some of that 500kloc custom code with open source libraries).
The article makes it clear that naval architects (who are not expected to learn C++) are programming, so I doubt high performance computing is required in 100% of the code (at least I hope not). If there are genuine performance issues, tools alone won't save you, you need to hire programmers that know how to optimize (and they'll be fine writing C/C++ libraries with interfaces to Python).
I strongly suspect the justification that interpreted languages don't meet performance requirements is more ideological than empirical.
Besides, it's likely the existing binary will run faster than its transpiled descendant!
Nope, it is based on production code life experience and seeing everything crumble with high CPU usage for little work being delivered.
Any programming language without a JIT or AOT as part of their default toolchain is not worth using for anything besides scripting.
> Besides, it's likely the existing binary will run faster than its transpiled descendant!
Given how old Extended Pascal compilers are and that D has three backends, two of them based on GCC and LLVM respectively, I pretty much doubt it.
2. We ourselves are heavy users of our software, and are likely to be the first to discover "oddities". More serious are subtle numeric differences, and we'll have to create tests against that. One approach to validating the transpiler is instrumenting the code. We could write a Pascal-to-Pascal transpiler that inserts extra code, for example logging the return value of every function. If we do the same for the Pascal-to-D transpiler, and do an identical run, we can compare the logs. And variations of that approach.
3. If a small percentage is unproportionally hard to translate mechanically, we can isolate that and translate manually.
4. I fail to see what you are getting at.
5. Of course we can link code. We think the community is wide enough for us.
6. You mean it gets passed to C.
Thanks for the suggestion. We have a working dual language situation with C++, we don't see that as an attractive option for our other tools. Why would Python be better than D?
What % of the codebase are you familiar with? What % of the code is known by your team? If you had multiple, conflicting errors pop up at once, how much effort would it take to isolate them (not fix, just identify). What about in a section where no one has worked in a long time (assuming such a thing exists)?
Like any old codebase, I'm sure there are functions that have "just worked" for years (or in this case, decades). With a single action, every piece of code that's been proven solid over time is about to be suspect.
For point two, I think you have a lot more confidence in your ability to detect problems than I would in your shoes.
For point three, I'd much rather manually port necessary code piece by piece over a period of years. Unit testing like crazy while doing so. Thanks to linking, this is an option... One I'd recommend.
Point four is similar to majormajor's comment about the aftermath of the switch. I'm attempting to project beyond the immediate... This port could become a whipping boy... That could bleed into political fallout. No matter what technology you introduce, people will complain. Doubly so if problems arise. Adding a custom transpiler on top of that increases the risk significantly. The easiest transition path to a ${NEW_LANGUAGE} is through linking.
I'm not sure about Python vs. D. My instincts are towards the former, especially if I'm supporting SEM (STEM minus the T) grads as programmers. I also wouldn't easily write off interpreted languages (I'd rather work with them when possible).
I think you have your heart set on this path. But losing the binary safety net in favour of a bespoke transpiler is not a gamble I'd make with large amounts of production code...
Good luck!
I don’t think it’s solely dependent on having afforded formal training.
Knowing enough to know that you might not know and just might need to measure or look something up and caring enough to do so is more of the core of it.
Indeed they do. It's completely unclear why they don't satisfy the OP's requirement.
x = func blah() { ... }
If that's enough to throw your translation process, you've got bigger problems....
(Cards on the table, D is probably a good choice and I'm not advocating for Go here.)
It's unclear where the root of that confidence lies. The translation is just as mechanical in either language as it would be otherwise: see an inner function, replace it with the one thing that can capture variables. Speaking for Rust, it would actually be even easier to do the translation using closures, because Rust closures can have their signatures inferred whereas Rust functions cannot.
- Rust closures don't really support recursion, at least without serious hacks.
- Rust closures borrow the variables they reference as soon as the closure is created; you can never write to those variables again as long as the closure is alive, and if the closure mutates the variable (resulting in a mutable borrow), you can't read from them either, outside the closure itself.
The latter isn't as a big a deal for some other use cases for closures, but if you want to use them to emulate nested functions, you're probably creating the closure at the beginning of the outer function and keeping it in scope for the whole function; this means that what you can do with mutable variables is severely limited.
Both issues could be worked around if the mechanical translator liberally sprinkled the code with Cell/RefCell – which would also be necessary to deal with pointers in the source language. But the result would be highly unidiomatic and verbose Rust code.
(In any case, I'm not sure I'd have recommended Rust to OP as they claim to put a high priority on fast compile times, but that's a separate issue.)
"In Rust, only lambdas (unnamed functions) can be closures".
The OP's post seems to show they didn't spend much time understanding whether Go closures could serve their need. Although from my own superficial understanding, D is "closer" to Pascal than Go.
Now you understand how all of these "How we chose this language" writeups work.
There is a reason why COBOL still exists, for example, what with its ability to ensure accuracy out to 38 digits. Nothing else comes close w/o tons of extra crap libraries, questionable code mangling, and TRUST. Banks trust COBOL because it has an almost 60-year history of trust.
When kids get all shiny-eyed over golang or Rust or any other "new" language or tool and think it would be a good fit in the financial arena, I start to get a little nervous.
.NET, Java are good COBOL replacements.
http://www.fujitsu.com/global/products/software/developer-to...
Not even close.
As for the links, those are still about COBOL the language.
Not even that COBOL codebases could be bridged and work "left as is" alongside those.
The main objection was with the word "good".
This statement is meaningless without additional information.
> COBOL has no decent replacements.
Huge pool of programmers. Huge pool of libraries. Tons of existing code that should continue to run. Excellent documentation. Very fast compilers. Debuggers and profilers a plenty. Top notch vendor support. Choices of IDEs. Works great in embedded. Predictable.
The alternatives being what, e.g. Rust, a 5 year old language with a single implementation that still tries to find its place, and brings extra baggage to the table?
>Not take any risk but stick with tried and wrong tools.
Engineers don't take risks. You wouldn't want risks in the people building your bridges and planes, why would you want in your OSes and network infrastructure?
C++ doesn't though, and that's why traditional C projects switch. At least in automotive it seems to be the language of choice.
I've worked on projects where people had gripes about the codebase and/or language, but they often couldn't agree on which language to move to. So a single person taking over based on their own preference is either a bug or a feature: as a bug, it's ignoring everyone else's preferences and disagreements, and possibly moving to something the team is even less happy with. As a feature, it's making a decision instead of endlessly trying to find a perfect decision, and at least ending up somewhere better off than before.
In this case, it sounds like it could be the positive sort of change if only because they were starting from a basically-dead language. It sounds like without that single person's initial effort, they may have not started any effort to migrate to anything.
But this gives me major pause: "My colleagues will continue to develop in Extended Pascal as usual, and once my transpiler is able to translate all or almost all of it, we will make the switch to D overnight." I'd love to fast-forward to see what happens the days after that... and how long that takes to get to, for that matter!
When talking about turing complete languages, everything is an ergonomic nice-to-have. All language features aspire to be so nice to have, that you come back around and call it a necessity. But you can still express a program without them, unless you strip out so much as to lose turing completeness.
(I'm generally pretty skeptical of the whole "write down your requirements with no awareness of available solutions and then choose a solution that meets them" approach to decision-making).
> (I'm generally pretty skeptical of the whole "write down your requirements with no awareness of available solutions and then choose a solution that meets them" approach to decision-making).
I also don't understand this. If there's a solution that meets all of the requirements what does it matter what there are other solutions? I mean, yeah, you be doing a local vs. global optimization, but as long it meets the requirements it's probably not a big deal. (The only exception is if the requirements are so broad as to be meaningless, but as long as they're reasonably specific, I don't see the problem.)
Rust does support nested functions that don't close over their environment, and has a separate syntax for closures which do, and closures can either move or borrow their captured variables. This makes it a bit harder to write a translator that produces good, idiomatic results; you could always use a closure that borrows the closed over variables, which I think matches the Pascal semantics, but that would mean the code would look a little odd for cases in which you had a nested function just to factor out some functionality and didn't need the closure.
fn main() {
let a = 10;
fn function() {
println!("Hi!");
// println!("a is {}", a)
// fails with:
// can't capture dynamic environment in a fn item
// use the `|| { ... }` closure form instead
}
let borrow_closure = || println!("a is: {}", a);
let move_closure = move || println!("a is still: {}", a);
function();
borrow_closure();
move_closure();
}My experience as a language designer is that syntax matters, and leveraging the users' existing knowledge is worthwhile.
Nested functions were a very common pattern on languages like Pascal, Modula or oberon. They are equivalent to code blocks inside a function in C.
Maybe I've misunderstood, but aren't Rust's closures [1] exactly what the author is describing? Specifically, you can capture variables from the environment into functions.
[1] https://doc.rust-lang.org/book/first-edition/closures.html
Here's a Rust version that does the same (without heap allocation!): https://play.rust-lang.org/?gist=bb5406062970b62088d58fb8a9f...
I think the real nastiness happens if you want to have more than one closure over the same variables, when those variables aren't Copy. Because then when you try to `move` them in, the compiler yells at you. At that point you have to resort to putting them in an `Arc<Mutex>` or something like that. But that's par for the course with Rust: the compiler is preventing you from creating multiple mutable references to the same object.
Once "non-lexical lifetimes" lands, the inner scope won't be necessary. That may have tripped the OP up.
[1] https://users.rust-lang.org/t/inner-functions-not-closed-ove...
This is achieved by adding a hidden parameter that is a pointer to the enclosing function's stack frame. This forms a threaded list so variables in nested enclosing functions can be accessed by walking that list.
This is the same way Pascal does it.
It also means that taking the address of a nested function produces a pair - a pointer to the function, and this hidden parameter to the enclosing stack frame.
The environment is a struct containing what is captured; so for example, it would contain an &T if your closure captures a T by reference. You don’t need to walk a set of stack pointers to find it, just follow the reference directly there.
If it is the enclosing function, it's just a single pointer to it. The "walking" comes from if you're accessing the enclosing function's enclosing function's frame.
* http://jdebp.info./FGA/metaware-iterator-driven-for.html
In theory, the extra frame pointer mechanisms of the ENTER and LEAVE instructions in the Intel ISA were there for supporting this kind of nested function. MetaWare High C/C++ did not use them, preferring to use a variant calling convention where (if I recall correctly) ECX contained the extra hidden parameter.
ENTER copies M words from the caller's stack frame. The pointer passed by MetaWare High C/C++ in the register could equally well have been placed at the relevant position in the caller's stack frame, just as other arguments are, and an ENTER N,1 used as part of the perilogue.
The only consideration here is speed of the ENTER instruction versus having the hidden pointer in a register, not that this sort of nested function implementation is somehow impossible with ENTER. It's just another case of ENTER is slow and other instructions do the same thing better.
* http://jdebp.info./FGA/function-perilogues.html#Standardx86
(Part of that consideration is making non-nested functions usable via full function pointers.)
Generally, I prefer mainstream languages myself, that’s why I mostly code C++ and C#. But “generally” is the key here. For some projects, or some parts of it, picking a non-mainstream language can be a huge win in terms of productivity.
I never used D in production, but I can remember several occasions where I picked unusual languages, both specialized and general purpose ones. Among others, I’ve used XSLT, sed, awk, VBScript, VB.NET, R, power shell, AutoLISP, MATLAB.
Theoretically all modern programming languages are equivalent because Turing-complete. Practically, for a particular problem, it can be orders of magnitude difference in development time between them. The reasons include availability of specific libraries, integration with other code be it third-party or legacy, or specific runtime features.
I'm not surprised to see criticism for D on HN. While reading HN comments on posts related to D, I've always found pro Rust/Go comments.
1)Expressiveness in the given problem domain. Some languages are really great in particular niches.
2)Availability of devs. The ability to hire and retain a really great team makes or breaks projects, and while a smart, motivated dev can learn any language, it's definitely a factor in their choice to work or not work with you. If you're in a niche language it will greatly affect the ability of people to use external resources (stackoverflow etc) if they hit blockers and some people will be reluctant to railroad their career into a tech that doesn't appear to be going to the mainstream. On the flipside, if you're lucky you may be able to get really passionate devotees of the language to work for you just to use the language where they might not have done it otherwise.
With all that said... Lisp' siren call is still out there, and experience tells me it will be the same. (As it was once before)
Since then, I do by and large dive into new languages but don't use them. What I do is bring back home some of the practices which I think are of benefit to me.
This was 15 years ago, but I honestly don't know what language I would use today. It's funny but sometimes binary data handling isn't a particular language's strong suit simply because they don't have the primitives to handle it, or if they do, the APIs to handle them are awkward or unnecessarily verbose. It's kinda neat that in C/C++ you can take a 1MB byte buffer and pass it to a function starting at 30% in. In other languages its like you have to pass in the whole thing and another parameter to tell it where to start. I get that's why C/C++ is in the situation it's in (security - but that's why I don't use C/C++ to handle web requests), but sometimes tasks need that.
Could make use of a saner type system, nice frameworks (Turbo Vision, OWL), while being able to painless create safe wrappers around C libraries.
> motivated dev can learn any language
The key word is "motivated". That's why I'm an expert user of C++98 and D but I can't learn Go, Python, C++11, etc.
> able to get really passionate devotees of the language
That.
I lol'd at this
But I've found that lots of places seem to be focused on if you know X rather than being a well versed, developer who enjoys growth. And it feels more and more like the reason is because they don't expect anyone to stay there for more than a few years tops.
I feel that splitting as much as possible the project into chunks/components and then replacing them one by one sounds more realistic (with the appropriate test harness). Maybe that is the plan but it is not touched in the post.
And then maybe an issue that is not that bad, but you are going to have a hard time comparing old_codebase with new_codebase once you've switched. If you have to maintain old branches, porting the fix to recent branches will be more time consuming that simply do a merge.
It sounds like the author has never transpiled a large code base into D before.
I think the Go compiler did something similar.
It certainly requires iteration of continuously attempting the translation until it works, and fixing bugs either in the transpiler or in the original codebase. But once everything seems to be in order, you can do the transition all at once.
My hobby project languages or choice are currently D and Kotlin. I prefer the semantics and flexibility of D, but Kotlin has the backings of the tried and tested Java ecosystem, which is hard to beat when you just want to get down to it. Also web servers in Java are vastly superior at the moment. Vert.x is so very fast.
https://dlang.org/blog/2018/06/11/dasbetterc-converting-make...
I'm really sad to see that the author didn't even consider the Nim programming language. They could have saved themselves a LOT of work, Nim's original compiler was written in Pascal and then translated to Nim using a `pas2nim` tool that is freely available on GitHub[1].
Out of Go, Rust and D, Nim is most similar to Pascal by a long shot. Did the author simply miss Nim completely or did they dismiss it for another reason? It seems to me like it would be absolutely perfect for their project.
I'm curious what manual translation and rewriting would be necessary when moving to Free Pascal compared to their Pascal2D transpiler and compatibility library.
I love nested functions. It's good when you need a quick local function, and don't want to expose it to the rest of the code base.
But, how do you systematically unit test nested functions?
You expose it.
If you want to test a local nested function, you test the enclosing function.
You don't. You test the enclosing function. The nested function is just an implementation detail like any other line of code in that function.
Why, with nested unittests, of course! Just put them in a nested type first:
void main()
{
static int nested(int x)
{
return x * 2;
}
struct Test
{
unittest
{
assert(nested(2) == 4);
}
}
}
Shameless plug/pre-alpha demo: https://www.c2rust.com
Edit: missing word.
Edit: reflect resolved status of issue.
While variables which were declared prior to it in its outer function were available to its inner functions.
Unless there is an obvious example of where this breaks down in Go and Rust, I’d certainly like to know how.
Also I inagine the qt bindings are better in D.
To me, the evaluation goes solely as ecosystem vs. ecosystem
There D is ahead not so much by number of features, than by lack of showstopper flaws
* D has generally better metaprogramming, however, rust has real macros
* D has a garbage collector and defaults to code being memory-unsafe. Rust uses ownership semantics and defaults to safe code.
* Rust has only an llvm-based compiler (although possibly moving to a self-hosted one soon?). D has a self-hosted, gcc-based, and llvm-based compiler.
But a killer feature that's available in C++, but not available in D is pure RAII idiom. Sure D does scoped destruction (via struct's destructor), but is in no way equal to the contructor+destructor combination in C++. struct's in D don't have default constructor, so RAII in D is just incomplete.
Automem[1] is a boon for D, but is useful only for heap memory resource. Not for all kinds of resources (like locks, etc).
On the other hand D has scope exit/failure, which is not available in C++.
[1] https://dlang.org/blog/2017/04/28/automem-hands-free-raii-fo...
Surely Go has this?
func main() { a := 1 func() { b := a g := func() { c := b fmt.Println("c is", c) a = 7 } g() }() fmt.Println("a is now", a) }
The article title is misleading. They still have not migrated.
I mean, they have chosen to migrate. That doesn't imply they're done with the implementation.
(Edited to be less glib. bad habit)
It's not that they're not capable, but some will not want to risk several years of their career to learn it.
I could see a C++ dev giving this a try without hurting their career prospects, but C# devs are not exactly known for trying out non-mainstream technologies. If they switch for a couple of years MS will pump out a couple of frameworks and .NET programming languages. Fire and motion as such.
(that may not be entirely fair, i guess i just tend to be skeptical of people who get hung up on languages.)
It’s a legit concern.
Airbnb dropping react native due to a lack of qualified candidates that were articulate in both the native and javascript realm is a great example. Something can work flawlessly with the right conditions (for instance, developers that understand and can implement the path of least resistance across native and react), yet if its prohibitive to create those conditions you are back where you started.
If the dominant ideology among developers is that learning X language or technology costs more of their career than its worth, it dies out. Same goes for the company and their resources.
Erlang has been around in a niche role for >30 years and still going strong, even stronger now that Elixir has reinvigorated interest in the platform.
Quite the opposite: The new tool saves time and money.
Airbnb example just shows that one should not jump on the newest thing on the market.
Yes, they also had difficulties hiring devs which know both native and react, but that's normal. Most mobile devs wouldn't touch react.
Either the company focus solely on finding enthusiasts or they pay the premium to overcome this.
I was assuming a more rational choice but then developing serious technical software in pascal instead of fortran in the 80's was not a rational choice in the first place.
There are many companies (e.g. Weka.IO, Sociomantic, etc.) that hire C++ developers and have them happily write D code. I did work at Weka.IO for a while and never met a single D programmer that was unhappy.
Ali Çehreli