Using Java 9 Modularization to Ship Zero-Dependency Apps
steveperkins.com
steveperkins.com
1. Setting up a project is still a pain in the butt. Tools like Gradle are a nice improvement over Ant (and some would say maven), but still most people don't even understand them. You have some serious reading ahead of you if you want to set something up that isn't already templated somewhere for you. You can lean on an IDE for sure, and for most Java devs this is probably a no-brainer. I'm weird in that I don't like magic. I prefer to know what the tool is doing on my behalf, and the Java IDE world is so complex that it isn't practical to learn that unless you're in the ecosystem for years. Then dealing with weird exceptions from the JVM can be maddening.
2. The Java world moves slowly. It could reasonably be years before many shops transition to Java 9, when you would actually realize the benefits of this in your work life.
3. So much Java runs on the server side anyway, where executable size and entrypoint doesn't really matter that much. Because of this, it may only be a small subset of Java shops that really get into Java 9/Jigsaw and iron out the bugs, and create tools/tutorials for others.
While Qt is great, we've been running into logistic problems - like we have plenty of tools that use it, and we are forced to place all the binaries into one folder ("bin"), and the Qt dlls there for legal reasons, license, but buying the license for static linking won't be a problem for us reallly. The real problem is that once we link our binaries statically, then our plugins that also use Qt won't work with them, unless we do some magic, and forward all QtXxx.dll end points to be from the main executable, or use more abstract interfaces - e.g. our plugins should not use Qt directly.
As build system, we used to have IncrediBuild, but since we are PS4 developers we get SN-DBS for free - http://www.snsys.com/products/SN-DBS.asp - it speeds up the build, but once the Qt moc/etc rules come in, the build needs to wait for these to be generated, and then proceeds.
There are ways to overcome this, and we are looking into improving this.
I've tried using .pro files, but first, I can't sell them to the rest of the engineers, as pretty much almost everyone knows Visual Studio, .sln/.vcxproj files. There are some ideas about alternative build systems, but not there yet. (at the end most of these just generate .sln/.vcxproj, like CMake, or premake, etc.).
You can statically link with LGPL code, as long as you provide a means to relink (typically, shipping .o files if requested).
https://www.gnu.org/licenses/gpl-faq.html#LGPLStaticVsDynami...
With BAZEL things were working pretty smoothly, but I've never used maven, gradle, ant, etc. Really weird, because If I post that I have 3 years experience with Java and not knowing any of these tools...
What I'm trying to say is that with BAZEL this might be easier for cross platform development (not sure though)...
http://www.red-lang.org/p/about.html
I gather the GUI story on Linux isn't great at present, and there's no iOS support, but all of the other important bases seem to be covered.
I've tinkered with it a bit today, and so far it's as advertised.
Except accessibility for blind people (through screen readers), and probably people with other disabilities as well. I just tried the Red GUI with the NVDA screen reader on Windows, and it's mostly inaccessible.
BTW, this is a problem with nearly every GUI toolkit developed from the ground up (that is, from the host OS's low-level graphics primitives). The exceptions are mostly the big, "bloated" GUI toolkits, like those provided by the major operating systems, the major browser engines (and by extension Electron), Qt, JavaFX (though only on Windows and Mac), and maybe a few others.
Edit: SWT is one of the most accessible options for cross-platform desktop applications. That's mostly because SWT primarily uses the host platforms' native widgets. But it does have some custom controls, and for those, it implements the platform-specific accessibility APIs. I understand that a lot of people don't like the conventional desktop look and feel though.
I find this is not Java specific and many of those other technologies have similarly painful project config, especially as soon as you do something non-trivial.
> The Java world moves slowly.
"Vendoring" the stdlib, which essentially is what is happening here, alleviates that (at the cost of not sharing source, which is usually OK for the JVM in non-security contexts). As for work life vs home life, that is not very Java specific. Many companies hang back on old standards by choice.
I find this is not Java specific and many of those other
technologies have similarly painful project config,
especially as soon as you do something non-trivial.
If I'm doing something that'll never exceed 1KLOC and doesn't depend on any exotic libraries, I can throw together a crate, module, bundle, package, etc in a matter of minutes and then forget about project configuration. Java... anything other than "local-machine CLI project with no dependencies and an IDE" will have you spending more time on config than code.
Dependencies and portability were by far the biggest issues. eclipse, netbeans, idea, et al didn't seem to do dependencies at all; even if you had a library jar actually on your filesystem it was a huge PITA to point at it, and getting a project that'd also compile on someone else's machine was horrible. leiningen didn't have a single tutorial that I could find and I voodoo'd my way through a clojure tutorial before giving up. maven broke in strange ways when I tried pulling in deps and figuring out the correct incantations in the pom files was a pain. More recently, I could probably have gotten along with sbt if I'd had a week to play with it, but it also has the "no useful tutorials" problem and I was on a time limit.
Some of this is certainly lack of experience, or even simple Doin It Rong. I could see maven being much easier with a directory full of examples to look at, for example, and sbt I'm sure is great once you wrap your head around it. I just feel like, even if I was limited by typing bandwidth instead of figuring-things-out bandwidth, those tools wouldn't hold a candle to python's or rust's tooling. Even makefiles and cmake seem to have better happy paths.
I'll look at gradle next time. There are situation where I could use a jar.
These things are oriented towards working with a dependency management/project management thing rather than manually specifying libraries. I'd really recommend maven, using the IDE's integration with it, particularly if you're getting started - Gradle can be more concise but it's definitely buggier and less well documented/supported.
> maven broke in strange ways when I tried pulling in deps and figuring out the correct incantations in the pom files was a pain
Dependencies should just work unless you were trying to do something weird like using jars from the local filesystem - don't do that (if you really must, install them into the local maven repository and use them from there: https://maven.apache.org/guides/mini/guide-3rd-party-jars-lo... ). You shouldn't need any incantations, just simple XML that should autocomplete in the IDE, or you can use a GUI rather than directly - e.g. in eclipse you just right click on the project -> maven -> add dependency, and search/select the library you want. Do you remember what problem you had more specifically?
> Some of this is certainly lack of experience, or even simple Doin It Rong. I could see maven being much easier with a directory full of examples to look at, for example, and sbt I'm sure is great once you wrap your head around it. I just feel like, even if I was limited by typing bandwidth instead of figuring-things-out bandwidth, those tools wouldn't hold a candle to python's or rust's tooling. Even makefiles and cmake seem to have better happy paths.
SBT is awful throughout, but my experience is that Maven is much much better than anything in Python-land (no messing around and mode switching with virtualenv, no need to separately run pip, one consistent tool that everyone uses, one packaging format, one central repository, better integration of testing with the project), and directly comparable to Cargo.
How do I get a preconfigured quickstart project for say iron and diesel with rust? I read the rust book. I must have missed it.
The book doesn't cover everything. That said, we currently don't have a built-in solution for this feature; we had one but it never hit stable. Since it didn't get accepted before the impl period late last year, the discussion was effectively on-hold; it can get started up again now that it's past the new year.
That said, many projects provide their own way to get started. Diesel's CLI has a "diesel setup" command to get going with it, for example. I'm not aware of one for Iron.
Java 9 is short term support release, ending support with the release of Java 10 / Java 18.3 (March 2018). Next LTS JVM release is supposed to be 11 in 1.5 - 2 down the road.
A lof to large organizations will be holding back for the next LTS release before considering migration.
I guess you could argue this might allow Java on the Desktop to be feasible again, but it might be too late.
I surely get to use it more than Electron apps, thankfully.
Many coworkers I've worked with have called it complicated and then I find out they really haven't learned the (very simple) lifecycle and how build behavior gets attached, etc. This is all very simple but it is something to learn.
I don't know why this happens. I think programmers are so used to imperative (ant/Grunt/etc/etc) programming/building that it becomes confusing for them when they encounter a declarative system. And their brain revolts. Because they are like how do I just do a thing (i.e., copy some file somewhere, call a "build" method, etc, etc.)!? Instead of realizing the value of the declarative + idiomatic Tao and learning it...they say no, no gimme gimme let meee do it. Much like my two year old. And then I'm left fighting non-reproducible builds and getting their projects going in my IDE like it's somehow 2005 again. But Maven is great, declarative systems -- especially declarative build systems are the optimal way to do builds (in most cases).
Because, after all is said and done, programming is about having your computer "do" something - whether it be to write some bytes to the disk or to send some bytes to the network.
It's just that sometimes there's an advantage to use an abstraction - such as say "SELECT * FROM TABLE WHERE ROW_ID =14" is an abstraction over something like "for (int i =0;i<table.size();i++) if(table.row[i].elem[1] == 14) ... ", but often you don't need such a complicated abstraction, you just "need something done" before (or after) building. Whether it be that you want to download an HTML file to package, or to move the resulting file to a certain directory, or to clean a certain directory, your build language shouldn't be that arcane that you have to create a "build.sh" to get things done.
— Maven is simple, not arcane. I’ve produced countless “custom” builds with wild variations of need and never has Maven let me down. In some sense this is easy to prove (see maven-antrun-plugin); however, Maven’s core plugin/idioms supports 90+% of use cases I’ve seen over many many “weird” build requirements. To your example, “copying files” is fundamental; however you have to learn the idiomatic way to do it if you want to reap the benefits. Otherwise, just call out to maven-antrun- or similar. But then you’re chasing your tail.
The issue with convention over configuration is that you don't see how things should happen, because they're hidden behind the convention.
For example, what are the steps behind your Android build? Can you tell me exactly what steps gradle takes when you run gradlew assembleDebug?
So if for whatever reason you need to modify something, you're completely on your own.
$ gradlew -m assembleDebug
Learn the tools you use..
Not always. Often, particularly in scientific computing, you just want to ... compute ... a result. I mean sure, the computer needs to display the result to you, but the computation isn't about displaying it, it's about computing it. Just like adding up a bunch of numbers on a calculator isn't about getting the calculator to "do" anything.
> often you don't need such a complicated abstraction, you just "need something done" before (or after) building.
Abstractions should be simpler than the thing they abstract over, that's their whole point. Treating my computer's memory as a flat array of numbers is simpler than thinking about how much current is flowing through each RAM wire. "Compile all source files" is simpler than "execute javac Foo.java, then execute javac Bar.java, then..."
> Whether it be that you want to download an HTML file to package, or to move the resulting file to a certain directory, or to clean a certain directory, your build language shouldn't be that arcane that you have to create a "build.sh" to get things done.
You shouldn't need a build.sh, agreed, but nor should you need particular things to happen to particular directories. Maven has great support for things like "package my documentation in the standard way". It's only when people try to fight it about the low-level details that they really shouldn't be caring about that they run into trouble.
For example, if you have a multi-pom project with a parent pom (not unusual in a medium-sized codebase, never mind large), and you have unit tests in your child poms, they'll get executed with dependencies resolved locally for that child pom, and not with the global dependency resolution that satisfies all constraints that need satisfying at the parent level.
The upshot is that you easily end up running tests using different versions of dependencies than what you deploy with, misleading you into thinking things are working, and can end up with hilarious NoSuchMethodError at runtime.
The "solution" is to pin your versions using dependencyManagement, but that's no real solution at all, not least because it's neither automatic (though dependency convergance using Maven Enforcer helps, and I've developed some scripts that help further) nor is there an easy route forward when you want to bump versions safely.
As it should, otherwise you'd get the even more hilarious situation where your tests pass when you run the single-module build but fail when you run the multi-module build.
> The "solution" is to pin your versions using dependencyManagement, but that's no real solution at all, not least because it's neither automatic (though dependency convergance using Maven Enforcer helps, and I've developed some scripts that help further) nor is there an easy route forward when you want to bump versions safely.
This is not a maven issue, it's a combination of Java's flat classpath and the reality of diamond dependencies on versioned libraries in any system. We could allow multiple versions of a transitive dependency on the classpath but then you get Rust's hilarious "expected Foo but was Foo" type errors. Java 9 modularisation should reduce the problem by allowing modules to have explicitly private dependencies, but the problem exists everywhere.
This isn't true; there are better solutions already existing. E.g. Ruby with the Gemfile.lock approach. I'd like to be able to do a two-step: do a global analysis of dependencies and pin the common versions for diamond situations, but not need to maintain or calculate those pinned versions manually. And I still get to use the pinned versions from the global resolution for executing tests etc. on submodules.
As it is, I've had to write a bunch of tools to rewrite the dependencyManagement as and when required. The main app I work on is linked into the Hadoop ecosystem, and that dependency set ends up being quite large.
Another issue with Maven is rebranding: when the groupId of an artifact changes, but the package and class names don't; see e.g. asm / asm vs org.ow2.asm / asm, or the whole codehaus debacle.
I have spent more than two weeks of my life updating, maintaining, fixing, researching and tooling Maven pom files, and it was almost all around deficiencies in dependency management.
Let's not get started on trying to get builds to work reliably offline, caching maven repositories, etc.
I dislike this for the same reason as before: I find it really valuable that building a single module on its own behaves the exact same way as building that module as part of a multi-module project. If you want to ensure that the tests run against the same version as in the final product, I do that by making eviction an error.
> Another issue with Maven is rebranding: when the groupId of an artifact changes, but the package and class names don't; see e.g. asm / asm vs org.ow2.asm / asm, or the whole codehaus debacle.
Maven has better handling of this than most; it can at least warn/error when this has happened, and if you're using the shade plugin or similar it will warn or error when two jars contain the same classes. Yes it's a problem, but it's much more of a problem in other ecosystems.
> As it is, I've had to write a bunch of tools to rewrite the dependencyManagement as and when required. The main app I work on is linked into the Hadoop ecosystem, and that dependency set ends up being quite large.
There are standardized plugins for the common use cases e.g. "mvn versions:use-latest-releases". Most things that can be automated are. But since most of the library ecosystem predates version ranges and semVer, there simply isn't an automated way to know when using a newer version of a transitive dependency is safe or not.
Building is a simple concept: dependency DAG + inputs -> artifacts. The "->" here can involve arbitrary intermediate artifacts and steps.
Maven ignores this fundamental concept of building. Rather than treat interim dependencies as part of the DAG, there's a hard-coded recipe of lifecycle steps.
This is why I personally cringe at maven - it has nothing to do with the fact that it's declarative (virtually every build system is largely declarative). It has everything to do with hard-coding one particular set of build steps into a supposedly general build tool.
With Maven, if you follow Maven's way of doing things, everything just works. But, if you need to do something different, or have some rare use case Maven's designers weren't focused on, you start struggling a lot to get Maven to do what you want.
Gradle is a lot more flexible and if you have some uncommon requirement you are far more likely to successfully implement it using Gradle than using Maven.
You'll be fine as long as you realise the Maven-ey way of doing the thing is to write your own plugin.
But most of the time people just want their build to do something different for no real reason, IME.
A short while later I was wondering why I'd never tried that before: Maven plugins are really quite simple to write.
This assumption is why developers are abandoning Java in troves with the serverless trend, where exe size and init time matter immensely.
And good riddance!
>You have some serious reading ahead of you if you want to set something up that isn't already templated somewhere for you.
Start with a maven project,
$cd /path/to/maven/project/
Create your archetype,
$mvn archetype:create-from-project
Install it in your local repo,
$cd target/generated-sources/archetype
$mvn install
$mvn archetype:crawl
Generate a new project from your new template,
mvn archetype:generate -DarchetypeCatalog=local
I'm glad freedomben likes nodejs, but he shouldn't talk dirty about Java while pretending to know something about Java. His one comment to this story dominated what could have been an otherwise interesting discussion about a new feature of Java 9. Sad.
My read is that Java is doing this now because they ignored microservices and containerization at their own peril. 8 and 9 between them include a bunch of things to squeeze runtime and disk size to be comparable to other systems.
Really? Most of the "microservice era" or "cloud native" or $timely_buzzword platforms are really big. Golang binaries for large services are bloody huge.
TL;dr don't assume other systems have small footprints until you deploy a Node.JS app and do "du -h node_modules/"
I am yet to do any enterprise Web project that isn't a classic application server.
Not everyone and his dog are jumping into it.
People make monoliths for performance reasons and then can't maintain the monolith, so they go back to forced independence. Every company does this on a different cycle but the epicycles are the 'fads' you see, and it's been going on since at least the 70's.
Similarly and perhaps more forgivably, people organize a system (think, client-server, peer to peer) for reasons of the current laws of physics as they apply to clocks per second, storage space, and latency. Those inequalities shift with each new generation of hardware. So what was the right thing to do 15 years ago might be the right thing to do again next year. It's not that we've been wrong for 15 years, it's that we are wrong now. And will be wrong again in another half a dozen years. But we have to use the tools available to accomplish the goal now, and that means we have to adapt, repeatedly.
And that is a simple template that just takes advantage of curated OSS ecosystem dependencies by the Spring team.
There’s really no magic, I find Maven or Gradle to be as opaque as say Gem bundles, Bower, or NPM and Grunt (Or whatever the new flavour du jour in Node land is). That is to say, the core concepts are all mostly similar but just slightly different enough to cause hair loss. And let’s not talk about Golang dependency managers...
Also, from the Dropwizard tutorial:
1) configure Maven 2) create a config class 3) write a mapping.yaml for “Jackson” 4) create an application class 5) create a representation class 6) create a resource class 7) register the resource 8) configure maven again (!) for building a .jar 9) finally run it.
With node you literally write 5 lines of code importing the http module and run `node server.js`. I can’t possibly imagine what steps could be more complicated than Dropwizard. I consider webpack overly complex myself, but it’s a completely self-inflicted option and not a standard for nodejs development at all. I’d be curious to know what docs you were trying to follow if you can recall.
1) Install node.
2) Write script. e.g.
console.log('hello');
> node script.js
hello
1-10) Download Java *
11) Ensure JAVA_HOME is set right
12) Write java
public class Hello {
public static int main(String[] args) {
System.out.println("hello");
}
}
> javac Hello.java
> java Hello
hello
EDIT: Code formatting thanks to https://news.ycombinator.com/formatdoc
1. Dependency management is probably a lot easier with Maven than with C++ or Python. Hope that needs no reference...
2. No idea why, but Java is backwards compatible. So not updating to the latest version is only laziness.
3. Having much smaller containers or virtual machine disk sizes should lower your AWS/Google/Cloud... bill a (little, but whatever) bit and therefore make your CFO happy...
Apart from that, there is another reason why companies don't upgrade and that's because they are locked to a certain java version by their application server. Upgrading to a new java version would require upgrading the application server as well, which would cost a lot in terms of licensing and which would introduce some risks and overhead of its own.
[1] https://stackoverflow.com/questions/23801950/spring-4-and-ja...
ADDENDUM: Yes, C++ is technically even more backwards compatible - if your code base and all dependencies was/were 100% standards compliant (har har). But again, for me, the main point is that most newer languages first have to prove they can keep up to such levels of backwards compatibility (e.g., see the Python 3 fiasco...).
Will we be able to create runtime-less executables that are fully dependency free? That would be awesome to write a bootloader in Kotlin or any other Java Synthax Substitute.
Obviously it's Spring centric.
I still remember a Oracle JRuby / Truffle Ruby developer said Orcale has no plan to open source SubstrateVM.
This is Great News!
The plan is to support all major OpenJDK platforms.
There is also Project Metropolis, which was recently started, with the long term goal of rewriting all existing C++ code into Java, so AOT needs to exist in all platforms.
The problem is not artifact size but the JVM slow start-up time. This is made worse by almost all Java frameworks that by definition do all their own stuff before passing control to your real app code.
Such executables are not something you would put in a loop in a shell one liner, and that is the space for command line utilities Go has occupied successfully.
https://purelyfunctional.tv/article/the-legend-of-long-jvm-s...
But frameworks (and runtimes for things like Clojure) absolutely do. It would be possible to write a fast-starting framework for Java, but i suspect nobody has because there's no demand for it, because nobody who needs fast startup uses Java, because it doesn't start up fast!
That said, i think the real reasons Go has done so well in devops are (1) it's easy to pick up for devopsists coming from Perl/Ruby/etc backgrounds, (2) static binaries are easier to deploy than a jar plus a JVM (not massively easier - but easier enough), (3) a focus on systemsy stuff in the standard library and community, and (4) sheer snowballing momentum, in that Go has become the default choice for stuff like that.
Things like this are why all the "Java isn't slow" articles miss the point completely --- 100ms to do nothing useful, on a presumably quite fast machine, is ridiculous.
For comparison, 100ms is roughly the time it takes to grep an 18MB file:
http://dtrace.org/blogs/brendan/2011/12/08/2000x-performance... (look near bottom of post).
"Ridiculous" is like "unprofessional", it's what you say when you don't have a real argument. Yes, ha ha, 100ms, very droll. What does it matter in practice?
The major pillars of DevOps today are long-running processes, very complex pieces of infrastructure. Which might play into the strengths of the JVM, if the packaging and deployment story simplifies and becomes more competitive with Go.
Sure, "shell one liner" utilities might (?) be best served by native compilation. But DevOps is not merely another word for sysadmin.
It actually was totally fine and surprisingly snappy. I’d suggest giving it a shot.
I don’t know how to revert the jvm can’t be used for command line tools because of slow startup times zeitgeist but I don’t think it’s accurate anymore.
I hope some work will be done there. One could scan the libs in parallel, caching results, building/shipping a search tree on packaging time and so on.
In pure servlet times you could directly define your starting points, define, which libs to exclude or include for the search and so on.
At least no one has to edit xml files anymore to wire components and impls together.
Maybe I'm a little overly jaded, but the Java apps I've worked on (which is admittedly only a handful) have all had these giant sprawling messes of dependencies; it feels like there's a lot of code to accomplish very little. The Golang-based services I've worked on feel like the opposite; very little code to accomplish a lot of things, without much ceremony.
Edit: And that's not to say that I'm opposed to dependency injection; rather, I'm quite happy to do constructor-based dependency injection by hand. When that starts to feel painful, I look at the system I'm working on and wonder if it's getting too complicated/has too much coupling/etc.
For us as a little company without access to many developers with affordable pay expectations, it is a big relieve, that a lot of specialist knowledge is bundled together in spring. Such as how to do central configuration, logging, monitoring, failover, loadbalancing, streaming, backpressure support, db connection pooling, message queuing, and so on. And you need special knowledge only when something fails. And when it fails, you can assume, its done the same on each service. It helps a lot. You have then a wider pool of people you can ask to help you to do the actual business coding work.
It reminds me a little on how maven defaults said: code belongs into src/main/java and test below src/test/java and then have a little pom.xml,that describes your project. It has brought sooo much relieve to get rid of so many flavors, how people have structured the code. Not to forget the style of make scripts. You can now open many projects in any IDE without any hassle, you do not have to search forever, where the starting point is and so on. I always forget that. Less work for my little brain, when navigating between projects.
So, even with the dependency hell (i totally agree), i am ok with it, as i seem to forget, what i dont have to do on each individual service anymods. And i have access to a rich ecosystem, that is maintained and supported and people react on issues.
Edit: and very very fast startup.
The current dominant culture of the Java community is very weird. Although Java has a huge standard library, if you don't use a lot of third-party libraries, you will be viewed as a non-professional Java programmer.
Most popular Java tech stack look open sourced, but are backed by lots of money provided by many companies with bad reputations on pure open source culture.
BTW, slow start-up is not the only problem of many Java apps. GUI Java apps often lag for one or more seconds from time to time, which hurts the experiences of Java GUI apps much.
The third problem of Java apps is they often consume much more memory than apps written in other languages, in particular for the long running Java apps. The longer they run, the more memory they will eat.
[edit] the library use way of Java, by putting library jars into your projects, is not good as the way of other languages, by putting library sources into your projects. By putting library sources in your projects, you can view how the libraries are implemented easily. Yes, you can also put Java library sources into your projects, but the main stream culture of Java doesn't recommend to do this. Just look at maven.
[edit 2] just my personal opinion. I don't like the way to use a separated VM to run many apps. This makes it is hard for me to use a different permission setting for each app. Sometimes I want to block a Java app to access network, but I must block the whole Java VM to achieve this.
Second, i am surprised to read, that people would still put dependencies/libs into their projects, even in their source code form? I thought, that was one of the reasons, why dependency management was introduced, to get rid of that. One ships a descriptor, what libs were required/used, when the lib has been packaged. And in one's IDE one can view all the dependencies as one wishes, in binary or source code form. I am very confused about your statement. (though in the final App, one ships all the libs in the packaged form of the app, unless provided by the runtime environment. But only then. Though its optional, one could also download the deps when starting the App)
And third: The longer an App runs, the more memory it eats? Its like with any other App: if it needs that much memory and it does not release it afterwards, well, then it is a feature or a bug. If it releases the memory, then it is only released within the JVMs memory space. The common default for thag is 4gb per JVM.
Its like the page cache in linux. Once a file has been read into memory, it will usually sit there for a while, unless someone else needs that memory. Thats why on a system with 128gb and a lot of file access, you see free memory going down and down. But its most of the time the page cache eating up that memory and is mostly not a problem. But can be confusing, when one is not familiar with it.
Ah, so many things one can tell.
What I mean is dependencies presented as sources, vs presented as Jars. It has nothing related to dependency management.
Did you mean that?
Now, when i use this library, my IDE knows how to lookup the source jar, so i can always browse the source code. When i see a bug or need a feature, i go to the repository, patch the code and release a new version of that library. And then i bump up the version number of that library in my own project as well.
If i do not own the code, then i ask for a change with a patch (in best case). And use a workaround in the meantime in my project. Earlier or later the updated version of thr library has been shipped (maven repo) and i bump version number as well and remove my patch.
And worst case: owner does not want to change, well, then i fork the code and add it myself, tag the library, release it to my artifact repository and so on. But i barely do that.
In any way, it all relates to my artifact repository (which proxies through official repos such as maven repo) and dependency management from maven, gradle, ant, and so on.
How do you do it?
But the fork will only be made if I have confirmed the fork is worthy to make. Before the potential fork, I temporarily modify the code in the local clone of "others.com/projects/foo" directly. If I find my modifications are useless, I will revert the modifications so no forks will be made.
Other processes are like what you described.
Yes, there is still not a perfect dependency management tool in Go world, but there is one official dependency management tool in developing, which will be released alongside with Go 1.10.
The standard library is where modules go to die. The fact that this phrase originates in the Python world tells you that this isn't a Java peculiarity.
> Most popular Java tech stack look open sourced, but are backed by lots of money provided by many companies with bad reputations on pure open source culture.
The sad reality is that those companies are the only organisations that commit serious resources to open-source development, outside a couple of specific niches (web development, unix-like OSes, scientific research up to a point). Those stacks aren't open-sourced in other languages, they just don't exist. And the whole point of open-source is that it benefits everyone regardless of how pure or otherwise the original motives were.
> the library use way of Java, by putting library jars into your projects, is not good as the way of other languages, by putting library sources into your projects. By putting library sources in your projects, you can view how the libraries are implemented easily. Yes, you can also put Java library sources into your projects, but the main stream culture of Java doesn't recommend to do this. Just look at maven.
WTF are you talking about? You declare your maven dependencies and they can be resolved as source or binary. In eclipse I can click through to any library function and see its source immediately. (It's one of the best dependency-management systems going, in any language; there's a single central repository that everyone uses in practice, but it's easy to run your own if you want. It's years ahead of everyone else in terms of package signing. There are multiple independent codebases using the same repositories, not just in theory but in practice).
> [edit 2] just my personal opinion. I don't like the way to use a separated VM to run many apps. This makes it is hard for me to use a different permission setting for each app. Sometimes I want to block a Java app to access network, but I must block the whole Java VM to achieve this.
Again completely normal - Python, Ruby, C#, OCaml... will have exactly the same issue. Get a better firewall.
Surely, you can view Java dependency sources. It is just that most Java programmers don't care about the sources, they are just care about the jars (the default culture).
In my honest opinion, cross-platform by bytecode has few advantages over cross-platform by source nowadays, on the other hand, cross-platform by bytecode has many inconveniences.
And, (in my taste), single central repository is bad. That's why I don't like Node also. The npm central repository is the source of many Trojans. Many Java and Node developers are not aware of and have no ideas on what they have downloaded through chain dependencies.
> Again completely normal - Python, Ruby, C#, OCaml... will have exactly the same issue. Get a better firewall.
Normal != good.
Not my experience. Actually getting to the source of your dependencies, in practice, is easier in Java - just one click in your IDE - than in any other language. Doesn't that suggest that Java programmers care more about sources than other language users, not less?
> In my honest opinion, cross-platform by bytecode has few advantages over cross-platform by source nowadays, on the other hand, cross-platform by bytecode has many inconveniences.
Well you're entitled to your opinion but you should probably back it up with something more specific if you want to convince anyone else.
> And, (in my taste), single central repository is bad. That's why I don't like Node also. The npm central repository is the source of many Trojans. Many Java and Node developers are not aware of and have no ideas on what they have downloaded through chain dependencies.
You don't have to use a central repository if you don't want to, and some people don't, but it's very convenient to have the option if you want it. Maven central requires PGP signatures on anything published there, so it's very easy to enforce that all the dependencies you depend on come from trusted people if you really want to - as far as I know there's no non-JVM language that does that, certainly not with anything like as large a base of signed libraries available.
> Normal != good.
Agreed, but you started out by claiming that Java's culture is very weird. It isn't.
The only benefit of bytecode is to save compiling time. Nowadays, CPU becomes much faster than the time Java was invented, 20 years ago. So the benefit is weak now. Another promoted benefit of bytecode is compile-once-run-anywhere. However, I think this is a hoax. It is not special, for script languages code runs anywhere without compiling.
I was once working on a large project on the JVM: our Fat-JAR was really working on Linux, Mac and Windows. The exact same JAR. And it didn't matter on which system we have built that JAR. This may not be possible for every use case - but I don't think it is a hoax.
> It is not special, for script languages code runs anywhere without compiling.
IMHO scripting languages like Ruby, Python, etc. are quite hard to deploy, especially if they have dependencies. In comparison to languages like Java, Go or Rust.
In terms of throughput I agree with you. But start-up latency is already Java's weakest point, and having to compile on startup would make it worse.
> Another promoted benefit of bytecode is compile-once-run-anywhere. However, I think this is a hoax. It is not special, for script languages code runs anywhere without compiling.
True as far as it goes, but certain types of code simply aren't practical in scripting languages. You'll never see people implementing e.g. an image format decoder in a scripting language; they'll use bindings to a native-compiled (C) library instead, with all the problems distributing that entails. This is why things like docker are so popular with Python users: in theory Python is write-once-run-anywhere, but in practice most Python codebases end up depending on one or more native modules and need to have the correct version compiled. Whereas Java bytecode is adequate for things like image codecs, and in the real world you really do see large Java codebases that don't use any bindings to (non-JVM) native libraries.
Depending on what definition for "slow" you are using it absolutely starts slow.
As mentioned in a cousin comment, the JVM takes ~100ms to get to user code. This makes it impractical for small applications where the user's code takes less than a second to complete. It certainly gets slower with more libraries, but for some applications ~100ms added to startup makes using Java not viable.
Just some hard data to support Java being slow. I ran Hello Worlds in Java, C, Node, Python, and Ruby. Java is the slowest out of all the languages I tested.
time java HelloWorld
Hello, World
real 0m0.170s
user 0m0.148s
sys 0m0.012s
time helloWorld
Hello, World!
real 0m0.028s
user 0m0.004s
sys 0m0.008s
time nodejs hello.js
hello world
real 0m0.136s
user 0m0.104s
sys 0m0.012s
time python hello.py
Hello World
real 0m0.050s
user 0m0.016s
sys 0m0.020s
time ruby hello.rb
Hello World
real 0m0.111s
user 0m0.064s
sys 0m0.028sMaybe in SV, because I am yet to see it being used in any of our projects or hiring processes.
Docker and Kubernetes are the most prominent, but there is a whole ecosystem behind them.
Again, yet to use any of them, beyond watching some talks at a few conferences.
Still classical server deployment processes.
Of course, companies using software written in Go, are open to use Go somehow.
Yes, you can compile a QT or a GTK app for every platform, and use e.g. Python or Go to write the cross-platform part + package it into a single file. It's still as many files as you have platforms.
I see e.g. my wife, not an IT person, use a number of them. An eBay betting app, several utilities for bead art, etc, most written in Java years ago. Do they look polished and native? Hell no. Do they get the job done? They do. Could they look and feel better with a JavaFX UI? Quite likely. Would having a single binary to download, without installing a JRE first, improve their adoption rates? Quite likely again.
I have worked on a Java desktop app with Swing UI for over a decade. Mid 2000s we would get asked to "modernise" our applications UI, which really meant: "make it look like Windows XP".
After Apple released iTunes, which totally ignored the native look and feel on Windows, people stopped caring so much as long as it looked ok and was intuitive. The boom in web apps with every SPA defining it own internal LAF through CSS and Javascript and the fad for flat UI components seems to have further watered down the arguments for conforming to a native LAF.
I haven't played much yet with JavaFX so can't answer your actual question but I believe it is more flexible than Swing and Swing is already very customisable (though you sometimes need to create your own widgets by extending/implmenting off the default widgets).
In order to practice what I preach, I'm looking for something that looks native for most cases that I can use for side projects and become proficient in. For my current project I chose SWT over Swing because SWT supports a native look on Linux (via GTK) while Swing doesn't (as far as I can tell). However, SWT is kind of clunky and I wonder if there are better options or if my previous research is out of date.
try {
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());
} catch( ClassNotFoundException | InstantiationException
| IllegalAccessException | UnsupportedLookAndFeelException e ) {
}
On the otherhand, with Swing all you need to do is add the following line:
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());While I was at Google, instead of having multiple .so files, the main launcher was a C++ binary, and all external C++ dependencies were linked into it (that's it if you have the source code), then you didn't need that.
I guess now it's up to the BUILD system to achieve that, where if you have more flexible BUILD language you can instruct everything to go in the main launcher file, then have all the .jars into one, slapped at the end of the file.
At the end, ship or update only one binary to your service, desktop machine, etc.
Native libraries used by JNI can be shipped in the jar files and are read from the classpath, in addition to reading them from the ordinary linker path.
(edit: turns out I'm wrong about this on Windows)
I think the same is with the Python packagers, or C# shadow copy.
This way there is no need for extra install, uninstall. Possibly too much over-engineering for full blown product, but if it's something that needs to be run, and updated on tons of machine - not having to deal with extra artifacts might be a win.
Looking at an example of a library that does native binding: https://mvnrepository.com/artifact/com.github.fommil.netlib/...
It appears the artifact is a DLL, and not a JAR. Thus your original question stands.
I've tried loading dlls from memory (there are several examples on the net) only to understand that there is more to be done (like getting the PDB information correctly loaded, and then some more).
Unless you consider that basic support services are what the Java runtime should be providing on its own. Fundamentally, a command line Hello world should only take a few dozen bytes of bytecode.
Now all engineers are "angry" :) :) :) after seeing the same process spawned multiple times, and taking lots of gigabytes!!! (hehee)... I simply don't care and love the Slack app! (Teams is also nice looking, and Visual Studio Code rocks)
Of course there are far smaller libc implementations, but they may be missing functionality.
[1] Unless, of course you require an installation of Java... which would kind of go against the premise of this whole exercise.
you only pay for what you use if you dynamically link it.
Note, this on 32 bit arm, size might differ on different arches, all I have access to off the cuff:
# printf '#include <stdio.h>\n int main(int argc, char**argv) { puts("hi"); }' > /tmp/hi.c
# make /tmp/hi
cc -c -o /tmp/hi.o /tmp/hi.c
cc /tmp/hi.o -o /tmp/hi
# du -hs /tmp/hi
12.0K /tmp/hi
# ldd /tmp/hi
/lib/ld-musl-armhf.so.1 (0x7f5af000)
libc.musl-armhf.so.1 => /lib/ld-musl-armhf.so.1 (0x7f5af000)
# strip /tmp/hi
# du -hs /tmp/hi
8.0K /tmp/hi
# rm /tmp/hi
# make LDFLAGS=-static /tmp/hi
cc -static /tmp/hi.o -o /tmp/hi
# du -hs /tmp/hi
92.0K /tmp/hi
# ldd /tmp/hi
ldd (0x7f5e3000)
# file /tmp/hi
/tmp/hi: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped
# objdump -p /tmp/hi
/tmp/hi: file format elf32-littlearm
Program Header:
LOAD off 0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**16
filesz 0x00001f68 memsz 0x00001f68 flags r-x
LOAD off 0x00002ea4 vaddr 0x00012ea4 paddr 0x00012ea4 align 2**16
filesz 0x00000204 memsz 0x00000798 flags rw-
DYNAMIC off 0x00002eb4 vaddr 0x00012eb4 paddr 0x00012eb4 align 2**2
filesz 0x000000e0 memsz 0x000000e0 flags rw-
STACK off 0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**4
filesz 0x00000000 memsz 0x00000000 flags rw-
RELRO off 0x00002ea4 vaddr 0x00012ea4 paddr 0x00012ea4 align 2**0
filesz 0x0000015c memsz 0x0000015c flags r--
Dynamic Section:
SYMBOLIC 0x00000000
INIT 0x00000468
FINI 0x00001f38
INIT_ARRAY 0x00012ea4
INIT_ARRAYSZ 0x00000004
FINI_ARRAY 0x00012ea8
FINI_ARRAYSZ 0x00000004
GNU_HASH 0x000000d4
STRTAB 0x00000250
SYMTAB 0x00000120
STRSZ 0x000000ee
SYMENT 0x00000010
DEBUG 0x00000000
PLTGOT 0x00012f94
PLTRELSZ 0x00000018
PLTREL 0x00000011
JMPREL 0x00000450
REL 0x00000340
RELSZ 0x00000110
RELENT 0x00000008
BIND_NOW 0x00000000
FLAGS_1 0x08000001
RELCOUNT 0x0000001c
private flags = 5000400: [Version5 EABI] [hard-float ABI]
# strip /tmp/hi
# du -hs /tmp/hi
16.0K /tmp/hi
isn't this only true for the first time a function is used?
once all of the linking is done, the GOT will just contain the address of the function and it's just as if it were linked statically, right?
Yes, but it will happen each time the program loads, which means your LD_* variables will get parsed/iterated over. Statically linking stuff in can help if you have a program that needs to be exec()'d in a loop. While it'll probably only shave off 0.001 seconds per exec() call it can add up, more so if you have LD_LIBRARY_PATH set to look on some slow storage like nfs.
But its a bit of a hyper optimization at that point. I've used this to really annoy a c++ developer saying c++ was faster than c. Evil trick on my part but my euler solutions were always faster. >.<
That said I really need to get back to debugging the segfault in building ghc 8.2.2 on armhf in the ghc rts. But its gotten me going through everything but ghc and whatnot. Annoying when things worked before and don't now. Even weirder that only the final stage build is at issue, the other two are fine.
Even if you don't buy that reasoning, it's just not very common to use non-glibc on most common distros.
... and honestly, I should probably just have been more explicit about the absurdity of comparing "hello world". That would sort of agree with the OP's post, but not for the same reason. I think reasons are important.
[1] ... but then you end up with discussions about the importance of locales, nsswitch, etc. etc. Even though I didn't bring it up, it necessitated this footnote. You see what I mean?
https://www.gnostice.com/nl_article.asp?id=225
The difference in abstraction level is amazing.
Bazel is still a bit away from having Java 9 support [1] and it still requires some system dependencies (for the other languages and cross system support).
I wish more tools would go the "static binary" path that make it easy to build and use wrappers without having to figure out system configuration every single time.
For Windows, it's mainly perception - if engineer sees that a tool is like 200mb in the depot, he/she be like - that's too much :) "Remember why we pulled boost, and no longer use it - as it's huuuge!" :)
The AOT compiling though would require Clojure-AOT followed by JVM-AOT I guess.
> [...]
> and superior to web-hybrid options like Electron
How is Java or Electron (JavaScript+WebKit) considered native nowadays?
Please use native APIs, so my laptop can actually hold 10h load on battery.
edit: can we remove native from the title?
Does the app use the most efficient methods provided by operating system? Is the graphical user interface consistent with the other applications on the same system? Is number of syscalls predictable?
That wouldn't make sense. Not only would it go against the Java philosophy of write-once-run-anywhere (Java is very hostile to cosying up to platform-specific frameworks), it would also be a mistake in practical terms - they're right to just keep maintaining JavaFX, which is a perfectly competent GUI framework.
I think it's reasonable to say that if the the user code is transmitted as a managed intermediate representation then it's not native. Arguing beyond that probably isn't productive.
But that's this particular blog post. Java can be native, if you compiled all the user code to machine code ahead of time.
Seems like lots of projects are likely to become non-native over time, by this definition, even if they start out "native" with a basic C/C++ codebase, since lots of projects accrete some sort of scripting or extension interface that might use a runtime of some sort. Is Emacs not native? Does it really matter if the runtime (which is machine code) was made by the compiler developer or by the application developer in some cases?
I feel like there's an arbitrary line, and it's not entirely useful distinction to make.
A native app can be big and slow and ugly and fail to follow the OS UI guideliness. An app with a runtime can be small(ish) and fast and beautiful and strictly adhere to the UI guidelines for the OS in question. I want the small, fast, beautiful, app with standard UI behavior, no matter whether it's delivered with a runtime embedded or built from C/C++ (and a few others) and without a JIT.
I mean, we're just arguing semantics here, which is kinda silly. I think it's pretty neat; I don't really like working in Java, but I'd rather work in Java than C/C++, so if I needed to deliver a cross-platform GUI app, I'd certainly consider it.
Of course the JVM has advantages that it can dynamically optimize code according to actual execution profiles.
If I am not mistaken with the right instrumentation LLVM can also makw your program generate a special optimization profile which you can then apply to your final build compilation.
Well I think that's a different meaning of the word 'native'. They're native in terms of being written using the recommended techniques for that platform, but not native in terms of being natively compiled.
But as you say it's just a discussion of semantics.
Correct.
> Are programs written in C# not native, even on Windows?
No they are not, because .Net uses an intermediate representation, being Windows and Microsoft confers no magic "native"-ness.
> Are programs in Java for Android not native, even though it is the language much of the OS is written in and what the standard UI toolkit is built for?
Programs in Java for Android are not native.
Much of the kernel (Linux) is written in C. According to "Francesco Iovine, AndroidStudio lover" over at Quora[0] the UI toolkit is written in Java and the virtual machines (Dalvik and Art) are written in C++.
> Is Emacs not native?
Emacs is a strange and evil beast whose name we shall not invoke. No, but seriously – I imagine Emacs is a hybrid native / Emacs lisp program. Many programs with embedded scripting languages are hybrid beasts but they ought to be thought of as native apps if they are compiled for a certain bare metal hardware platform. I presume the core of Emacs is C. Ergo, native.
> Does it really matter if the runtime (which is machine code) was made by the compiler developer or by the application developer in some cases?
Yes, I believe it does matter.
> I feel like there's an arbitrary line, and it's not entirely useful distinction to make.
No, I don't believe it's arbitrary.
> A native app can be big and slow and ugly and fail to follow the OS UI guideliness. An app with a runtime can be small(ish) and fast and beautiful and strictly adhere to the UI guidelines for the OS in question. I want the small, fast, beautiful, app with standard UI behavior, no matter whether it's delivered with a runtime embedded or built from C/C++ (and a few others) and without a JIT.
True. But that is neither here nor there, that's a whole other issue.
> I mean, we're just arguing semantics here, which is kinda silly.
No, we are not. A native app is one which is either hand-crafted assembly for a specific hardware platform or compiled ahead of time targeting a specific platform. Native code is non-portable, the source code might be portable, but generally is not unless care is taken. Assembly code is never portable. Non-native (whether by VM: Java, C#, … or interpreted: Ruby, Python, …) tends to be more portable.
[0] https://www.quora.com/What-programming-language(s)-is-Androi...
If it's native, why isn't JIT to machine code not native? Because of the convenience of all of the above happening on a fine-grained level (individual functions), and all on the target machine?
Also, what is "compiled ahead of time"? Ahead of what time? Why is that specific time important? Obviously, code is compiled ahead of its compiled image being run; the compiled version cannot be run any sooner. Any JIT-ted function is compiled ahead of being called.
On most operating systems, we can load programs after the OS boots. Those programs can be developed after the OS booted.
I think that the only true definition of "native" is that the program image was compiled before the hardware was built and turned on for the first time, and was present in its ROM.
Anything else is not "ahead of time" and not native, damn it!
Installing a program on your PC is just a form of JIT. The system was fired up without that program being there, and the program was snuck in just moments before the user's intent to use that program. Not ahead of the True Scotsmans proper time: the time when components were stuffed into the circuit board and it was powered up.
If the software image is compiled in its entirety before being installed on a target, that is "monolithic". If parts of it can be dynamically loaded, it is "modular". Dynamic loading of functions with JIT is just heaping on more modularity: hyper-modular.
A situation involving JIT is non-native only in the sense or to the extent that there is non-JIT-ted code in the image. If there is an interpreter so that some things execute without being JITted, those things are not native.
So AS/400 applications are not native, given that they use an intermediate representation with a kernel level JIT?
Same applies to other mainframes that use a micro-coded CPU to execute the portable binaries using intermediate representation on their executables.
This includes C and C++ on those platforms, by the way.
.NET has had AOT support since early days, even though NGEN is not that good.
> Programs in Java for Android are not native.
Java is AOT compiled to native code on installation on Android 5 and 6.
https://source.android.com/devices/tech/dalvik/#AOT_compilat...
Starting with Android 7, it is interpreted in a tight Assembly interpreter, JITted with PGO, finally AOT compiled into a code cache.
https://source.android.com/devices/tech/dalvik/jit-compiler
There are AOT compilers available for Java and C#, just as there are interpreters for C and C++.
> So AS/400 applications are not native, given that they use an intermediate representation with a kernel level JIT?
Thanks for making me read up on AS/400 (or System i as IBM now calls[0] it.) I found this article: [TIMI – Protecting Investments and Integrity in IBM i](http://ibmsystemsmag.com/blogs/you-and-i/archive/timi-protec...) – it says,
“Each program that is compiled on IBM i gets turned into a set of TIMI instructions. Essentially, it gets compiled down to a set of intermediate code. For decades, that’s been a pretty common step for a compiler to take – get to an intermediate representation of the program before creating the machine-level instructions. One big difference, though, is that the compiler on IBM i does not go any further. It leaves the intermediate code connected to the program object, and then the lower levels of IBM i act on the intermediate code to translate it into the instructions used by the Power hypervisor and Power processor beneath the OS. IBM i does this only when necessary, not every time the program executes. Typically, the translation takes place once when the program is moved from one release of the operating system to a later one, and never again while it runs on that later release.”
> Java is AOT compiled to native code on installation on Android 5 and 6.
Then by your reasoning and terminology I think you'd also call IBM's tech AOT compiled, not JIT'd given that it happens once per OS release or app move, not each time the app is executed.
And I would say that the applications that get delivered this way on AS/400 and Android 5 and 6 are delivered as non-native (intermediate code as the article says). And then get translated AOT to native which is stored somewhere else and the non-native code is kept around.
Neat, I didn't know that about Android 7 and Dalvik. Is it the same for Art?
> There are AOT compilers available for Java and C#, just as there are interpreters for C and C++.
This I did know. :)
Always welcome your insights, thanks!
> Correct.
Oh, come on. You can't seriously believe this nonsense. A native C++ program embedding (for example, to parametrically specialize some constructs at run time) an LLVM JIT which is in C++ again is going to be as native as they come.
> I presume the core of Emacs is C. Ergo, native.
And that C++ program I mentioned isn't, according to you, even though most of Emacs is delivered in interpreted byte code form. And if Emacs stated JITting the bytecode, it would cease being native. You really seem to have some weird and/or inconsistent definitions.
C# always had AOT compilation to native code via NGEN.
Just that few people bother to use it, because it only allows dynamic linking, the optimizer is tailored for fast startup time and requires an extra effort with application signing.
Windows 8 apps were always AOT compiled on the store via a "cloud compiler", based on Bartok from Singularity project. Using a format known as MDIL.
Windows 10 apps are always AOT compiled via .NET Native, which shares the backend with Visual C++.
Mono always supported AOT compilation and it is the way Xamarin apps are deployed on iDevices.
Which the method described in this particular blog post is going to support very soon – today you still need to invoke subtratevm or jaotc separately.
Superior exactly in what way? Electron allows access to an enormous audience of developers and the web ecosystem. You really can't compete with that and would be silly to try at this point.
I haven't done much more than dabble, but it was an enjoyable dabble. Unlike vanilla Java.
It even compiles to JS with a relatively small-ish runtime. It's big enough to cause problems embedding it, but it's small enough to be workable.