HiJon89

Gigafactory is a proper noun referring to the specific battery factory built by Tesla; I just didn't bother to capitalize it

How long will it take them to catch up with Tesla (while Tesla is simultaneously advancing their own battery tech)? And even if they can hit that milestone, they'd presumably have some amount of profit margin built into the prices. So to actually compete with Tesla on battery price (and why would they even want to) they'd need to advance beyond Tesla in terms of manufacturing efficiency and/or volume. I don't know of any company poised to fill this gap in the foreseeable future.

There's good reason to believe that electric cars are the future and will soon be a massive market. I think there's also good reason to believe that Tesla has a big head start on the competition and has the opportunity to dominate this market if they execute well.

For example, just look at the battery packs which are essential to an electric car. Tesla was unable to find a supplier that could meet their requirements, so they designed and produced their own batteries and now have a $5 billion gigafactory to increase production volume and decrease costs (with more gigafactories on the way). If you're another car company trying to enter the EV market, what are your options? Buy inferior battery packs at a higher cost from a different supplier? Try to replicate Tesla's battery technology and then convince your board to build a gigafactory so you can produce them at the required volume and cost?

This similar to our setup with the slimfast-plugin. We use the maven-jar-plugin to add the Main-Class and Class-Path entries to the manifest. At build time, we use the upload goal of the slimfast-plugin to upload the dependencies. It automatically reads the configuration of the maven-jar-plugin to make sure the paths it generates match the classpath in the manifest. Then it spits out a JSON file that has info on each dependency, including its location in S3, file size, checksum, and relative path where it needs to be copied to match the Class-Path in the manifest (the JSON entries look like this https://gist.github.com/jhaber/3029dc55a568f0954b1c4b459657e...).

On deploy, the simplest way to get up and running is to use the download goal of the slimfast-plugin. It reads this JSON file, downloads each dependency (using an optional cache folder), verifies the file size and checksum, and copies it to the correct relative path. The application will then start up happily with java -jar.

At HubSpot, we instead integrated this download step more transparently into our deploy process. At build time we read this JSON file and store the dependency information in our build database. Our deploy infrastructure already accepts S3 URLs and handles downloading, caching, verifying checksums, and copying to arbitrary directories so we just piggybacked on this existing functionality to have it download the application plus all of its dependencies for us.

We definitely have some dependency cruft that could be trimmed (lots of relocated copies of Guava due to incompatibilities, for example).

We do frequent production deploys (of individual services, there's no such thing as deploying our entire application). To give an idea, it's a little before 1pm here and across our team there have been 180 production deploys already today.

We're actually transitioning from Nexus to S3 as our internal Maven repository. It alleviates disk space concerns, better uptime, better throughput, etc.

We actually still produce executable JARs with this approach. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR and no runtime or ClassLoader funny business.

We still use java -jar with this setup. We configure the maven-jar-plugin to construct the classpath at build time and add it as a Class-Path entry to the manifest. This is a special manifest entry and on startup the JVM automatically adds these files to the classpath, so our thin JARs are runnable with java -jar assuming the dependencies are dropped in the right place. The slimfast-plugin reads the configuration of the maven-jar-plugin to make sure it's generating the right paths for the dependencies. We end up with an executable JAR with no runtime or ClassLoader funny business.

Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.

It's roughly 300 REST APIs and the rest are cron jobs, kafka workers, Hadoop or Spark jobs, etc. etc. Each part of the product gets its own API which is where most of them come from as our product is quite broad

Agreed, we might not have done it if we had to resort to having the application download its own dependencies at startup. In our case, our Mesos scheduler, Singularity, handles S3 artifacts at deploy time for us. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.

What was the process for adding or removing a dependency? Changing a version? If two apps needed different versions?

20-30 seconds ends up being 50% for most of our Java builds, so it was a huge speedup. It also saves 20-30 seconds at deploy time when we need to download this file again. And with lots of concurrent deploys we were occasionally seeing these large downloads saturate the NIC on our application servers.

Also, we use Singularity as our Mesos scheduler and it handles S3 artifacts out of the box. Previously we gave it a single S3 URL for the fat JAR, but now we just give it a list of artifacts (the app plus its dependencies) and it handles everything for us so it's not much more complexity and ended up being really easy to integrate into our deploy process.

When something goes wrong we rarely downloaded the JAR, but if you really wanted to you could download the thin JAR and use the SlimFast metadata bundled with it to download the exact dependencies it would get when deployed.