krallin

(engineer working on Buck2 here)

Buck2 is actually used internally with Git repositories, so using Sapling / hg is definitely not a requirement

I love SQLite (and use it in a large number of places in our application stack!), but I feel that it suffers from a case of bad defaults. To name a few:

- Foreign key checking (and cascading deletion for that matter) are turned off by default. You need to enable them using `PRAGMA foreign_keys = ON;`.

- There are practically no downsides (and a number of upsides) to using the WAL journalling mode (at least for "use a local database and store it on the disk" use cases). The main one being that reads will conflict with writes if you don't enable it! (which is a problem in a multi-threaded environment) Unfortunately, that's another feature you must remember to enable: `PRAGMA journal_mode = WAL;` (for obvious reasons, this one "stays enabled" after you turn it on).

- Full auto-vacuum cannot be enabled after you start writing to the database unless you enabled incremental auto-vacuum. If you're unsure, it's a good idea to enable incremental auto-vacuum to keep that option open. But, here again, that's not the default: you need `PRAGMA auto_vacuum = INCREMENTAL`.

This tends to be explicitly problematic if you have to perform one or more ad-hoc queries using the SQLite command line on an app's database, and forget to apply the relevant PRAGMAs! I wish there was a way to add "default" PRAGMAs on a sqlite database file to avoid this.

(note: some of these defaults are configurable when compiling sqlite from scratch, but if you're dynamically linking with an OS-provided instance of the library, you can't really do that).

AWS provides you with a number of DNS records for each NLB:

- One record per zone (which maps to the EIP for that zone) - A top-level record that includes all active zones (these are all zones you have registered targets in, IIRC)

The latter record is health checked, so if an AZ goes down, it'll stop advertising it automatically (there will be latency of course, so you'll have some clients connecting to a dead IP, but if we're talking unplanned AZ failure, that's sort of expected).

That said, this does mean you probably shouldn't advertise the IPs directly if you can avoid it, yes.

(disclaimer: we evaluated NLB during their beta, so some of this information might be slightly outdated / inaccurate)

Author here - let me know if you have any questions!

(whether that's about using Supercronic... or making regular cron work in containers)

Yes, you're entirely correct; that's actually what I meant by "provided the Kernel can represent [it]" (albeit perhaps not very clearly!).

Cheers,

(Note: the following is only applicable to Linux)

The default for overcommitting on Linux is heuristic; it doesn't always succeed: if you try and allocate several exabytes or RAM, allocation will definitely fail (in fact, trying to allocate e.g. 2GB of RAM if you only have 1 free will usually fail just the same).

There is an option for "always overcommit" (incidentally; the one Redis recommends you use), in which case allocation will always succeed provided the Kernel can represent what you're trying to allocate (what you're describing), but it's definitely not the default

Reference: https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

Unfortunately it looks like the distros have not been particularly diligent about releasing a fix via the security channel (according to the article), so this PPA is a workaround

Note: if you're using Ubuntu, there is a semi-official PPA that has a non-vulnerable version (2.7.3): https://launchpad.net/~git-core/+archive/ubuntu/ppa

This means "install the `requests` package with the `security` extras.

In this particular case, this installs 'pyOpenSSL>=0.13', 'ndg-httpsclient', 'pyasn1'. See: https://github.com/kennethreitz/requests/blob/46184236dc177f...

This principle doesn't necessarily mean functions should never return booleans, though.

Booleans are used in a variety of (popular) Python libraries when checking whether a password is correct (e.g. Django's `check_password` returns False if the password is wrong).

Quick disclaimer: I'm the author of Tini (thanks for the hat tip, by the way!).

Note that for interactive usage, Tini actually hands over the tty (if there is one) to the child, so in that case signals that come "from the TTY" (though in a Docker environment this is an over-simplication) actually bypass Tini and are sent to the child directly. This should include SIGSTP, though I'm not sure I tested this specifically.

That being said, both tools are probably indeed very similar — after all there is little flexibility in that kind of tool! Process group behavior is probably indeed where they differ the most. : )

While not as common as nounset and errexit, pipefail is a useful option as well (set -o pipefail).

Using pipefail, if any program in a pipeline fails (i.e. exit code != 0), then the exit code for the pipeline will be != 0.

E.g. pipefail can be useful to ensure `curl does-not-exist-aaaaaaa.com | wc -c` doesn't exit with exit code 0..!

Ha, interesting trick! : )

The GitHub API allows up to 5,000 requests per hour. I currently poll once an hour, so there's a bit of headway. If that becomes a problem, I think I can always space out checks for less-active repositories (there's also the option of allowing users to authenticate with GitHub and use their own quota so they don't have to share it with anyone else).

Cheers

Indeed, not yet! Private repo support would require authenticating with GitHub, which makes the project a bit more complex.

I definitely plan to add this if there's enough interest. There should be a link to a mailing list to be notified when that rolls around (http://eepurl.com/bILhR1).

Thanks for checking it out anyway! : )

Ha, this is a great idea! I've added this now :). Just reload.

I made this mainly to scratch my own itch :).

It's slightly more convenient for me to get a notification than to periodically pull repositories and check them for changes, but I understand that might not be the case for everyone!

Here's the email I got inviting me to re-post the story. That's all I know : )

  Hi there,

  https://news.ycombinator.com/item?id=10690114 looks good, but didn't
  get much attention. Would you care to repost it? You can do so
  here: https://news.ycombinator.com/repost?id=10690114.

  Please use the same account (krallin), title, and URL. When these match,
  the software will give the repost an upvote from the mods, plus we'll
  help make sure it doesn't get flagged.

  This is part of an experiment in giving good HN submissions multiple
  chances at the front page. If you have any questions, let us know. And
  if you don't want these emails, sorry! Tell us and we won't do it again.

  Thanks for posting good things to Hacker News,
  Daniel

These are great points; thanks for your feedback! :)

I'm trying to validate whether there's any interest in something this, so I definitely cut a lot of corners (e.g. the app doesn't maintain a copy of the repository — it just uses the GitHub API). Also using GitHub in the name might not have been the best idea.. though! ;)

I agree with you regarding the usefulness of watching arbitrary repositories. There's already git-notifier [0] that does pretty much this, but you have to set it up for yourself and it has to be configured using a text file.

[0]: http://www.icir.org/robin/git-notifier/

When I worked at Scalr, we used Chef's Omnibus (https://github.com/chef/omnibus) to create "full stack" RPM and Deb packages, which basically include all your dependencies all the way down to libc (meaning libc is your only dependency) in a single "fat" package (this does mean you have to keep tabs on security updates to your dependencies; that's a pretty big tradeoff). Chef (obviously) does the same thing for their Chef server.

We'd originally tried shipping the software as a package with dependencies on other OS packages (installed using shell scripts originally, and then using Chef), but this never really worked: every enterprise would have slightly different OS configurations and the install scripts would practically never work.

We did try Docker as a deployment method, but most customers weren't comfortable with it for production deployments (or simply had systems that were too old to run Docker on). Also, orchestration tooling wasn't there yet. Bear in mind that this was a year ago, though.

--

Locked-down VM images (virtual appliances) are to my knowledge quite common as well (e.g. that's how Github enterprise is distributed). With more and more customers using AWS (and other cloud providers), cloud images are becoming another viable option.

GP is most likely referring to VPCs with dedicated tenancy. Those are not free, and they're pricey [0].

  [0]: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/dedicated-instance.html

Locust is really focused on load tests (rather than acceptance tests). It's an alternative to ab rather than one to a headless browser.

You definitely wouldn't use Locust to test a UI, or even to test your API responds correctly, but you'd use it to know how many requests per second that API can serve without falling over.

---

Two key differentiators (advantages?) Locust has over ab are:

+ It's scriptable, so you can design complex interactions like described in the blog post (or even something as simple as logging in prior to hitting the app).

+ It supports clustering. It's easy to setup a multi-host Locust cluster, and results are aggregated to your locust master.

I don't have an affiliation with Locust, but I did find it extremely easy to setup and use when I needed it (for HTTP load tests with ~20K RPS).

I do recall a few scale issues (e.g. running out of client ports would break the clustering), but with a bit of fine-tuning, it runs like a charm.

This is exactly what we're doing at Scalr (http://www.scalr.com/). Our experience has been the following:

+ An open-source version that folks can download, install, and trial is a great way to get your software in as many hands as possible. However, that's also more work for you: if folks download your software and it doesn't install / has bugs they can't troubleshoot, you're providing a very bad experience, and that will reflect badly on your commercial offering.

+ Research your market first. In our case, it's increasingly difficult / unpopular today to provide developer / IT tools that aren't open-source. In other words, being open-source is becoming a requirement.

+ Open-source offers reassurance to enterprise customers who might not have considered your product otherwise (you're small, and they don't want to tie their success to your survival, but if your product is open-source, this concern is more limited). There again though, it depends on who're selling to.

+ Our product hasn't been forked / used by a competitor. This wasn't really ever a concern. Note that we're following a model where we do open-source releases on a 3/6 month basis (we don't develop in the open); that's a reason, too.

+ We did lose several prospects to our own open-source software, but we also acquired several customers who transitioned from the OSS edition to the commercial edition. the only incentive at this time to choose the commercial edition instead of the open-source one is "support and more frequent updates" (which is good, but not always enough to get prospects to sign you a check). Before you decide to open-source your product, think very hard about how you'll differentiate your open-source and commercial editions (options include delaying updates, differentiating on features, support...).

+ Most of our revenue is now coming from the commercial single-tenant edition, whereas it used to be the SaaS edition. This happened after we pivoted to the enterprise, and made it much easier to deploy your own Scalr installation (it used to be a pretty daunting task).

+ We're not getting that many contributions (probably largely because of the model I mentioned above), and because so far we haven't had the energy to encourage them. In other words: just putting your product out on GitHub doesn't mean some folks from the internet will suddenly start working on it for free. That just doesn't happen. We did get a few very-well-thought-out bug reports, though!

Ultimately, I'd conclude with saying that open-sourcing your product is a solid way to get more attention. However, it's also more work: you're creating a competitor (your own product) that you must somehow beat in deals, all the while making sure that that competitor is solid enough that it doesn't turn people off your product.

Hope this helps!

You can reach me at thomas@scalr.com if you have questions you think I could answer!

----

One last thing that isn't stricly-speaking sales related is people. We have an open-source culture here at Scalr, and there are employees (including myself) for whom that matters a lot.

AWS actually does that already — they'll ping you by email if they find one of your keys on GitHub!

It seems like you're limited to 25 concurrent lambda functions executing (at least at this time).

This looks like it's more geared towards enabling you to automate / orchestrate your AWS resources, that run customer-facing code.

Apparently the URL isn't correct. Try: http://www.kevlar.io/

One small thing that makes Consul exciting (especially when compared to etcd) is its -bootstrap-expect flag [0].

It's the recommended way to stand up a Consul cluster without a risk of causing split brain if it turns out multiple nodes are coming up at the exact same time, and it achieves that without an external / pre-existing coordination service.

In comparison, etcd's recommended way to bootstrap a cluster is to use CoreOS' "discovery" service, which is great, but does constitute one extra dependency.

--

Now, the downside of Consul is that it seems etcd has much more momentum.

  [0]: http://www.consul.io/docs/guides/bootstrapping.html

If you're using gmail / google drive, a good option is to make the google drive file public and use goo.gl (or bit.ly) to shorten it!

The added benefit is that you can do this from mobile, which may not be possible with S3!

Note that Spark 1.0.0 makes it possible to trivially submit spark jobs to an existing Hadoop cluster.

It leverages HDFS to distribute archives (e.g. your app JAR) and store results / state / logs, and YARN to schedule itself and acquire compute resources.

It's pretty amazing to see how you use Spark's API to write functional applications that are then distributed across multiple executors (e.g. when you use Spark's "filter" or a "map" operations, then the operation potentially gets distributed and distributed on totally different nodes).

Great tool — exciting to see it reach 1.0.0!

Note that the HTTP RFC mandates that browsers should not send referers when doing HTTPS -> HTTP, so this may not be the case here.

But, yeah, this was probably to work around the duplicate detector.

Under this model, how do you store user (database) credentials?