23david

Fixing the H1-B program so that it's harder to game would probably be a great help. H1-B workers also should be given a way to voice complaints or report abuse without fear of repercussions from their host companies.

But not sure that reforming H1-B will be an easy battle to fight. Many of the companies that have large H1-B workforces are also very politically connected and stand to lose a lot of money.

As PG states:

  it's easy to imagine cases where a great programmer might invent things 
  worth 100x or even 1000x an average programmer's salary.

If this is true, rational companies should be willing to pay salaries between 100-1000x of average for great programmers.

All hail the $100M/yr rockstar programmer.

I can't wait to see the cool things that will come out of a greater distribution of wealth to hackers, geeks and programmers. Think of all the neat kickstarter campaigns that will get funded... and all of the startup ideas that can find angel funding. And all the open-source and Makerspaces and rockets etc, etc.

Skilled programmers are able to generate incredible economic value, and companies having difficulty hiring should consider whether they are undervaluing programmers when they set salary ranges.

It's possible to quickly test this by simply adjusting the salary ranges upwards until good candidates start accepting job offers. Believe me... it works.

I think it's important for all startup founders to realize that non-executive salaries in the U.S. tech industry have been incredibly distorted by the illegal wage fixing practiced over the last 10-15 years.

The wage fixing set an artificial ceiling on programmer salaries. And if it's now stopped, we should expect a major correction over the next 5 years as market wages adjust. I'm hopeful that recent salary increases by large companies are evidence that things are moving in a good direction.

But it will take time to rebuild trust between labor (programmers) and management in the tech industry. I don't think we're there yet, and in the meantime I am cynical about any effort to address hiring difficulties by increasing immigration.

Definitely don't set up LDAP for hologram... but if you already have LDAP set up, it could be worthwhile to modify your setup to support hologram.

But then again, if your org is large enough to have LDAP set up, getting LDAP policies changed to support something like hologram would generally be a non-starter in my experience.

In large enterprises, the LDAP service is a core security service that is rarely if ever modified except for periodic maintenance and upgrades.

Messing up the LDAP service for any reason will compromise or disable all authentication company-wide. It's not fun when the CEO comes over and says that his wifi and email logins aren't working anymore...

Interesting. Not sure about the requirement for "an LDAP server with sshPublicKey attributes on user records to store their SSH public keys." https://github.com/AdRoll/hologram

I know you can put anything into LDAP, but why treat it like a metadata service? Wouldn't it make more sense to store keys in something like etcd so they're accessible via simple REST api calls? In my experience, the less LDAP does, the happier everyone seems to be.

  In the case of the kernel, this prevents us from distributing ZFS as part of 
  the kernel binary. However, there is nothing in either license that prevents 
  distributing it in the form of a binary module or in the form of source code.

Source: http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue

  ZFS cannot be added to Linux directly because the CDDL is incompatible with the GPL. 
  ZFS can, however, be distributed as a DKMS package separate from the main kernel package.

Source: https://wiki.ubuntu.com/ZFS

So why not just distribute the DKMS package with a distro? Easy enough. Will it ruin the user experience somehow?

The author is a developer and professional security researcher. He responds to the security disclosure issue on the github issue page:

  > @ewindisch:
  > I also remind everyone that if they feel there may be possible attacks 
  > against the current format, not to publicly discuss this on GitHub. 
  > If you do feel this way, I'll happily entertain a private discussion. 
  > While I generally prefer and encourage transparency in open source 
  > projects, we should be careful to practice responsible disclosure.

  Disclosing security vulnerabilities is a responsible thing to do. 
  As a security researcher, it is entirely my choice how/when/if I disclose
  security issues. In this case, I'm not dropping any 0-days, just pointing out
  fundamental flaws in the current system. Fixing these flaws should be an 
  open discussion, not a private one.

  As far as "responsible disclosure" goes, it is only one vulnerability 
  disclosure approach (the alternative is not "irresponsible disclosure"), 
  and there is zero consensus about it.

Source: https://github.com/docker/docker/issues/9719#issuecomment-67...

Some good reading by Bruce Schneier re:full disclosure vs responsible disclosure: https://www.schneier.com/essays/archives/2007/01/schneier_fu...

If you want to really understand the security world, I definitely recommend attending Defcon in Las Vegas at least once to meet our Cyber brethren... :-)

Companies with the resources who publicly state that they care about security should be willing and excited to at least partially sponsor employees to go. Definitely worth every penny.

The business model seemed clearer before with the commercialized dockerhub. What is the business model with all these new features?

Seems to me that they'll either license tech to enable enterprises to build their own docker platforms, or offer their own hosted platform... dotcloud 2.0

I think the "handwringing that docker should do less" has a lot to do with the fact that just getting Docker containers right is a huge effort in itself, and they have a long way to go even there. It's not a solved problem by a longshot.

The inevitable CVE's coming from this report will definitely get their attention. Hopefully the adults in the room will help make sure that the Docker team addresses what up until now has been a really lax approach towards security.

Who is the architect in charge of this, and do they have any security chops? If not, it's just a matter of $$$ to get a 3rd-party security review before every major release. I've done it before, and it's really not a big deal.

hrm... what's the failure scenario here? Are there any good btrfs management utils that can help with running btrfs in prod?

Funny thing is that if ZFS happens in Docker it'll be only because the community demanded it over the opposition of the core Docker maintainers. AFAICT, Aufs is being dropped due to massive opposition from Redhat. (Apparently AUFS code is too horrible to merge/support???) Despite its spotty track record, BTRFS is the only current recommended/supported CoW file system, and OverlayFS looks to be the docker-blessed next-gen Aufs alternative.

Based on past experiences, I don't see Docker-blessed ZFS support coming anytime soon, but I hope I'm wrong. Maybe someone over at Joyent or Oracle can grease the wheels here? :-)

One thing you might want to check is if you're setting a limit in the driver for the amount of memory ZFS uses for caching. By default it'll use a LOT of memory, so I usually just set to a max of 2GB and don't see any issues.

This is ridiculous. CoreOS is managing their own distro, so why not just package in ZFS and get it over with.

It's been how many years now waiting for the next great ZFS competitor? If nobody is able to improve on ZFS, how about we just all jump on the bandwagon and move on with our lives?

If nothing else, having more people using ZFS may inspire someone to actually improve on it. BTRFS has such a limited feature set in comparison to ZFS that with the advent of ZFS on Linux, it's really hard to understand the community's continued backing of BTRFS as the Linux community's CoW filesystem competitor to ZFS. Spend a few weeks using ZFS on Linux on a machine with a few SSDs and a few HDDs, or just enable it on a linux laptop with a SSD drive, and it's hard to go back to anything else.

For managing volumes, ZFS on Linux works great. But for managing NFS, I'd definitely go with a separate NFS implementation if I wanted to heavily use NFS. The primarily developers/users of the ZFS on Linux port are mainly using it for highly-available single-machine volume management, and exporting volumes to gluster or other clustering filesystems for use in massive HPC clusters like those over at Lawrence Livermore National Labs.

http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-...

ZFS is meant for managing local drives, and to make it performant you need to configure SSD partitions to act as an L2 Arc cache. The online documentation is pretty good, so after going through the docs it should be pretty clear how to set ZFS up properly for your use cases.

But it sounds like you're using EBS drives? If so, not sure why you'd want to use ZFS. Last I checked, ext2 or xfs was the way to go with EBS drives on AWS. AWS has so much stuff going on in the background to ensure reliability/availability of EBS volumes that adding another layer isn't worth it IMO and I've seen similar kernel panics running other complicated volume managers on top of EBS.

Awesome^3. Keep it up :-)

For startups relying on Mongodb for critical infrastructure, the cost of having a poorly implemented mongodb cluster can actually be high enough from a morale and product performance standpoint to ruin the company. I've seen it happen a few times to very well funded companies where their inability to quickly solve MongoDB issues made it impossible to continue iterating on their product, and eventually caused most of their core backend team members to quit.

So I think it's super important to de-risk MongoDB as quickly as possible and then move on with life. If you can use bare metal, just switch to using SSDs or even FusionIO cards. The new AWS SSD instance types should also be ok. Pay the money you need to service providers like AWS, Rackspace or Softlayer etc. to get it done.

In a best-case situation, you can keep upgrading every 12-18 months and rely on Moore's law to allow your MongoDB machines to scale with your growth. But even in a bad situation where your database is growing in size too fast, this vertical scaling strategy should still give you 6-12 months of breathing room, to allow your dev team to cleanly switch over to a different database.

I see a lot of teams starting out with MongoDB, but the smart ones know it's a big risk and once they find market fit, their first major initiative is to switch over the database backend to use something like Cassandra + Mysql.

It would be awesome to see a Nix package manager module for puppet, saltstack, ansible, chef & co.

I think that for package installation and rollback, a simple model like that proposed by DJ Bernstein with slashpackage http://cr.yp.to/slashpackage.html seems to accomplish the same goals but with much less complexity.

After using Nix and NixOps, my sense was that a Nix package manager module for puppet&co would be really really useful. With NixOps, my concern was that I need to have a trusted OS, and I didn't feel comfortable having to use a custom build of something Ubuntu-ish.

Maybe there's a way to convert or bootstrap any Linux OS into a NixOps-compatible distro. But I didn't see anything.

i hope it works. i hope it works. i hope it works.

Agreed that this could be very cool.

You left out Juju and MAAS... I hope I'm not the only one who's been burned by those Canonical 'Enterprise' projects.

DevOps tools are rapidly evolving. Now that they've basically finished with machine provisioning, dependency management and orchestration, all major DevOps automation frameworks are going into managing reactive infrastructure. Enforcing a task schedule is just another form of state enforcement.

These are all pretty similar: - configuration management: ensure package oracle-java-8 is installed on machines A,B,C with this specific configuration. - orchestration: ensure my-awesome-java-app on machines A,B,C is running to databases on machine D,E,F - deployment with constraints: ensure that four instances of my-awesome-java-app are running on at least 2 physical machines with over 4TB free disk space. - job runner: ensure that script X runs on a cluster every __ minutes. when script X runs, send the output to script Y

I think that you'll see task placement and job scheduling primitives being integrated into DevOps tooling in the next 6-12 months.

Saltstack already has many of the primitives in place for building out reactive infrastructure, http://docs.saltstack.com/en/latest/ref/runners/all/salt.run...

Would be great to hear about tools for Puppet, Chef, Ansible, other…

Curious why you wouldn't consider Openstack to be a Datacenter OS? :-)

Task distribution primitives are handled by Mesos and implemented in the various apps running on Mesos (Marathon, Chronos, etc.)

It seems inaccurate to call this a feature of DCOS.

Puppet and the like can easily configure and manage Mesos and the various apps running on top, giving all the task dispatch functionality goodness.

DCOS will hopefully be a great integrated Mesos distribution. Seems to me that by supporting machine provisioning and having a per-node licensing model, it's being positioning as a direct competitor to Openstack and VMWare.

lol. mind blown.

"Mesosphere Announces First Data Center OS And $36M In Funding" - Techcrunch

"Mesosphere’s new data center mother brain will blow your mind" - GigaOM

They're general purpose devops automation frameworks... you can choose to take any of them and make this 'deep integrated' layer you speak of.

For example, Openstack deployments are commonly automated using Puppet scripts. (Really amazing stuff if you haven't seen it before...)

DCOS may choose to reinvent the wheel, but unless there's a core innovation in the way they're handling orchestration or automation, I think they're better off if they leverage an existing battle-tested toolchain. Adding another tool will just mean more work for the (underworked and lazy?) DevOps teams who will be responsible for managing DCOS.

Having 2000+ VMs without automation or monitoring in place is pretty unusual. Especially for a site like healthcare.gov which must have had all sorts of HIPAA and data security requirements.

But it's honestly not hard to hook all those machines up to a system like Chef/Puppet/Saltstack/Ansible and start automating common tasks within a few days.

Migrating databases between networks and rotating passwords would generally be outside of the scope of a tool like Mesosphere. Again, this is something that can be easily handled by existing automation tools. With most databases though, there's nothing straightforward about migrating data to nodes on different vlans or password rotations. If it's a one-time task, I recommend hiring a database consulting firm to do the migration or rotation.

I think that have good defaults and enforcing best-practices is a good idea. But I think that a lot of this can be achieved with existing tools. IMO, it makes more sense for organizations to automate and orchestrate Mesos deployments via existing/mature DevOps tools like Chef/Puppet/Ansible/Saltstack. Would also be exciting to see deployments working via NixOps/NixOS.

Master Controller to Minions... configure thyselves...

Datacenter Controller maybe, but calling this an OS?

Chef/Puppet/Saltstack/Ansible/CFEngine/Nix OS