785 karma · joined July 10, 2008
http://not.mn
me@not.mn
[ my public key: https://keybase.io/notmyname; my proof: https://keybase.io/notmyname/sigs/o8HTTJBpNeFcOdVzr5_I2BPq80EJisc6g3UvrokAxN4 ]
https://www.superhouse.tv/product/wheelchair-control-breakou... is the result
found via https://twitter.com/imadsousou/status/1032680311753072640
In my experience with a particular open source project, more than half of new contributors started by communicating privately with myself or some other leader in the project. This is only bad if the communication never moves out of private conversation. I've found it normally takes a few nudges to get people to build their confidence and participate publicly.
Interestingly, it's not only new contributors who suffer from this. I've been told by some of the most prolific contributors to our project that they are still intimidated to speak out publicly. Many times these feelings come from deep-seated cultural norms that don't really fit with traditional Western/American ways of communicating. And that's ok! We all learn together how to best communicate with each other. And we get a healthier community and better code out of it.
There's different levels of "open source" (everything from one-way code dumps to full-on maintained-as-a-whole-distributed-global-team). In my experience, it's easier to start with a simple "here's the code, bug reports welcome". This starting point is generally an easier sell to management who's worried about project management taking too much time away from other work.
Like all things, practice, start small, and grow from there. Get's easier as you go. But yeah--licenses/legal, written policies, governance, marketing, time prioritization, and more all take a lot of time to figure out.
NB: My own background in open source biases me to thinking open source is "easy". It's not; it takes a lot of work. The good news is that there's a lot of tools and help available for anyone wanting to start.
It's an object storage engine (think S3, but it's open source and you can put it in your own data center) that's excellent at storing unstructured data.
It's completely deployable and usable without any other OpenStack projects.
There's S3 API compatibility for it. It supports globally distributed clusters. It supports multiple storage polices that can be either replicated or use erasure coding. It's designed for very high availability, very high durability, and high aggregate throughput.
One of my favorite features is being able to create sharable, expiring signed URLs to any object in the cluster.
Some of the common uses for Swift include storing user-generated content (eg images, videos, game saves), static web assets, movies, scientific data sets, backups, document sharing, VM and container images, etc.
API docs: - https://developer.openstack.org/api-ref/object-storage/
Docs: - http://swift.openstack.org
Vagrant All-In-One setup: - https://github.com/swiftstack/vagrant-swift-all-in-one
Come say hi! - #openstack-swift on freenode IRC (I'm notmyname)
The only way to durably store data so that it survives a hardware failure (e.g. drive dying) is to store more than one copy. Full replicas are the simplest way to do this, but you've got a relatively high overhead (e.g. Store 1GB of data with 3x replicas, and you store 3GB of data). Erasure codes are a way to effectively store fractional replicas, so you only use 1.5x or 1.7x of the original data.
Erasure codes are great when you've got a lot of data and you need high durability but don't want to pay for the storage space required for full replicas.
Why don't we always use erasure codes for everything? EC isn't great when you've got small bits of data, and since there's a bit of math involved in reading and writing the EC data, EC has higher latency than simple replicas.
https://www.swiftstack.com/blog/2015/04/20/the-foundations-o... is a great into to how erasure codes work.
I've been working on Swift for the past 6 years, and I'd be happy to help with your OpenStack Swift API integration. See my profile for contact info.
There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.
https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.
https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.
I'm curious about how you're managing the data on a drive itself. Are you storing the blocks as individual files on a filesystem? Are you doing direct management of the block device itself? Something else?
We build tools to help our customers build object storage based on OpenStack Swift. Our customers have big storage needs and need to store their data easily, safely, and cheaply in their own data centers. We're active contributors to OpenStack Swift, and everything we put into Swift we push upstream.
One reason I love working at SwiftStack is because it lets me see a big range of storage use cases. We've got customers using SwiftStack to store everything from backups to game assets to genomics data for cancer research.
Check out one of the things we've recently been working on: adding Erasure Codes to OpenStack Swift: https://swiftstack.com/blog/2015/04/24/behind-the-scenes-“un...
Reach out to us if you have any of the following:
* Experience writing high-performance server software (Python/C/C++/Go)
* A deep and abiding interest in distributed (storage) systems
* Vision that simplifies complex user workflows (front end/UI/UX)
But if you're interested in ACLs in OpenStack Swift, http://swift.openstack.org/overview_auth.html is a good place to start.
(all tongue in cheek)
We're a storage company focused on solving a lot of the hard problems around deploying and managing scalable storage systems. We build a management controller [0] around the OpenStack Swift [1] storage engine, and we sell that controller to customers. Some of our customers include eBay, PAC-12, and HP [2]. Our focus is on ensuring that the OpenStack Swift storage engine is well integrated into enterprise IT infrastructure and processes. This means we spend a lot of time automating day-to-day operations tasks, providing good visualization of monitoring data, integrating with auth systems, selecting hardware, building utilization tools, and on and on.
But we also get to build against new technologies[3], actively contribute to open source[4], and partner with major companies to build cool new features into Swift[5].
Interested? Send us an email at jobs+hn@swiftstack.com. Tell us who you are, why you're interested in SwiftStack, and why we'll be interested in you. We'll get in touch with you shortly to continue the conversation.
[0] https://www.youtube.com/watch?v=7Aue0eoMGCA
[1] https://wiki.openstack.org/wiki/Swift
[2] https://swiftstack.com/blog/2014/06/25/ebay-pac12-and-hp-hel...
[3] https://swiftstack.com/blog/2013/10/22/kinetic-for-openstack...
[4] http://stackalytics.com/?release=all&company=swiftstack
[5] https://swiftstack.com/blog/2013/07/17/erasure-codes-with-op...
SwiftStack is a storage company building and deploying object storage for our customers (think S3 in your own datacenter). Our product is a management controller that works with OpenStack Swift. Swift gives deployers the ability to create scalable storage pools for modern applications, and SwiftStack provides the management tools for day-to-day operation of the cluster. We charge our customers a recurring license fee for the management software based on how much storage capacity they have deployed.
SwiftStack (the company) is about 2.5 years old. We've got about 30 employees, and we're actively hiring developers. Here's a short list of some things we're currently working on:
- New features and bugfixes in OpenStack Swift
- Making clean, responsive UIs to simply communicate information about large storage clusters
- Using metrics to diagnose errors and proactively fix them for customers
Everyone has data, and it's always growing. This growth in data means that we need an effective way to store massive amounts of data. OpenStack Swift is designed to effectively store massive amounts of data while providing high durability, high availability, and support massive concurrency across the entire data set. SwiftStack is building management, monitoring, and integration tools to turn the Swift storage engine into an effective storage solution that is used by millions, daily.
We're looking for engineers of all experience levels to join us. We do most of our work in Python, but we've got some Ruby, Javascript, and a few other things mixed into our stack.
So far this year, we've built out global clusters for a major expense reporting and travel support site[1], provided the storage system for Disney Interactive's gaming platform[2], and integrated Swift into radical new hard drive technology from Seagate[3]. We're currently working with companies like Intel and Box to build support for erasure codes into Swift[4].
My vision for OpenStack Swift is that it will be used by everyone, every day, even if they don't realize it. Swift is storage that is built for the web and already powers some of the world's largest storage clouds. I would like to work with you at SwiftStack to see this vision become a reality.
Interested? Send us an email at jobs@swiftstack.com. Send us your github profile, your LinkedIn account, a link to your website – whatever will best display the work that you’ve done. Tell us in a few lines of text why you’re interested in SwiftStack, and why we’ll be interested in you. We’ll be back in touch shortly to get the conversation started.
[1] Concur video: http://www.youtube.com/watch?v=qQmGfzewWQw&t=29m23s
[2] Disney + SwiftStack: http://venturebeat.com/2013/08/30/how-disney-built-a-giant-c...
[3] SwiftStack + Seagate: http://swiftstack.com/blog/2013/10/22/kinetic-for-openstack-...
[4] Building erasure codes into OpenStack Swift: http://swiftstack.com/blog/2013/07/17/erasure-codes-with-ope...
SwiftStack builds and delivers on-premise object storage to companies looking for highly available storage that scales to millions of users. Our product is built around the OpenStack Swift[1] object storage engine and provides management, monitoring, and integration for the entire storage system.
Working at SwiftStack, you'll work with new technology[2], participate in open-source community work, and build a storage engine that is used by millions. We're solving all kinds of interesting problems, and I'd love to work with you to solve them. Three things we're working on right now are:
- building erasure codes into OpenStack Swift[3]
- making clean responsive UIs to simply communicate information about large storage clusters
- analyzing system metrics to intelligently respond to and predict system behavior
Interested in knowing more? Send us an email at jobs@swiftstack.com. Send us your github profile, your LinkedIn account, a link to your website – whatever will best display the work that you’ve done. Tell us in a few lines of text why you’re interested in SwiftStack, and why we’ll be interested in you. We’ll be back in touch shortly to get the conversation started.
[1] OpenStack Swift dev docs: http://swift.openstack.org
[2] SwiftStack + Seagate: http://swiftstack.com/blog/2013/10/22/kinetic-for-openstack-...
[3] Erasure codes in Swift: http://swiftstack.com/blog/2013/07/10/erasure-codes-with-ope...
Everyone has data, and it's always growing. This growth in data means that we need an effective way to store massive amounts of data. OpenStack Swift is designed to effectively store massive amounts of data while providing high durability, high availability, and support massive concurrency across the entire data set. SwiftStack is building management, monitoring, and integration tools to turn the Swift storage engine into an effective storage solution that is used by millions, daily.
We're looking for engineers of all experience levels to join us. We do most of our work in Python, but we've got some Ruby, Javascript, and a few other things mixed into our stack.
So far this year, we've built out global clusters for a major expense reporting and travel support site[1], provided the storage system for Disney Interactive's gaming platform[2], and integrated Swift into radical new hard drive technology from Seagate[3]. We're currently working with companies like Intel and Box to build support for erasure codes into Swift[4].
My vision for OpenStack Swift is that it will be used by everyone, every day, even if they don't realize it. Swift is storage that is built for the web and already powers some of the world's largest storage clouds. I would like to work with you at SwiftStack to see this vision become a reality.
Interested? Send us an email at jobs@swiftstack.com. Send us your github profile, your LinkedIn account, a link to your website – whatever will best display the work that you’ve done. Tell us in a few lines of text why you’re interested in SwiftStack, and why we’ll be interested in you. We’ll be back in touch shortly to get the conversation started.
[1] Concur video: http://www.youtube.com/watch?v=qQmGfzewWQw&t=29m23s
[2] Disney + SwiftStack: http://venturebeat.com/2013/08/30/how-disney-built-a-giant-c...
[3] SwiftStack + Seagate: http://swiftstack.com/blog/2013/10/22/kinetic-for-openstack-...
[4] Building erasure codes into OpenStack Swift: http://swiftstack.com/blog/2013/07/17/erasure-codes-with-ope...
On the practical side, this means that the drive is "connected" to one or more servers (and can be re-homed on the fly). The storage system itself is responsible for coordinating the communication across the cluster of Kinetic drives. In our case, the coordinating storage system is OpenStack Swift.
On the humorous side, this means that technobabble like "Can you ping the boot record?" or "What does a traceroute to the directory show?" actually sorta make sense now.
1) Server groups of at least 3 mirrored servers, with a max of 5TB.
This seems like an interesting design choice. What do you mean by "at least"? Does this mean you'll have some data with more replicas? Are these server pools filled up and then powered down until they are needed? How do you choose which server pool to send the data to? And since you have a mirrored set of servers, when do you send a response back to the client?
Is the 5TB number something that is a limit for the storage server (ie 15TB total for a cluster of 3)? That seems rather low. It also doesn't divide evenly into common drive sizes available from HDD vendors today. So what kind of density are you getting in your storage servers? How many drives per CPU, and how many TB per rack? Since you're advertising on low price, I'd think very high density would be pretty important.
2) You say you split data into smaller chunks if it crosses some threshold. Let's suppose you split it into 1MB objects, once the data is bigger than 5MB. And each 1MB chunk is then written to some server pool which has replicated storage (via the mirroring). How do you tie the chunks back to the logical object? Do you have a centralized metadata layer that stores placement information? If so, how do you deal with the scaling issue there? If not, another option would be to store a manifest object that contains the set of chunks. But in either case, you've got a potentially very high number of servers that are required to be available at the time of a read request in order to serve the data.
Just as an example (and using some very conservative examples), suppose I have a 100MB file I want to store and you chunk at 10MB. So that means there are now 10 chunks replicated in your system, for a total of 30 unique drives. Now when I read the data, your system needs to find which 10 servers pools have my chunks and then establish a connection with one of the servers in each server group. This seems like a lot of extra networking overhead for read requests. What benefits does it provide that offset the network connection overhead?
And what happens when one of the chunks is unavailable? Can you rebuild it from the remaining pieces (which would essentially be some sort of erasure encoding)?
Overall, the chunking and mirroring design choices seem to me like they would introduce a lot of extra complexity into the system. I'd love to hear more about how you arrived at these choices, and what you see as their advantages.
In order to not make my long post even longer, I'll not pursue more questions around listings, failures, or hot content.
Starter questions:
How does data placement work? How is data checked for correctness? How do you do listings at scale? During hardware failures, do you still make any durability or availability guarantees? How do you handle hot content? How do you resolve conflicts (eg concurrent writes)?
SwiftStack (http://swiftstack.com/jobs/) is hiring software developers at all levels of experience.
Why Join SwiftStack?
You’ll work on a product that touches millions of people’s lives even if they don’t know it. The number of businesses using OpenStack in general and Swift in particular grows and grows, and it includes big names that make products that are used by both your little brother and your grandmother. You’ll work on a product that makes core OpenStack technology accessible to businesses of all kinds.
You’ll get open-source experience in a big way. At SwiftStack, we’re committed to strong participation in the OpenStack ecosystem in general and to contribution to Swift in particular. You’ll be a contributor to one of the most important Open Source projects currently active.
You’ll confront interesting problems every day. Writing a system like Swift and building a software ecosystem to surround it is the road less traveled. We aren’t writing yet another glorified CMS or social app or phone game. Whether it’s figuring out better algorithms for data placement, confronting a firehose of monitoring data, or determining how to integrate most flexibly with customers’ systems, there are always new and unusual problems to solve.
Interested? Send us an email at jobs@swiftstack.com. Send us your github profile, your LinkedIn account, a link to your website – whatever will best display the work that you’ve done. Tell us in a few lines of text why you’re interested in SwiftStack, and why we’ll be interested in you. We’ll be back in touch shortly to get the conversation started.