What kind of consistency guarantees does this offer?
134 karma · joined March 15, 2016
Right now, I am not aware of any cheap ip camera that can stream its H264 video to a regular web browser, with sub 500 ms latency. All manufacturers seem to have moved to an app, I guess they can show an RTSP stream in that way.
Older ip cameras had MJPEG which you could view in the browser, but that is really inefficient w.r.t. bandwidth.
Haskell allows this to be done in a way that makes if very hard for the user to break this abstraction (the type system will prevent you from doing this), while still allowing the full language power of Haskell to be used.
For Haskell to become a successful "industrial" language, I think most of the dependent-typing stuff should probably go (to Idris/Agda etc.), so that a clear and consistent Haskell subset can be defined.
The other arguments in the article are just weak. The rant about data not having a type, is missing the point. Sure, often you receive data that you need to inspect to know what it is. You can easily do this in Haskell (just label it with the UnknownData type), and have a function that inspects it and returns, depending on the contents, the right type). The big advantage is that you don't have to keep on doing this same check.
Types being the cause of difficulty in refactoring when business requirements change, is the opposite of my experience. In large dynamically typed codebases, being sure that a large refactor caught everything, is very hard / costly in test coverage. I have seen this go wrong many times. Having the compiler point out what you have missed, based on the types, is very helpful.
While I think the arguments are a bit weak, I do agree that it is at least unclear if Haskell is a sound choice as a production language at this time. Fighting against an ecosystem is not something you want to be doing while building your product. But in contrast to the author, I do think this is fixable, and see steps happening in the right direction (e.g. with IHP, but also with the efforts around the Haskell Foundation and the Haskell Language Server)
IMEC's research mainly benefits companies like Intel, Samsung, TSMC. The only relatively local link is ASML.
Language-wise, it is Ocaml with a different syntax layer, so all of the nice things about the language are still available.
With PPX, the barrier to use metaprogramming has become lower (especially compared to camlp4), resulting in most modern Ocaml codebases depending on it. But the ergonomics of the end-to-end software delivery have not really followed: the fact that PPX's break between compiler versions means that in practice, a compiler upgrade becomes a very costly thing to do. It is difficult to make the case for this in an industrial setting, where languages with almost religious backwards compatibility are common (Java, C++, even Go). For Ocaml, it is especially sad, because the core language is already quite powerful and expressive, lowering the need for metaprogramming (imo).
For scenarios where code generation is needed, I prefer a separate code generation step, which produces explicit "standard" code (e.g. like protoc)
Probably closer would be (not sure if this is possible in Typescript):
class CustomModule<S, T1 extends Module1Interface<S>, T2 extends Module2Interface<S> > {
constructor(t1: T1, t2: T2) { ... }
}
This does become a bit tedious notation-wise, if possible at all. In Ocaml, this would look like:
module CustomModule(M1: Module1)(M2: Module2 with type s = M1.s) module type Comparable = sig
type t
val compare : t -> t -> int
endThe power of the Ocaml module system is really in the functor, which the article only touches upon. You define a module and refer in its definition to types/functions of other modules that must be provided at the time you construct the module. Even better, you can define relations between the types and do type substitutions, e.g.: to construct this module, you need to give me 2 other modules, each with a specific set of functions, and for which type t1 of the first module, matches type t2 of the second module, while the actual type of t1/t2 does not matter.
See https://dev.realworldocaml.org/functors.html for more examples.
My second point was that the popularity of Go (or any language) does not mean that it is inherently a good language, where good can mean, productivity, defect rate, .... Adoption of a language depends on many things, I believe quality of the tools, availability of libraries, and good old marketing (the Google aura around Go has helped in adoption) are often more important. The effort to make an initial, crappy, proof-of-concept is in my experience often decisive for the choice of language. The cost of long term maintenance (where a good type system might bring value) is rarely considered.
For Go, it also hasn't hurt that some of the core developers where being paid by Google to work on it.
The metric of the "ability for mass amounts of programmers to program the computer", is indeed a very relevant one, and I think Go shines there. When reading Go code, I typically find it easy to understand what the goal of the code is. However, "to do the correct thing", is often less of a success: there are off-by-one error and corner cases that have bugs, and reimplementations of the same basic logic. If you have a lot of developers available, you simply have them fix these issues, and your Go project becomes a success.
My point is, that in this scenario, the language itself is of minor importance: success is determined by the fact that you have access to a large pool of relatively cheap labor of reasonable quality (Google or VC funded startups are perfect examples). A 20 or even 50% productivity gain by having a "better" language would simply not matter for the outcome, certainly not if it takes your labor longer to get up to speed.
So, in that context, compared to the alternatives, the quality of Go is relatively high. But I do think that, if during the design of Go they had taken a couple of basic concepts from ML like sum types, polymorphism, and a decent module system, they would have ended up with a language that would make the current Go look unproductive and poor quality, even in that same context. It wouldn't make a difference to Google, but it would make the life of many a developer a little bit more productive and joyful.
And yes, I do know who created Go, but arguments by authority carry little weight.
Haskell is surprisingly well suited for back-end services (e.g REST). Warp (an HTTP server), Servant (a way to define your backend API), and Mio [1] (the Haskell network IO implementation) are amazingly performant and scalable.
Of course, that does not prevent you from messing up on the business logic level. I suspect that they lacked awareness of performance implications of their choices, this is not something Haskell will save you from. On the contrary, parts of the Haskell community do tend to go to over-abstraction, and I believe this does have a negative effect on people writing production software.
Whether or not these advances make a difference (positive or negative) in production is different question, but I'd rather not use "how much is a language used", as an inherent quality metric for a programming language. Otherwise, I fear we might all end up doing old-school Java, Javascript, Visual Basic and ABAP.
However, in this case, MinIO makes the deliberate choice to deviate from AWS S3 behavior, on a very common call (listing objects). I do agree that this has the risk to break applications in non-obvious ways, e.g. the call succeeds, bu t your application behavior differs, compared to running against AWS S3, due to the missing directory entry.
I also find their argumentation in the ticket a bit worrisome, calling the AWS S3 behavior a "blunder". I saw the same hubris when looking at their data path, where they choose to ignore battle-tested approaches like Paxos / Raft for distributed coordination, and instead build their own distributed lock algorithm and implementation: it does not seem to persist state, so resilience to power failure and server/process crashes might not be what you expect.
It is also very hard to prevent people from using the escape hatch of the general purpose language to "fix a problem quickly", while introducing unpredictable side effects.
Scons (using Python) and Gradle (using Groovy / Kotlin) are 2 cases where I have seen this go very wrong, very quickly. I now strongly prefer Maven, even with its warts and XML verbosity.
On the other hand, configuration languages also often suck, the abomination that is HCL (e.g used in Terraform) comes to mind (especially in its early days). Turns out it is really hard to make a good language, and a bad configuration language is even worse than using a general purpose language.
I do like Dhall though: sufficiently powerful so you never have to repeat yourself, while also preventing you from doing fancy stuff you will later regret, all with a very readable syntax, and the type system helps to catch typos / mistakes early. I now use it to generate all config files for tools that consume json/yaml, and often for other text based config files as well (where you can use it as a powerful type-safe template engine)
Couple of things I took away from the experience:
- After an initial ramp-up, most of the developers liked or even loved the language. Downside was the limited tooling/libraries (2010-2015, seems better now).
- Quality of the resulting product was quite high: bug rate was fairly low, especially once experience with the language increased. Turns out sum types, gadts and the module system (functors) are very powerful to express invariants and design composable logic.
- OCaml was successfully picked up by team members without a formal computer science background, or even without a lot of programming experience. Not that it turns everybody into a great programmer, but it seemed to coerce new hires into delivering a working feature, without a huge risk of breaking a lot of other code.
- It is still possible to write crappy / unmaintainable code. But in OCaml it is often as easy (or even easier) to write good code.
- If something did turn out to be broken, it was painful. Debugger support (GDB) was initially non-existent. Under heavy load, we triggered a couple of bugs in libraries, which was quite painful to fix. On the other hand, we also triggered bugs in malloc and in the Linux kernel, which were as painful, so it might have had more to do with the high-load scenario, than with OCaml.
- For really low level stuff, or very high-performance cases, we did need to use the escape hatch to call C code. Interfacing Ocaml with C was not the most pleasant (was before ctypes library). Debugging bugs at that boundary was extremely unpleasant. One trigger for this was the fact that multicore OCaml never materialized, which also damaged the "political position" of OCaml within the company.
- Management (especially after the acquisition) was at best indifferent, and at worst hostile towards it. OCaml was perceived as being difficult to hire for (or outsource). Using more "standard" languages like Java, C++, Python was seen as the safer bet. They tried and often failed (C++ turns out to be damn complex, and maintaining a large Python codebase turned into whack-a-mole, but for bugs.).
Currently, Janus and Nginx (https/auth) run on a cheap VPS, and a device in the NAT network of the ip camera creates a Wireguard tunnel to get the RTSP stream to the VPS without needing to open/forward ports on the NAT.
Ideally, more of the components could run on the camera itself, but I haven't gotten to cross compile anything to the MIPS cpu of the Xiaomi camera. Will contact you so we can chat.
This article is about single-region storage vs. multi-region storage (and how to reduce the cost in this case). There is very little public info available about distributed storage systems in multi-region setup with significant latency between the sites.
- Dropbox did not want to be running a single version of the software / a single region, because they consider the risk of a single software bug / human error resulting in data loss too high. However, the alternative they choose introduces a completely new code base which will have to be battle tested. This increases the risk of a data loss bug, which would affect a smaller fraction of the data, but any significant data loss issue would be game-over for a company like Dropbox. Did they consider partitioning the system into smaller subsets (some single region, other multi-region), using staged roll-outs of new software versions? Or is there really some fundamental incompatibility between Magic Pocket and multi-region?
- The "New Replication Model" story sounds a bit too simplified. It seems to re-introduce some issues that the single region Magic Pocket solution had already solved: the size of the IO operations becomes quite small again (fractions of the 4M block size), placement of data on the disks becomes less predictable, which could cause increasing rebuilt times when a disk fails. Also, the number of IOs to read or write an object increases significantly (2-3x in the example), which means that the observed advantages in latency go hand in hand with a 2-3x lower maximum supported load than in the Magic Pocket case, before the latency explodes due to running out of IOPS on the HDD's. The whole design seems to ask for far more IOPS than the Magic Pocket solution, which sounds like an odd match to SMR HDD's.
These issues are maybe alleviated by the fact that moving data to the cold tier happens asynchronously, and the cold data is accessed very infrequently, resulting in far less IOPS being required for the cold storage region. However, it also makes the option of combining hot and cold data on a single disk much more difficult (which for HDDs is the way to make optimal use of the limited IOPS vs. their huge capacity - I suspect Amazon / Google use this for their near-line storage solution). Moving from the 2+1 example to e.g. 4+1, to reduce cross region storage costs even more, becomes now a though call as it now goes hand in hand with an even larger increase in IOPS cost.
- The claimed "simplicity" of deleting data in the proposed scenario is rather relative. If they are using SMR drives, deleting data and reclaiming space are complex and expensive operations. They might reduce it to a non-distributed problem (which is still a significant gain, of course), but it is far from trivial.
Probably a lot of the finer, left-out details of their cross-region system address these issues, and if not, maybe the cross region system and the single region Magic Pocket solution will converge again in a later phase.
Anyway, thanks for sharing this data, I really like the openness.
In effect, you get the total bandwidth of (N-1) HDD's working in parallel. And the bandwidth of 100 HDD's doing sequential IO in parallel is really massive ( ~ 10 GB/s).
Examples of companies claiming to use this approach are Qumulo (rebuild in couple of hours), Infinidat (couple of 10's of minutes), ClusterStor GridRAID (now part of Seagate I think), or "Declustered RAID" in GPFS (IBM)
I believe Ceph now has abstracted the drives away through BlueStore, which simply puts a large RocksDB database on the drive, bypassing most of the functionality a filesystem offers. It should be much easier to make an SMR compatible version of the LSM-tree backend of RocksDB, than writing a full-blown file system.
If you look at the design of, for example, DropBox Magic Pocket, or Infinidat & Qumulo, you'll notice that their HDD access is really as sequential as possible. And if your storage layer is thus already optimized towards sequential writes, why not take the opportunity and get some capacity "for free" by adopting SMR drives?
It contains optimized Galois Field multiplication, resulting in Reed Solomon (both Vandermonde as well as Cauchy) at multiple GB/s on a modern x86 CPU.
Still, I doubt that the Reed Solomon coding speed is the limiting factor in their rebuild time. There is a mention of a 6-day duration, so even with a very slow Reed Solomon implementation ( ~ 100 MB/s) that should not be a bottleneck for a 10 TB drive rebuild (assuming a distributed rebuild approach, not a traditional RAID style rebuild).
In my opinion, a nice middle ground between the B tree and LSM approaches.