Column – High-performance, columnar, in-memory store with bitmap indexing in Go
github.com
github.com
This project seems very similar to Apache Arrow, if OP or anyone else is around to explain why one might be used over the other that would be great.
Arrow is primarily a serialization format to transfer data between distributed systems. It uses zero copy and other techniques to quickly process, and store large data sets in memory.
Other libraries allow you to query Arrow data once processed.
This project is an in-memory columnar data store with querying and other capabilities.
On the flip side, you could actually fit more data in-memory than with non-columnar methods, since the storage is column-by-column, it compresses very well. For example boolean values are stored as bitmaps in this implementation, strings could be stored in a hash map so there's only one string of a type that kept in memory, even if you have millions of rows.
Recently I'm starting to like this type of programming over OOP. Each part of your application accesses a global in-memory DB with the full state. But unlike a traditional persistent db it's really fast.
https://www.amazon.com/Component-Software-Object-Oriented-Pr...
Programming against Objective-C protocols, COM interfaces, Component Pascal framework, and so forth.
Component programming with interface separated from state is exactly what Objective-C protocols, COM, VBX, SOM, Component Pascal were all about.
Those that promote ECS as not being OOP 99% of the times never read books like the one I linked on my comment.
Instead they reference a talk done at GDC by one of the very first engines that made it well known to those that never read CS papers or books.
In ECS (where Components are a bag of data, and systems handle all logic/operations), and like a DB, the object is defined by an id and its relations; from the relations, you can derive available capabilities.
That is, an object tells you what it can do. In a DB, what it can do tells you the object.
You can create the same system with interfaces by simply ignoring the methods part of it, and keeping the data part, but associating data with capabilities is pretty much the defining difference between objects and structs.
More importantly from an architectural perspective, in ECS the logic isn’t associated with the object, it’s associated with a system that takes the object as input. The system is shared across all objects. The object (entity) for ECS is little more than an id and some relations.
An ECS very directly corresponds to an RDBMS. To call it OOP is to deny the ORM’s classic Object-Relational mismatch.
Secondly most languages with OOP support aren't Smalltalk/Java, rather multi-paradigm, e.g. Objective-C, Component Pascal, C++, Delphi, Python, among others when Component Programming came into CS papers for the first time.
To argue that Component Programming is not OOP is just religious hate that shows lack of knowledge regarding CS literature.
Yes, I addressed this case. It can be done, but it rips the “object” out of it — you’re no longer sensibly organizing things under an “OOP” paradigm, but something else altogether. You’re quite directly reducing the “object” to a struct.
> Secondly most languages with OOP support aren't Smalltalk/Java, rather multi-paradigm, e.g. Objective-C, Component Pascal, C++, Delphi, Python, among others when Component Programming came into CS papers for the first time.
That’s fine, but doesn’t really support a case that it’s an “OOP” architecture/mindset/organization. In fact, that rather undermines your case..? A language that isn’t interesting in being purely OOP (is anyone?) is happy to introduce an arbitrary construct is not an argument that the construct must therefore be “OOP”
> To argue that Component Programming is not OOP is just religious hate that shows lack of knowledge regarding CS literature.
It’d be more about diluting the term into nothingness — closures are a poor man’s objects, and objects are a poor man’s closures. It’s technically true, but it wouldn’t be useful to go ahead and describe all functional programming as “OOP”, because the main interest in defining the terms is to indicate architectural and logical flow/patterns one would expect under such a paradigm/organization.
Interfaces and classes with no defined methods looks and feels much closer to Haskell and C than it does C#, python and smalltalk.
OOP is not a technical definition. It’s an organizational strategy, and the term itself is just a marker on a continuum. Everything is OOP, and nothing is, just as all things are Turing machines. But that’s not the point.
Apparently me in the mid-90's porting an particle engine from Objective-C NeXT into Visual C++ using COM on Windows, with a component based architecture, did not happen.
I didn't say it was. The key par was "state is managed seperately from logic."
> Component programming with interface separated from state
There is no such things as interfaces in ECS. Interfaces are a way of describing how state is bundled with logic. ECS does not do that.
> Those that promote ECS as not being OOP 99% of the times never read books like the one I linked on my comment.
I'll admit that I have not read that book. Your condescending appeal to authority here doesn't actually promote conversation.
Please, tell me what the "objects" are in ECS or what else qualifies it as OOP?
Funny why do most ECS frameworks use interfaces/traits/pure virtual classes/static polymorphism then, since they don't exist according to you?
https://github.com/skypjack/entt
Component orientend programming is a subset of OOP, as for why that is, I provided a book, feel free to educate yourself.
More CS papers are available on demand.
But state is managed separately from logic pretty much anywhere where you don't store lambdas as fields of structs.
If I had an array of a million things, and I wanted to specify some large subset of them via a separate million element array (like in numpy/pandas), is it faster to do it via a million bytes or via a million bits (ie I think this is bitmap indexing, right?). I would think that the bytes would be faster, even though terribly wasteful of memory. From my rudimentary knowledge of CPUs I thought they didn't really operate at the bit level, and so you'd have to do a few instructions of calculations. Or would it be made up for by the cache line reading in more of the indexer in one fetch?
[0] http://cowboyprogramming.com/2007/01/05/evolve-your-heirachy...
As a reformed Java developer I can say that docker didn’t add much time to the build cycle and gave us a better way to package resources for Java code, but Go is far more ergonomic, so taking a <2 second compile time for a small microservice and adding docker to turn it into a 30 second build time just isn’t worth whatever utility you get from containers at dev time.
Docker helps manage all of this, and does it fairly quickly, and made life relatively easy, but not without a cost in time and complexity.
Go, out of the box, produces statically linked machine runnable binaries, including embedded resources, so you get the equivalent of an uberjar, plus resources, plus the runtime, all in a single executable file. And all of this pops out in a second or two with `go build`.
AOT for Java might perhaps have similar advantages except that AFAIK (two years ago) the AOT compilers were expensive and had plenty of caveats with eg reflection. I expect they would be even slower than javac as well. So certainly a solution, and maybe you don’t need docker any more, but then you have a different set of problems. It was never feasible when I was doing Java.
To be clear, this isn’t a Java vs Go thing. The question was why don’t Go devs use Docker, and I’ve given some reasons. I quite like the Java language and miss some aspects of it, but there is a lot about the Java environment that I don’t miss and runtime deployment complexity is one of them.
I never been into US, plus the restrictions apply to any tech produced in US, regardless of the programming language.
Thankfully, by having such laws, US made us create other standards as well.
I also don't want to make it into a Java vs Go thing, rather make the point that many dismiss Java without really knowing what is around during the last 26 years on the ecosystem.
It appears everyone just learns the basics and then complains from there.
Not targeted at you, as you obviously got my point.
On the other hand, kubernetes and docker are all about runtime deployment complexity. It feels like using Websphere 5 all over again, with containers == EAR, thankfully so far I managed to stay mostly away from them.
Re the export restrictions, although you are right in theory, it doesn’t seem to affect Go. There is no special build, the crypto is just built in. Java is unique in how it dealt with this, I never understood why it was so hard.
I agree with you re K8s. And I like the comparison to EARs. Both container systems are pretty poor substitutes for a binary you can just run in an OS.
Go seems to recognise this. It knows its place in the deployment hierarchy and that’s made my life so much easier. Go feels like it’s part of the Unix world, rather than apart from it, and Java was never like that. That’s why docker became so important in the Java world. It gave Java the isolation from the OS that it always craved :)
Ask the developers on countries with blocked access due to US sanctions how they get to use Go without workarounds.
Go was written by two of UNIX people as better C, not as a cross platform tool. In fact there are still language features missing from Windows support.
Java always felt at home in Solaris thought.
Docker isn't that relevant on my Java projects, it is only used in projects where customers want to feel modern or already have it as part of their workflow regardless of the technology stack.
I am pretty much 99% of the time on VM + scripting world.
We build images (about 20, each with a Dockerfile) from a monorepo with a single go.mod. I have basically a full replica of prod running locally in k3s — letting k3s manage it all is easier than dealing with the pile of environment variables that would be needed to get everything hooked up properly. And with kustomize, we can reuse a bunch of yaml from prod.
Sometimes I’ll run go binaries locally on my machine for debugging (the builds still work because go’s packaging is finally stable). But the difference is minimal — using docker/k8s is more about streamlining deployment/config/rollback (and the occasional co-packaged asset) than anything else.
IT industry is full of Matthews that need to be proven wrong for us to advance.
“You believe because you see me. Great blessings belong to the people who believe without seeing me!” (John 20:24-31 )
Bringing it into the IT context, there are the visionaries that believe something is possible no matter what, and then there are those that even with stuff running in front of them cannot move beyond "yes but...".
Ironically, in the 80's in what concerns home computers and game programming, both C and C++ also belonged to the "yes but..." group.