jaltekruse

If you want to take part in a community effort to fix this problem, consider contributing the to Apache Calcite project. It has relational algebra that spans across many different databases and dialects, including NoSQL databases like MongoDB and Elasticsearch. I haven't been very involved with the project directly, but I've worked on several projects that heavily depended on it, and have written extensions for those projects.

A standard way to at least represent common data types, dates/times, integers/longs and fixed precision decimal would be nice. I know that the JSON format doesn't specify any limit or precision limitations on the size of numeric values, but almost all libraries assume they should be read as doubles. It is a real pain for people doing analysis on semi-structured JSON after apps have been deployed for a while and the data model has evolved over time.

Interesting combination of features, MPP query engine, but unlike the rest of the projects around Hadoop they can push down queries where data sits instead of requiring a bulk copy to HDFS first. This is a real pain point for modern analytical workloads, analysts want the experience they knew when organizations could run on a single relational database, but it just doesn't match how applications need to be made these days. Pretty impressive list of connectors for a first release, SQL databases, Hbase, Hive, Mongo, Elasticsearch. Definitely going to give it a spin today.

You might want to check out these two projects:

https://github.com/learningequality/ka-lite

https://github.com/learningequality/kolibri

I'm not associated with either, I just heard about them recently and was similarly excited by the potential.

@linux-modder would be great to exchange ideas. I see a public key on your profile but no e-mail. Feel free to send me a message at the address listed on my project homepage. developers AT opennotebookmath DOT com

Open Notebook is an application for helping math teachers create content for their classes. It is designed to be extremely easy for non-technical users to pick up in just a few minutes. The application allows creating documents for print, or in the case of classrooms with students that have computers accessible at home, the application doubles as an interactive environment for students to complete their work for digital submission.

http://opennotebookmath.com

Thanks! I just pushed a commit to fix it. I actually destabilized it a little when I refactored to start writing unit tests. I won't be publishing a new binary right away, but it should be soon. Also working on a cert to avoid the unknown developer warning.

Video Demo: https://www.youtube.com/watch?v=EnBE76kY2cw

Thank you for the kinds words, I did try to put a lot of thought into the feature set to make it useful.

I have seen a number of tools that are similar, one way I wanted to set this project apart was with a simple interface that even non-technical teachers could use. I decided early on I wanted the application to be latex-free. Creating the equation renderer from scratch was a bit of overkill (the engineer mindset "oh it'll be easier to write myself"). This is something I hope fix and instead plug in a more complete and better tested alternative like JLatexMath, with actual editing capabilities. Pulling the editor out of this application looks promising http://mirai.sourceforge.net/.

If you look at Khan-Exercises they have actually moved away from automatic generation (see their Perseus project), because it was too hard to write the kinds of problems they wanted. Also internationalization when the content of the questions was dynamic caused them more hassle than it was worth. I am considering de-emphasizing content creation and focusing on the student solution interface, as I believe it is the more novel feature of the system. But for now I wanted to get some teachers using it and find out what they found useful.

Thanks for trying it out! We are working on integrating the software with Moodle and other course management systems so that I don't have to force schools to adopt a new service grade the assignments.

Hello all, as it's a show HN post it's kind of implied, but I am the author of the project. Please feel free to leave feedback or ask questions here. Thanks so much for checking out the project, be sure to share it on your social networks with your friends with an interest in education!

I only took a brief look over the code to try to find what documentation is actually there, but I managed to find this

https://github.com/redox-os/redox/blob/master/kernel/fs/url....

On your question of how typical a lack of system-internal documentation is in systems projects, the answer from my experience is unfortunately more common than it should be. There are a bunch of usage and design patterns that cannot be expressed regardless of the language you are using. I know very little about Rust, but I work mostly in Java, which is safer than many other languages (less flexible than interpreted or dynamic languages, less dangerous than C/C++). Even java, which has interfaces and strict typing can save you some of the more mundane unit tests and input validation code, has a bunch of design patterns on top of the core language. Projects could serve themselves well to document their code well to allow new contributors to look at any of the code and reasonably modify it to fix a bug or add a feature.

Exactly this, the most basic example is an operation on a single column. If the data for a single column is mixed in with other data in a row-oriented layout, you are going to have to bring a new chunk of memory into the CPU cache sooner to process a given amount of data. If all of the data is packed together tightly you can read many more values out of the cache before you need to go get more from main memory.

Parquet is designed specifically to store large amounts of data efficiently on disk. As such it defaults to compressing and encoding data to save space. Arrow is designed for immediate consumption without any materialization into a different in-memory data structure. It is already in a format well suited to be used for sending over the wire or reading directly from an API.

I don't know as much about the internals of Avro, but I know it is a bit different from Parquet, in that it can be used to serialize and deserialize smaller amounts of data. It is used to store large datasets in files, although it will in most cases be less space efficient than Parquet. It has also been used as a way of embedding complex structures into other systems (similarly to how JSON can be embedded in a database), or for serializing individual structures between systems. The binary representation of Avro needs to be read into a system-specific format like a C/C++ struct/object, Java object, etc. for consumption.

In contrast, Arrow is designed to represent a list of objects/records efficiently. It is designed to allow for a chunk of memory to handed to a lightweight language-specific container that can immediately reference into the memory to grab a specific value, without reading each of the records into it's own individual object or structure.

We are just using the site as set up by the Apache Infrastructure team. I'll file a ticket to see if they can add the clone URL there.

Here is a page with the clone URLs: https://git.apache.org/

To be fair this isn't the best UI for git. We were waiting for the mirror to Github to be set up, so we decided to just include the link to the Apache git repo UI to be safe the link was live when the press announcements went out.

The Github mirror is now live, you can see it here: https://github.com/apache/arrow

Disclosure I am a committer on Apache Drill and Apache Arrow.

This isn't actually true. The java implementation has been complete and used in Apache Drill, a distributed SQL engine, for the past few years. While we anticipate a few small changes to make sure the standard works well across new systems, this is by no means an announcement without tested code.

https://git-wip-us.apache.org/repos/asf?p=arrow.git;a=commit...

Drill uses standards compliant SQL for flat data.

To work with all of the non-relational stores Drill supports complex types like maps and lists. The syntax for working with this complex data is the same javascript, i.e. a.b.c[3], so this is a compliant extension of SQL, but is non-standard.

Disclosure, I am a Drill committer.

If you are looking for an open source technology to meet this need, Apache Drill is a distributed SQL engine that can talk to multiple datasources and make them available in a single namespace. Joins across sources are supported and can actually perform well when you use enough nodes for your data volume.

Because Drill exposes ODBC and JDBC interfaces, it can very easily be used with BI tools to expose all kinds of data to analysts.

The architecture is fully pluggable so anyone can write a connector to a new datastore and just include it on the classpath when starting the cluster. In the current 1.2 release Drill can connect to any database that exposes JDBC, like MySQL, Postgres, Oracle, as well as a number of non-relational datastores like HDFS, MongoDB and HBase.

https://drill.apache.org/

This is a reasonable concern, and as the other response mentions, it has happened, even to well-respected projects. I will however say that there are some ways to combat this if you are in a situation where security is an absolute must. While it removes some of the performance benefits, a simple defense is to include a call to clear the state of the object before letting it out of the pool, or upon returning it to the pool.

Disclosure, I am a committer on the Apache Drill project.

Traditionally Hive has been focused on batch processing, it started as a SQL query engine built on top of map-reduce. Unfortunately many operations in the SQL model are not well suited for map-reduce. This is a problem that is being addressed by the Hive team with their vectorization/tez work, which is a completely different execution engine from map reduce. Around the time that this work began a number of new projects were trying to solve similar problems, making queries more interactive, providing more flexibility and greater coverage of the SQL language (Hive has supported a healthy subset of the SQL language, but not full support). It is true that this new Hive engine is solving some similar problems as Drill, as is the case for Impala, Facebook's Presto engine and a few other projects.

Drill is focused on providing standard SQL support, instead of the SQL-like HiveQL language. This allows it to be used to hook up to the analytical tools and ecosystem that has been developing around the SQL language for the past few decades.

Drill also has no requirement to define schema ahead of time when querying. This is best illustrated by the JSON read support, you can write a query against a bare file, and Drill will discover the schema as it reads. This is the case for simple files stored on a local or distributed file system or even for document stores like MongoDB.

While there will certainly be cases where schema will need to be known ahead of time, many users of Hadoop and semi-structured datastores are having a hard time with the overhead of trying to fit a wide variety of datasets into a traditional DB modnel, which is what the Hive metastore provides. In cases where the number of schemas in your data is manageable, this can give consumers of the data useful meta-data and structutre. For this reason, Drill has support for reading from Hive tables for these types of use cases.

However while this model has its place, our discussions with current Hadoop users has validated the design of Drill. Analysts want to be able to gain insights from data in Hadoop, and there are cases where the overhead of managing this data like a traditional DB is too resource intensive. Drill enables these types of users to explore raw semi-structured data as it enters the Hadoop cluster, instead of waiting for IT driven ETL.

I think that you are pointing at the wrong file, it looks like the library is actually around 4k lines. The point may still be valid, but it was not a trivial codebase to fork and they seem to have some architectural goals that they didn't think any of the current libraries were well suited to work with.

https://github.com/Raynos/mercury/blob/master/dist/mercury.j...

You are correct that schema-less systems can still require maintenance. However a schema-less system like Drill allows users of the data like business analysts to write interactive queries as soon as they bring new files into HDFS. Surely if you were using Drill to query data programmatically, you would want to make sure only the technical staff could modify the data during the dev process, and likely use an automated process to incorporate data as it comes in. However, this is no different than managing appropriate permissions on a traditional database.

While I don't jump for opportunities to fiddle with a word processor to make a document like a resume look good, I am curious if this is partially driven by an effort to populate databases with information. While I have been unimpressed by efforts to do this by scanning a doc or pdf file, I am afraid of the possible implications for encouraging poor recruiting techniques. Having a searchable database can be nice, but there are so many other layers to finding good talent.

The person tallying keeps a count in their head.