b0ti

Couldn't agree more. Not to mention timestamps that JSON simply doesn't handle and is essential for event data. The raijin database has an clever approach to solving the schema rigidity problem: https://raijin.co

NXLog - Log Management| DevOps Engineer | Europe | REMOTE ONLY | https://nxlog.co

We would like to hear from you if you are interested in building fantastic products in the log management space and would like to be part of our growing team. We are a remote-only company and you will will be able to work under a flexible schedule. As a DevOps engineer you will be responsible for maintaining our infrastructure, occasionally engage in support with customers, troubleshoot issues related to the products, research and integrate NXLog with third party solutions and shiny new tech that emerges.

There are other open positions available. More info at https://nxlog.co/careers

I'm running a remote-only company and we moved to GitLab.com last summer from cloud hosted trac+git/svn combo (xp-dev). The reason we picked GitLab.com was because the stack is awesome and Trac is showing its age. We also wanted a solution that could be ran on premises if needed. We spent about a month migrating stuff over to GitLab from Trac. Once we were settled the reliability issues started to show. We were hoping that these would be quickly sorted out given the fact that the pace of the development with the UI and features was quite speedy.

A sales rep reached out and I told him we would be happy to pay if that's required to be able to use the cloud hosted version reliably but I got no response. Certainly we could host GitLab EE or CE on our own but this is what we wanted to avoid and leave it to those who know it best. xp-dev never ever had any downtime longer than 10 minutes that we actively used during the last 6 years. I'm still paying them so that I can search older projects as the response time is instant while gitlab takes more than 10 seconds to search.

Besides the slow response times and frequent 500 and timeout errors that we got accustomed to, gitlab.com displays the notorious "Deploy in progress" message every other day for over 20-30 minutes preventing us from working. I really hoped that 6-7 months would be enough time to sort these problems out but it only seems to be worsening and this incident kinda makes it more apparent that there are more serious architectural issues, i.e. the whole thing running on one single postgresql instance that can't be restored in one day.

We have one gitlab issue on gitlab.com to create automated backups of all our projects so that we could migrate to our own hosted instance (or perhaps github) but afair gitlab.com does not support exporting the issues. This currently locks us into gitlab.com.

On one hand I'm grateful to you guys because of the great service as we haven't paid a penny, on the other hand I feel that it was a big mistake picking gitlab.com since we could be paying GitHub and be productive instead of watching your twitter feed for a day waiting for the postresql database to be restored. If anyone can offer a paid hosted gitlab service that we could escape to, I'd be curious to hear about.

We are remote-only now so you can be located anywhere. The company itself is in the EU (Hungary). The careers page is a bit out-of-date, it will be updated soon.

We are also looking to hire a technical writer for NXLog. If you or anyone else is interested, feel free to reach out. https://nxlog.org/contact-us

The linked page already talks about jsonb in postgres. Feel free to read it! :)

Note that this is not PG, Raijin is a new implementation and is not built on Postgres. Otherwise the question is valid. Raijin does not store NULL values though. In the above example a specific field (column) was selected and that's why it appears as null. Perhaps the output would be more appropriate this way then (which is a modification for us to consider):

  {"foo":"bar"}
  {"foo":"bar2","intfield":42}

Note that when you select the full record with the star it does not return null values:

  select * from tbl;
  --
  {"_id":1,"eventtime":"2016-07-20 10:37:12","foo":"bar","whatever":"xx"}
  {"_id":2,"eventtime":"2016-07-20 10:38:22","foo":"bar2","intfield":42}

The undef/exists thing can get a bit confusing when you are mapping this into SQL.

Having to work with unstructured datatypes in Postgres and other RDBMS products was quite hard a few years back, now with hstore and JSONB in Postgres this has gotten much better but it still feels a bit unnatural. When we started the Raijin database project (http://raijindb.com) one of the goals was to make it possible to treat these extra attributes (that you would shove into JSONB with postgres) as normal "columns" so you can do this:

   CREATE TABLE tbl(eventtime datetime);
   INSERT INTO tbl {"eventtime":"2016-07-20 10:37:12","foo":"bar","whatever":"xx"};
   INSERT INTO tbl {"eventtime":"2016-07-20 10:38:22","foo":"bar2","intfield":42};
   SELECT foo, intfield FROM tbl;
   --
   {"foo":"bar","intfield":null}
   {"foo":"bar2","intfield":42}

It won't complain about the missing 'columns'.

Anyway, glad to see that Postgres is steadily making progress in this area.

There is disk based buffering in NXLog CE. You might want to check it out with respect to the other woes you have with rsyslog.

You might want to consider NXLog if you need to do transformation at the source. For us this was an explicit design goal. Moreover it is also lightweight and a lot of people use it in place of other fat and bulky solutions, quite popular with ELK users.

I had a similar "open source burnout" recently. Since you asked for it, there is a blog post I wrote which might be of interest: http://nxlog-ce.sourceforge.net/why-the-gpl-does-not-work

> Why would you release something to the public and put your name on it if it's lazily done?

No software is perfect. It has either no features in it or has bugs. The assholes will either demand to get the missing features added or the bugs fixed.

Not sure about logstash but a lot of its users use NXLog (http://nxlog.org) as a shipper since it has a much lower resource footprint as there is no java and ruby in there. (I'm affiliated with NXLog.)

If you open source it, pick a license that's more restrictive (e.g. AGPL). This leaves a chance for you to get lucky and sell it to a company who wants it to use it as closed source so that you can buy a few beers. Otherwise they will just take it and use it without a single thank you. Probably I'll get downvoted for this but IMO releasing work under the BSD, Apache or a copyleft license that you created on your free time is like playing charity for large corporations.

Paid support is already there. There is a "support" menu on the webpage, the issue tracker has a polite note saying that this is available. Some people have been contacted directly in email to see whether they'd be interested.

For example some users at a big Taiwanese laptop maker company were after windows2000(!) support. When they were politely offered support saying "please decide whether paid support would be of interest or not", the response was "ok, thank you for your information but when can you fix the bug?". I could give countless other similar examples.

Anyway, thanks for your suggestions. I'll heed the advice and will change the website to make this more clear for those who are only after the free labor.

It's nice to hear about such success stories. Unfortunately most OSS projects never gain such a user base to be able to make a living off of it. I'm currently evaluating changing the license of our GPL-ed software to use something more restrictive or even dropping OSS and make it freeware only without providing sources. Read these emails if you want to know why: http://www.mail-archive.com/nxlog-ce-users@lists.sourceforge... http://www.mail-archive.com/nxlog-ce-users@lists.sourceforge...

Let me know if you have some advice.