Amazon Kinesis Available For All Customers | Hacker News Reader

Amazon Kinesis Available For All Customers

aws.amazon.com

72 pointsseaghost19 comments

I wish Amazon hadn't given up even trying to do RESTful APIs. Nowadays they just shrug and bang it out RPC style (payload is JSON instead of XML, but the song remains the same).

e.g. How do you delete a stream?

Is it a DELETE request for, say, /streams/<stream-id> ?

No, it's a POST to "/" with an entity:

  {
      "StreamName": "string"
  }

Sigh.

Not all proxies support DELETE and PUT. But you're right, it should support both.

Amazon success is a strong indicator: RESTful APIs don't really matter. You can build a very successful business without focusing on the format of your APIs, as long as you build something valuable.

It helps if you're already one of the biggest players. A smaller company may be able to make a more attractive offering with REST.

I can't wait to rip out my "client => server => SQS => queue processor => S3 => redshift" flow and turn it into a "client => kinesis => redshift" flow.

Web scale analytics capturing just got way easier.

We just wrote something very similar to this using Kafka which replaced a very similar system you have. Do it. We would have used Kinesis if it was around.

This looks interesting at first glance, but to be honest, I don't fully get it from the description. Is it some kind of Storm/CEP/Graphite combination? Where does the actual "processing" happen? Do you have rent additional EC2 resources for your workers? It looks like a potentially powerful and interesting service, but the documentation could be a little more clear and maybe include an example of a DAG layout for a typical application.

As I understand it, this is very similar to Linkedin's Kafka.

To understand how you would use it, read some of the kafa docs: https://kafka.apache.org/documentation.html#introduction

From my reading of it (and my use of Kafka in the past, as an input to Storm), Kafka is a message queuing system, where Kinesis is a message processing system. That is, Kafka is more similar to SQS or RabbitMQ and Kinesis is more similar to Storm or S4.

Of course, I haven't used kinesis and have only glanced at the docs, so I could be wrong - it could very well be that Kinesis is the glue between data input and data processor rather than the data processor itself, in which case, yeah, it looks a lot like Kafka.

Does anyone have any experience using this technology? I'd assume it works best for post processing or pseudo real time data feeds( ie within a couple of seconds as opposed to sub 20 millisecond reactions).

Where do people usually store the processed data from this system?

I can see something like this being useful for things like post trade analytics or post day analyzing the efficiency of an given algo.

We use an inhouse CEP to do this sort of work currently but I'd rather rent than buy servers for this offline processing.

By the way, if anyone is in the Toronto area and interested in working for a hedge fund my contact info is in my profile

When I think of streaming my first thought is that this is a cloud service that processes audio and video in real-time. But none of the example use cases mention that is a possibility. I'd love the ability to stream and convert audio then forward converted files to S3 in a simple way.

Currently it involves running an EC2 instance which only converts after the audio is received in full. Then when it's done, transferring it to S3 is... slow.

you might need to look into http://aws.amazon.com/elastictranscoder/ if it can satisfy your use-case

The audio file must exist somewhere, it offers no live audio support. So the bottleneck would be at the beginning rather than the end.

Elastic Transcoder doesn't support this natively, we ran into this issue so ended up rolling our own solution as the commercial versions were incredibly expensive for a non-profit.

Take a look at the Zencoder (YC W10) API: http://zencoder.com/en/

It's since been acquired by Brightcove. [Disclaimer: I work there.]

amazon says real-time, but i'm 99% sure they mean (comparatively) low latency.

It sounds exciting but I don't fully get what this is good for. It sounds like a stream (of "records") on this would be the equivalent of a queue and messages in Amazon SQS? only Kinesis is lower latency and billing is both capacity-based and quantity-based rather than only quantity-based in SQS?

How many visitors you'll need to have to make some sense out of their behaviour?

Looks similar to my logstash -> elastic search -> kibana workflow.

Reply on news.ycombinator.com