Hacker News Reader

Comment by gerhardi | Hacker News Reader

Happy resident of a MongoDB environment here! In one of my projects, we are collecting IOT "clickstream" with slightly varying contents, around 20 million inserts per day. These are then transformed into "sessions" that are inserted in another collection. Everything works as it should.

Reporting is based on PowerBI and aggregations are scheduled through cron jobs to refresh the data sources that PowerBI uses. Realtime metrics are not needed, but could easily be implemented through the "session" generator by calling PowerBI Streaming API in parallel when writing a document.

View on HN

I think what you're describing here is something that is often overlooked. You are treating the same data in 2 ways - events and entities and there's an important distinction.

The events come in and you store them. They are immutable descriptions with a timestamp which provides sequence. There is no relation between any 2 events.

Then you keep another collection of entities which you call "sessions". These comb through the event data and combine them into sessions which can answer different questions.

The event data is the authoritative source and because there are timestamps they are sequenced. This way you can generate session from them and if anything were to happen to make the session collection invalid you can "rollback" to a certain point in time and recalculate the sessions from there.

In the NoSQL world you often want to begin with "events". And then form "entities" from the sequence of events that are useful to you. Because you've stored every action you can always go back and recalculate and create new entities you hadn't yet discovered.

Yes, this is exactly our use case. Every now and then a need for new attributes or metrics appear that are to be calculated or otherwise logically determined for the "sessions" generated. Sometimes a new device model / software version starts providing new attributes with the transmitted event and we can just add metrics and aggregations based on those attributes into the session generator if those attributes are present on an event.

Reply on news.ycombinator.com