alexatkeplar

The event producer in this case is one of our Snowplow trackers (https://github.com/snowplow/snowplow-javascript-tracker) - and these tracking SDKs do indeed attach a UUID as a unique event ID at event creation time.

However, this event ID is not enough to identify and then dedupe all types of duplicate events. This blog post provides more information:

https://snowplowanalytics.com/blog/2015/08/19/dealing-with-d...

Big thanks to pragmacoders for putting this tutorial together! It's awesome seeing what you are doing with the Snowplow platform :-)

With Kinesis, you would just use the Kinesis Client Library (https://github.com/awslabs/amazon-kinesis-client-python) which would automatically handle committing the offsets to DynamoDB.

Home-rolling a checkpoint-free event pipeline is a rookie mistake; it's a pity they didn't come across our Snowplow project (Apache 2.0 event pipeline running on Kinesis, Kafka and NSQ, https://github.com/snowplow/snowplow/).

Great write-up! I tried something similar for modelling out the Snowplow release dependency chain (https://github.com/snowplow/) - I gave up and started looking at SaaS alternatives, but inspired by this I am going to try again.

For the record, I didn't find any good product management or project management tools (SaaS or desktop) which acknowledge that roadmapping any sufficiently complicated platform or product family is best expressed by a directed acyclic graph. Casual.pm had the right concept (https://casual.pm/) but the implementation was very primitive.

As far as I know, Snowplow is one of the largest active users of JSON Schema, and we maintain one of the larger repositories of JSON Schemas on the internet, https://github.com/snowplow/iglu-central

After something of a hiatus, we are re-starting active work on our schema registry technology (https://github.com/snowplow/iglu). We'd love to be involved in new design work on the JSON Schema spec. How do we get the old (or new!) band back together?

Thanks for the mention! We are working on our port of Snowplow to run cloud natively on GCP currently - you can see our RFC here:

https://discourse.snowplowanalytics.com/t/porting-snowplow-t...

We've been doing a lot of thinking about how to support GDPR at Snowplow (Kafka and Kinesis but plenty of other logs and stores) - for our first phase we're just going to support irreversible pseudonymization of tagged PII:

https://github.com/snowplow/snowplow/issues/3472

For later phases, yes user-specific encryption of PII or hashing-with-lookup table are the way to go...

Inspired by Valve, here's the $15 solution we use:

https://www.amazon.com/Mount-Adjustable-Universal-Computer-H...

Paul are you based in London? I've been thinking about dusting off my Unified Log London meetup (https://www.meetup.com/unified-log-london/) and it sounds like you would have a ton to talk about there. Email address is in my profile!

Apologies - the winternship will be for 6-8 weeks, not 8-12.

Snowplow Analytics | Open-Source Engineer | Remote, INTERN.

Snowplow Analytics is looking for 1-2 open source software interns this Winter (November through March), for a 8-12 week paid internship. Our winterns will work directly on and contribute to projects within the Snowplow open source stack (https://github.com/snowplow). A Snowplow intern loves coding, enjoys experimenting with new technologies and is happiest working "in the open" on community/team projects. Technologies we use at Snowplow include Scala, Rust, Go, Kinesis, Redshift, Spark, BigQuery, Beam, Akka and Kafka.

This is a paid internship; we will consider remote candidates who are up to UTC +/- 5 hours maximum. Interested? Please email intern@snowplowanalytics.com, and tell us about a piece of software you are proud to have written. (And don't be afraid to suggest specific projects/initiatives/features that you would like to work on in your internship.)

For background on two of our recent internship programs check out:

  http://snowplowanalytics.com/blog/2016/03/17/2015-2016-winternship-wrapup/
  http://snowplowanalytics.com/blog/2015/07/10/introducing-our-2015-summer-interns/

Until recently we used them at Snowplow for orchestrating data processing pipelines, per this blog post:

https://snowplowanalytics.com/blog/2015/10/13/orchestrating-...

We gradually swapped them out in favour of our own DAG-runner written in Rust, called Factotum:

https://github.com/snowplow/factotum

Snowplow is hiring in London, Berlin and remote. Front-end engineering, data engineering, systems, marketing and product management.

https://snowplowanalytics.com/company/careers/

Join our team and help us give companies control of their own data.

Snowplow Analytics | Front-End Engineer | REMOTE, UTC+0 to UTC+3

We are now looking for our first front-end engineer to join the Product Engineering team working on Snowplow Insights. Snowplow Insights represents the next phase in the evolution of Snowplow, from a consulting company through a services firm (the Snowplow Managed Service now has over 100 customers) to a full-fledged product business.

You'll be building our product front-end in React, written in TypeScript with Redux, and interacting with our growing collection of micro-services via GraphQL and REST. This is an opportunity to get in extremely early in Snowplow's product evolution, playing a major role in developing the Snowplow Insights user experience, product design and technical architecture.

For more information: https://snowplowanalytics.com/company/careers/front-end-engi...

Snowplow Analytics | Sysadmin | ONSITE London, UK

Snowplow is building out a dedicated Technical Operations team in 2017, and is looking for experienced systems administrators to join it. Initially this role is open only to candidates based in London or the South-East of England.

The sysadmins within our Technical Operations team will have four key responsibilities:

1. Handling deployments, upgrades and other maintenance of Snowplow-related infrastructure (load balancers, Redshift clusters, ASGs etc) for our Managed Service customers, across over 100 AWS accounts

2. Responding to customer issues and questions concerning Snowplow-related infrastructure, as escalated to you by our L1 Support team

3. Working with Snowplow SREs to design, deploy and operate Snowplow’s internal infrastructure, responsible for running the Snowplow Managed Service, the Snowplow website and other services

4. Being on call rotation to triage and resolve operational incidents relating to internal or client infrastructure

For more information: https://snowplowanalytics.com/company/careers/systems-admini...

Snowplow Analytics | Open-Source Engineer | Remote, INTERN.

Snowplow Analytics is looking for 1-2 open source software interns this Summer (May through August), for a 8-12 week paid internship. Our interns will work directly on and contribute to projects within the Snowplow open source stack (https://github.com/snowplow). A Snowplow intern loves coding, enjoys experimenting with new technologies and is happiest working "in the open" on community/team projects. Technologies we use at Snowplow include Scala, Rust, Go, Hadoop, Kinesis, Redshift, Spark, Akka and Kafka.

This is a paid internship; we will consider remote candidates who are up to UTC +/- 5 hours maximum. Interested? Please email intern@snowplowanalytics.com, and tell us about a piece of software you are proud to have written. (And don't be afraid to suggest specific projects/initiatives/features that you would like to work on in your internship.)

For background on two of our recent internship programs check out:

  http://snowplowanalytics.com/blog/2016/03/17/2015-2016-winternship-wrapup/
  http://snowplowanalytics.com/blog/2015/07/10/introducing-our-2015-summer-interns/

Snowplow Analytics | Sysadmin | ONSITE London, UK

Snowplow is building out a dedicated Technical Operations team in 2017, and is looking for experienced systems administrators to join it. Initially this role is open only to candidates based in London or the South-East of England.

The sysadmins within our Technical Operations team will have four key responsibilities:

1. Handling deployments, upgrades and other maintenance of Snowplow-related infrastructure (load balancers, Redshift clusters, ASGs etc) for our Managed Service customers, across over 100 AWS accounts

2. Responding to customer issues and questions concerning Snowplow-related infrastructure, as escalated to you by our L1 Support team

3. Working with Snowplow SREs to design, deploy and operate Snowplow’s internal infrastructure, responsible for running the Snowplow Managed Service, the Snowplow website and other services

4. Being on call rotation to triage and resolve operational incidents relating to internal or client infrastructure

For more information: https://snowplowanalytics.com/company/careers/systems-admini...

Snowplow Analytics | Open-Source Engineer | Remote, INTERN.

Snowplow Analytics is looking for 1-2 open source software interns this Summer (May through August), for a 8-12 week paid internship. Our interns will work directly on and contribute to projects within the Snowplow open source stack (https://github.com/snowplow). A Snowplow intern loves coding, enjoys experimenting with new technologies and is happiest working "in the open" on community/team projects. Technologies we use at Snowplow include Scala, Rust, Go, Hadoop, Kinesis, Redshift, Spark, Akka and Kafka.

This is a paid internship; we will consider remote candidates who are up to UTC +/- 5 hours maximum. Interested? Please email intern@snowplowanalytics.com, and tell us about a piece of software you are proud to have written. (And don't be afraid to suggest specific projects/initiatives/features that you would like to work on in your internship.)

For background on two of our recent internship programs check out:

  http://snowplowanalytics.com/blog/2016/03/17/2015-2016-winternship-wrapup/
  http://snowplowanalytics.com/blog/2015/07/10/introducing-our-2015-summer-interns/

Snowplow Analytics | Sysadmin | ONSITE London, UK

Snowplow is building out a dedicated Technical Operations team in 2017, and is looking for experienced systems administrators to join it. Initially this role is open only to candidates based in London or the South-East of England.

The sysadmins within our Technical Operations team will have four key responsibilities:

1. Handling deployments, upgrades and other maintenance of Snowplow-related infrastructure (load balancers, Redshift clusters, ASGs etc) for our Managed Service customers, across over 100 AWS accounts

2. Responding to customer issues and questions concerning Snowplow-related infrastructure, as escalated to you by our L1 Support team

3. Working with Snowplow SREs to design, deploy and operate Snowplow’s internal infrastructure, responsible for running the Snowplow Managed Service, the Snowplow website and other services

4. Being on call rotation to triage and resolve operational incidents relating to internal or client infrastructure

For more information: http://snowplowanalytics.com/about/jobs/systems-administrato...

Snowplow Analytics | Front-End Engineer | REMOTE UTC-3 to UTC+3

This role isn't on our website yet - but we are looking for our first React/Redux/TypeScript front-end engineer.

For all other jobs, please see: http://snowplowanalytics.com/about/jobs/

Snowplow Analytics | Sysadmin | ONSITE London, UK

Snowplow is building out a dedicated Technical Operations team in 2017, and is looking for experienced systems administrators to join it. Initially this role is open only to candidates based in London or the South-East of England.

The sysadmins within our Technical Operations team will have four key responsibilities:

1. Handling deployments, upgrades and other maintenance of Snowplow-related infrastructure (load balancers, Redshift clusters, ASGs etc) for our Managed Service customers, across over 100 AWS accounts

2. Responding to customer issues and questions concerning Snowplow-related infrastructure, as escalated to you by our L1 Support team

3. Working with Snowplow SREs to design, deploy and operate Snowplow’s internal infrastructure, responsible for running the Snowplow Managed Service, the Snowplow website and other services

4. Being on call rotation to triage and resolve operational incidents relating to internal or client infrastructure

For more information: http://snowplowanalytics.com/about/jobs/systems-administrato...

Snowplow Analytics | Support Engineer | REMOTE UTC+8 to UTC+10

Having grown our team to two support engineers with a broad timezone reach (Alberta, Canada to Berlin, Germany), we are now looking to move to a full “follow the sun” model, and hire our third support engineer in the UTC+8 to UTC+10 timezone range.

This is a support engineering role - not a support agent role. We are looking for candidates who can learn, troubleshoot and explain the many complex technical systems that make up the Snowplow offering. You will be supporting the Snowplow Managed Service, under which we orchestrate and monitor the Snowplow event pipeline for over 100 customers.

The support that we provide to our customers is a core part of the Managed Service offering, and we strive to provide the best technical support of any analytics vendor.

You'll find more information here: http://snowplowanalytics.com/about/jobs/support-engineer/

I agree there is some fuzziness. Long after we wrote this I read:

https://www.w3.org/2001/tag/doc/versioning#iddiv371153984

which has a very succinct explanation of forwards and backwards compatibility as it relates to producers and consumers.

It's high time we did a second draft of SchemaVer which explains it in terms of forwards/backwards compatibility; the actual behavior of it (when to bump etc) would barely change.

Snowplow Analytics (http://snowplowanalytics.com) | Support Engineer | REMOTE UTC+8 to UTC+10

Having grown our team to two support engineers with a broad timezone reach (Alberta, Canada to Berlin, Germany), we are now looking to move to a full “follow the sun” model, and hire our third support engineer in the UTC+8 to UTC+10 timezone range.

This is a support engineering role - not a support agent role. We are looking for candidates who can learn, troubleshoot and explain the many complex technical systems that make up the Snowplow offering. You will be supporting the Snowplow Managed Service, under which we orchestrate and monitor the Snowplow event pipeline for over 100 customers.

The support that we provide to our customers is a core part of the Managed Service offering, and we strive to provide the best technical support of any analytics vendor.

You'll find more information here: http://snowplowanalytics.com/about/jobs/support-engineer/

For the rest of our open positions see: http://snowplowanalytics.com/about/jobs/ We are additionally hiring for a Sysadmin in London (job role not yet posted).

It's quite old now, but this is still one of the best tutorials on dealing with failure in FP (here, Scalaz):

https://gist.github.com/oxbowlakes/970717

Hey clumysmurf - author of the book here; yes indeed, the book is heavily influenced by Jay Kreps' LinkedIn paper.

There are other reasons for duplicates in event streams - not just the dupes introduced by at-least once processing in Kinesis or Kafka workers. We've done a lot of thinking about this (all open-source) at Snowplow, this is a good starting point:

http://snowplowanalytics.com/blog/2015/08/19/dealing-with-du...

Our last release started to tackle dupes caused by bots, spiders and dodgy UUID algos:

http://snowplowanalytics.com/blog/2016/12/20/snowplow-r86-pe...

Snowplow Analytics (http://snowplowanalytics.com) | Support Engineer | REMOTE UTC+8 to UTC+10

Having grown our team to two support engineers with a broad timezone reach (Alberta, Canada to Berlin, Germany), we are now looking to move to a full “follow the sun” model, and hire our third support engineer in the UTC+8 to UTC+10 timezone range.

This is a support engineering role - not a support agent role. We are looking for candidates who can learn, troubleshoot and explain the many complex technical systems that make up the Snowplow offering. You will be supporting the Snowplow Managed Service, under which we orchestrate and monitor the Snowplow event pipeline for over 100 customers.

The support that we provide to our customers is a core part of the Managed Service offering, and we strive to provide the best technical support of any analytics vendor.

You'll find more information here: http://snowplowanalytics.com/about/jobs/support-engineer/

For the rest of our open positions see: http://snowplowanalytics.com/about/jobs/

I spent 4 months working from Cyprus in 2016 (my wife is Cypriot). There's very little tech there (apart from Wargaming.net), and minimal public transport, but pretty much everything else is great (climate, food, infrastructure etc) and extremely ex-pat friendly. Plus the tech scene in Israel is 40 minutes away by plane.

I don't get what's so difficult to do on-prem. Your target market on-prem all use Kafka as their event bus and either Mesos or Kubernetes as their execution fabric - what's so tricky about fitting into that?

*

1 point by alexatkeplar 31 days ago | parent [-] | on: Ask HN: Who is hiring? (September 2016)

Snowplow Analytics (http://snowplowanalytics.com/) | London, Berlin, Remote | Full-time, Intern

Snowplow (https://github.com/snowplow/snowplow) is the leading open source event data pipeline, used by some of the most data-sophisticated organizations in the world. Headquartered in London UK, Snowplow is looking to fill a variety of positions in London, Berlin or working remote.

Positions:

Technical product manager - http://snowplowanalytics.com/about/jobs/technical-product-ma....

Junior data analyst - http://snowplowanalytics.com/about/jobs/junior-data-analyst/

Remote, INTERN.

Snowplow Analytics is looking for 1-2 open source software interns this Winter (sometime November through January), for a 6-8 week paid internship. Our interns will work directly on and contribute to projects within the Snowplow open source stack (https://github.com/snowplow). A Snowplow intern loves coding, enjoys experimenting with new technologies and is happiest working "in the open" on community/team projects. Technologies we use at Snowplow include Scala, Rust, JRuby, Go, Hadoop, Kinesis, Redshift, Spark, Akka and Kafka.

This is a paid internship; we will consider remote candidates who are up to UTC +/- 5 hours maximum. Interested? Please email intern@snowplowanalytics.com, and tell us about a piece of software you are proud to have written. (And don't be afraid to suggest specific projects/initiatives/features that you would like to work on in your internship.)

For background on our last 2 internship programs check out:

  http://snowplowanalytics.com/blog/2016/03/17/2015-2016-winternship-wrapup/
  http://snowplowanalytics.com/blog/2015/07/10/introducing-our-2015-summer-interns/

Exactly this - here's some simple windowing/counting in JavaScript with Kinesis, Lambda and DynamoDB, no JVM in sight :-) https://github.com/snowplow/aws-lambda-nodejs-example-projec...

I came here to say the same thing. The article has things the wrong way round: rather than moving the computation on individual webhook event streams into the SaaS provider of that webhook, I want to consolidate all the processing of all my disparate webhooks into my company's own unified event log.

Shifting the computation to the SaaS provider is wrong for all the reasons you mention, and others too - for example, if I want to sink my webhooks into a database, I'm not going to want to share those database credentials with the SaaS provider.

At Snowplow we support ingest of various webhooks (yes, standardization is a total pain), and then you can write your own "webtasks" on them in whatever tech you like (Lambda/Spark/Hadoop/SQL):

https://github.com/snowplow/snowplow/wiki/Setting-up-a-Webho...

Can you share your interests / capabilities upfront? Definitely have some side projects gathering dust but they need to go to a good home...

Snowplow Analytics (http://snowplowanalytics.com/) | London, Berlin, Remote | Full-time, Intern

Snowplow (https://github.com/snowplow/snowplow) is the leading open source event data pipeline, used by some of the most data-sophisticated organizations in the world. Headquartered in London UK, Snowplow is looking to fill a variety of positions in London, Berlin or working remote.

Positions:

Technical product manager - http://snowplowanalytics.com/about/jobs/technical-product-ma...

Data engineer - http://snowplowanalytics.com/about/jobs/data-engineer/

(junior) Data analyst - http://snowplowanalytics.com/about/jobs/junior-data-analyst/

Remote, INTERN.

Snowplow Analytics is looking for 1-2 open source software interns this Winter (sometime November through January), for a 6-8 week paid internship. Our interns will work directly on and contribute to projects within the Snowplow open source stack (https://github.com/snowplow). A Snowplow intern loves coding, enjoys experimenting with new technologies and is happiest working "in the open" on community/team projects. Technologies we use at Snowplow include Scala, Rust, JRuby, Go, Hadoop, Kinesis, Redshift, Spark, Akka and Kafka.

This is a paid internship; we will consider remote candidates who are up to UTC +/- 5 hours maximum. Interested? Please email intern@snowplowanalytics.com, and tell us about a piece of software you are proud to have written. (And don't be afraid to suggest specific projects/initiatives/features that you would like to work on in your internship.)

For background on our last 2 internship programs check out:

  http://snowplowanalytics.com/blog/2016/03/17/2015-2016-winternship-wrapup/
  http://snowplowanalytics.com/blog/2015/07/10/introducing-our-2015-summer-interns/