145 karma · joined February 17, 2015
I did a quick search and couldn't find it. Does anybody remember it? I'm wondering if my memory or my search skills are failing or if it has simply disappeared.
At Intent HQ we are working in a close relationship with our clients to help them have a better understanding of their data so they are able to provide better services to their own customers. We are ~80 people from all over the world (we speak 15 different languages!) based in London, Barcelona and NYC. As we are small, we love sharing ideas and really like to work along the principles of valuing 'individuals and interactions', 'customer collaboration', 'responding to change' and 'working software'.
Tech stack: Scala, Typelevel stack (cats-effect, htt4s, fs2...), Cassandra, Elasticsearch, PostgreSQL, Kafka, Docker, Nomad, Terraform, Consul, Vault, AWS, TypeScript, React, Redux
We have several positions open: https://intenthq.teamtailor.com/
Salary ranges depend on location.
If you want more information feel free to drop me an email: albert (at) intenthq.com
So, I'd go for what others have already mentioned. Learn about the person, how they are, how they learn better, what are their interests.
Find something they like and enjoy, so they are motivated in learning.
Be ready and available to answer questions and adapt to their rhythm and needs.
Prepare materials and different options so they can chose their path as they go.
Facilitate them changing their mind, going back and forth, making their own mistakes.
In my case, for example, I learn by doing, and pair programing with somebody helps me a lot, but other people might prefer having a theoretical background first and will want to read a book before diving into coding.
EDIT: adding paragraphs for clarity
>> rows to randomly sample ... hash (using ... 32 bits) the column ... mod the result by the [constant] value
> This is not random. It deterministically selects the same very predictable fraction of rows.
Yep, you are right. We didn't intend the sampling function to be part of the anonymisation but just something we tend to use and we thought it would be useful to have it.
Its objective is to pick a portion of the input data. No more.
>> UK format postcode (eg. W1W 8BE) and just keeps the outcode (eg. W1W)
>> Given a date, just keep the year
> Partial postal codes and dates quantized to the year are still very revealing. Combined with other data (such as a hashed name), the partial postal code may allow a lot of people to be uniquely identified.
You are absolutely right. Depending on the use case and your data, having the outcode, the city or the year might be very revealing. In some other cases even having decades or centuries might be revealing.
We don't pretend that each function provided applies to all use cases. But in certain use cases partial postcodes or years can be good enough.
>> Hash (SHA1) the input
> Hashing does not provide anonymity.
We are very aware of that. That's why we offer the option to add a salt (that the user of the tool can make as long as possible and throw away after the anonymisation process).
>> range
> This is the only feature that could provide anonymity, if it is used correctly to group large numbers of individuals into the same bucket. This is probably more difficult that it first appears.
We usually work with sets of data that are tens of millions of users. Choosing the right ranges and, specially, analysing the data and making sure you anonymise the outliers (by choosing your bottom and top ranges carefully) it's crucial.
Again, this tool is a hammer. We expect a person that understands about wood and nails to analyse their problem and use it.
I agree with you. there are other options for configuration that are much better than json (yaml, toml).
Main reason for chosing json was simplicity. This was my first project in go and I didn't want to spend much time in it either. I found an example that was using json and I saw that I didn't need any external library to decode it. I thought that was good enough, at least for now.
Will probably look into using a library that supports yaml/toml for configuration in the future.
Besides needing the jvm installed, are there any differences between rambler and flyway command line [1] that make it a more suitable choice?
While doing the benchmarks, we could see that citus was always taking full advantage of all the cores in the cluster, while postgres parallelization was not.
Disclaimer: I'm not a db expert and I don't have any relationship with either citus or postgres.
It may be worth downloading it from github.
It's not the same as Google, but it's good enough in many cases.
Under my limited experience working with wikipedia and wikidata I've seen it's not the best option to a) store structured data and b) edit the pages (markdown is, imho, much better for that).
We are looking for Scala developers that will be:
- In the core of our business, being responsible for our most valued core IP
- Making sense of huge amounts of data
- Solving problems that don't have yet a solution
- Developing clean, robust and scalable code in Scala
And, most of all, we promise you won't ever get bored and will be having fun doing what we like the most, creating.
We offer flexibility and occasional remote work.
Contact me for more information: albert at intenthq dot com
Said that, I believe the process described in the blog post is not loading the whole Wikidata dump into memory and it would work the same to process Freebase or even larger data dumps with your laptop.
From the post: How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop.
[1] https://developers.google.com/freebase/data http://dumps.wikimedia.org/other/wikidata/
(Disclaimer: this is a referral link that will give us 5gb https://copy.com?r=b2yUAQ ;-))