Hacker News Reader

Comment by mamcx | Hacker News Reader

Of curse, is unnecesary (ie: the lack of have a math solution is not make this task hard/impossible to do).

If a person know database/sql, then "math" is not necessary here. Or what exactly are you talking about?

Nope, there's plenty of math and stats involved in solving this problem correctly.

The amount of math you would apply here depends on your familiarity with the problem domain (this is where CS education comes into play), and your willingness and ability to work with higher level math and stats (this is where math proves necessary even in mundane software engineering tasks). For the record, even basic relational algebra can be seen as part of applied math, but the rabbit hole goes way deeper.

The best solutions for de-duplication of records without direct clues like unique IDs involve pretty cool applications of probability theory and machine learning, you'd find things like the Expectation-Maximization algorithm as prerequisites for understanding the current research. The foundational solution in this space is Fellegi-Sunter record linkage [1]. More advanced solutions include things like hierarchical Bayesian models [2]. If you dig deeper, especially if you are dealing with large datasets and basic techniques turn out to be too slow, you'd look into locality sensitive hashing and approximate nearest neighbors, which is a very mathematical subject, see for example [3]. In other words, math is encountered in even the most mundane software engineering problems, and you have to know where to look and what to look for.

[1] http://www.purdue.edu/discoverypark/vaccine/assets/pdfs/publ...

[2] http://arxiv.org/pdf/1207.4180.pdf

[3] http://www.cs.princeton.edu/courses/archive/spring05/cos598E...

I do this sort of task routinely. Can you point to a specific dataset where the regular DBAs skills falls apart? Because just see a lot of formulas without the context is part of the problem with math education (ie: show the solution and the claims it solve a contrived problem.. but where is the actual problem?)

The actual problem I described above (bulk de-duplication of records without any direct clues, such as unique ids, to decide whether two records are duplicates) falls under this category.

The first paper I provided is a survey of commonly used methods in this space, it's the exact opposite of a lot of formulas without the context.

> If a person know database/sql, then "math" is not necessary here

SQL? Like in relational databases? Like in relational algebra?

I think this demonstrates the point someone made upthread that people don't consider a lot of math to be math.

The irony here is that you mention SQL (the language of Relational Calculus) and in the same sentence claim that math is unnecessary.

but you see, my point is that the actual hard problem of executing a SQL query has been solved by those who designed the DBMS. our task boils down to writing SQL, which is way easier than designing the machinery that implements relational algebra.

EDIT: not to mention that you don't even have to understand relational algebra to write SQL. There are HORDES of programmers who don't know what it is yet they write SQL.

Relational Algebra is about executing the query. Relational Calculus is about formulating the query in the first place. Unless you copy-paste all your SQL code, you wind up using Relational Calculus. I would agree that most developers don't have a formal understanding of it (I sure don't) or even an awareness of it. They may not think of it as doing math. They may be thinking "how do I use a select with a join to get this particular set of data". But it's still math.

Yeah, I know from where come the relational databases. Still, claim that using SQL is using math is like say using a car is using math. Like using python is using assembler.. and at the same time is not.

This is kinda zend: If a developer don't think about math....

Reply on news.ycombinator.com