pbh

Personally, I would run:

  split -d -C 1G FILE split.FILE # separate into 1G files on line boundaries
  # then, on separate cores
  sort -u split.FILE.00 -o sort.FILE.00
  sort -u split.FILE.01 -o sort.FILE.01
  ...
  sort -u split.FILE.N -o sort.FILE.N
  # then, on one core
  sort -m -u sort.FILE.*

You should be aware that you may not get what you expect from sort unless you set LC_ALL="C". You should also pass "-S" to sort to set the main memory size, probably to a value like "1G". Of course, at some point when you're doing a lot of distributed processing you'll just want to use Hadoop, as other posters have noted.

I don't disagree, but it seems like you're overstating your point by calling OOP "optional" in Scala (and possibly F#).

Scala is explicitly marketed as a hybrid OOP/functional language. At least from "Programming in Scala," it seems as though writing idiomatic Scala really requires both paradigms. I see Scala as adding functional programming rather than de-emphasizing OOP, the former being a common thread in new (multi-paradigm) languages.

I 100% agree, but think there is no chance of this happening. The incentives simply do not line up, and some sort of all-or-nothing transition is needed.

It's not (exactly) my community, but SIGMOD has been trying to do this since 2008 with the "repeatability/workability committee." See this:

http://www.sigmod.org/sigmod-pods-conferences/mirrors/sigmod...

There's also an interesting FAQ about the repeatability requirements specifically here:

http://www.sigmod.org/sigmod-pods-conferences/mirrors/sigmod...

... as well as various follow-ups from that work, if you search on Google Scholar.

The issues as I understand them are:

1. Research code is often of poor quality, usually thousands of lines of unchecked output by a single graduate student. There are probably bugs, some of which may change the results. In the absence of code, results are assumed to be correct. It takes substantially longer to write good code, and hurts researcher "output" to do so. As a result, good quality code is actively discouraged.

2. Releasing code and/or data usually makes it substantially easier for others to duplicate or catch up to your research program, which is seen as a disadvantage if you are still in an area. It may lead to more citations, but that probably isn't enough for the effort. Other researchers also have an incentive to try to find bugs in the code/data, which they may overstate as they try to get their own work accepted.

3. The code and/or data itself my be copyrighted or have unclear distribution terms, for example, if you are doing experiments on a web crawl.

4. Actual production quality code that does something useful can be used as the basis of a startup or other venture, especially if the researcher is the only one who has and understands it. Furthermore, research groups can make money licensing their code to outside companies if they do not release it openly.

5. Many (most?) industrial papers involve code or data that cannot be released. Ultimately, in highly competitive conferences, it is hard to balance "unverifiable" papers written by industry with academia papers. A blanket ban on papers without code or data would remove a huge number of industry contributions, but an optional requirement for code or data mostly continues the status quo. Many of the most interesting recent papers (e.g., MapReduce) might not have been published with a code/data requirement.

I had the same problem (and, really, 5 years?), but "solved" it at the time with ISO8601 strings. They sort, they're standard, they support microseconds, they're fairly common in web services. Slower than a reasonable native type though, obviously.

My ill informed opinion: the WSGI specification (i.e., PEP 333) together with a decade or so of cross-pollination seems to have made most Python web frameworks more similar than different. You can opt for more out of the box functionality at the expense of less flexibility, but not much else. (The exceptions seem to be where the code is really old and/or asynchronous.) Django seems to be the BDFL blessed option.

Example: Everyone seems to think that Google copied "their" Python web framework in creating the "webapp" framework for App Engine. webpy, Django, ...

I completely agree. My intent was to suggest that one should learn to the standard first rather than any particular implementation, not that one should read any of the actual standards documents directly. (Yikes!) By analogy, if you want to learn C, read K&R to learn something approximating C89, rather than picking up a book on how to code to the specific dialect of C understood by GCC 4.5.

For what it is worth, I learned from Database Systems: The Complete Book.

http://www.amazon.com/Database-Systems-Complete-Book-2nd/dp/...

DS:TCB is pretty explicit about which of the SQL it teaches is part of which standard. That said, I suspect that any general database book should do a reasonable job.

I want to be careful in how I phrase this, because I feel like this was a good effort, but I disagree with many parts of this description.

First, databases are not "just" about persistence, and I don't mean that in a pedantic way. The point of a database management system is to manage your data over time and allow you to make use of it. This includes things like stating constraints to ensure that the data in the database is always correct and providing methods for querying your data at varying levels of complexity.

Second, there are a number of in-memory databases. Oracle sells one. SQLite is often used as one. The fact that they are in-memory does not make them not databases.

Third, relational versus non-relational is not really an argument about "how the data is stored on disk" or about being "standards compliant." A database traditionally consists primarily of three things: a data model, a query language, and an implementation. A data model, like "everything is a table with column headers and rows" can be written to disk in many different ways and queried in many different ways. (It's an abstraction.) Indexing structures and data formats may be mostly the same even when the data model is different. The (current) "relational vs non-relational" debate, such as it is, is usually about whether it makes sense to change the data model and query language in order to ensure that the implementation is scalable on "cloud-like" platforms with specific data access needs. (That said, there are many different types of non-relational models and query languages designed for different purposes.)

Fourth, there isn't one SQL. There are in fact a number of standards, SQL-86, SQL-89, SQL-92, and so on. It makes sense to learn the standard first (at least up to 92 or so), rather than learning particular implementation specifics. Most databases implement SQL-92 or above, though there is somewhat wide variance in additional features like indexing structures and functions and in data modeling languages.

I asked a very similar question to this on MetaFilter a few weeks ago. To summarize, I think you're actually conflating three things: design, graphic design, and graphic design for the web.

As far as I can tell as a non-designer, there seem to be (essentially) two phases to graphic design for the web: wireframes and final design. The first phase is sort of a high volatility phase where one tries out many different approaches, mocking them up with extremely basic wireframes (in something like Balsamiq or even on paper for the best designers), figuring out what the client really wants (they don't know) and figuring out what you want. The second phase is making something nice looking based on the wireframes and other details from the first phase.

These two phases are completely different. They can, and often are, done by completely different people. To be good at either, you need to have spent years developing a good sense of design. However, other than that the skills seem quite different. For the first phase, one might need layout and communication skills. For the second, one might need to know about typography and color theory, maybe be able to draw proficiently, and so on. Often, the second phase includes turning the design into HTML and CSS, but this is rarely a good use of the designer's time, and can be delegated to specialty providers like psd2html.

Without more knowledge about what your background is, and why you want to study graphic design for the web, I think it is difficult for anyone to suggest what you need to study first. However, given how little of the above was really specific to web design, I think you might be better served by a more general design program, which might also be easier to find.

Obviously, everyone loves a good challenge, but is there any evidence that the find-ee wants to be found?

They seemed interested in the thread as to how they might be found, but don't seem to have given any permission for a site-wide (wo-)manhunt. (This might have happened out of band, though.) I guess they do say that they were just using a one time account for rhetorical emphasis.

Further, if this were to really be a contest, it seems like there should be some sort of rules, such that the result isn't determined just by exhaustion of currently in use usernames by guessers.

The Real Player installer, over the course of maybe five to ten years, was so pushy about installing extra, unwanted software and sending private data that it garnered a reputation that caused people to be really careful when installing it, if they installed it at all. It is not really a perfect example for people actually opting-out, because I think one of the many criticisms was that it was often either not possible or extremely difficult to determine how to opt-out of its features (sending titles of files being played, annoying message center and ad popups, packaged additional software). That said, check out the Wikipedia page for further details.

I do not have any data regarding opt-in, but my impression was that as a rule, no one opts-in, and no one opts-out of basically anything (excepting the notorious cases, e.g., Real Player). If it worked, and somehow gave an at least somewhat unbiased sampling of the data, opt-in would obviously be best.

I am completely unsatisfied with the submit-and-run model. Feature engineering seems to rely on knowing your data really intimately, and that does not seem possible in a submit-and-run model.

I think the parent is referring to an element of we-told-you-so in this letter. I think around the "instead, you brushed off our claims" part of the letter. It sounds a bit confrontational, though perhaps it's too late to rewrite.

It seems to me that any data at all will necessarily reduce the entropy of the probability distribution of members' preferences, likes, dislikes, habits, and so on (i.e., their privacy). The authors seem to brush off the "greater good" argument here, but I don't understand how any large scale data release can happen without at least some reference to such an argument given that context. Given that, the authors seem to be making a fairly strong claim here: that no large scale "anonymised" data release should ever happen. Is that helping anyone in the context of movie viewing? And is it hurting anyone other than academic researchers, given that companies share more sensitive data anyway?

Related: "Carnival Booth: An Algorithm for Defeating the Computer-Assisted Passenger Screening System," Samidh Chakrabarti, Aaron Strauss.

tl;dr Anything other than random is arguably worse.

That makes sense. I think the key will be the community norms (and social features to support those norms) that you build, but you seem aware of that. Looking forward to seeing how things evolve!

I have two top level questions for Kyle if he's still around (even though this isn't an Ask HN or Tell HN or anything).

First, what is the problem you are trying to solve? Second, what do you think are/will be the key components of your approach?

At least to me, the problem I have is that I'm a developer who isn't any good at design. I'd love to be able to make small (and big) projects that look professional. Unfortunately, while I know lots of good developers, I don't know any good designers.

Furthermore, working on a web site with someone seems to involve a lot of trust: there's a lot of asymmetric information involved. How does the web designer know I'm really doing anything? How do I know that the web designer is really doing anything? Doing things iteratively can help, but it's obviously not possible to do everything iteratively.

I could also try to learn some design skills, but ultimately I think that I'm unlikely to be able to match a decade of specialisation in my free time.

It seems like most attempts at solving this problem amount to either portfolio sites or dating/outsourcing style sites. But these approaches don't seem to solve the trust problem.

Forrst seems to be trying to make a forum/community for designers, developers, and designers/developers, but I wonder if designers really want to learn about Common Lisp web frameworks, or whether developers are really interested in design problems. I also wonder whether trading links, code snippets, and images can produce the same kind of trust as that produced by working side-by-side on an actual project. That said, this may not be the goal.

It seems like you are looking for a functional answer to this question, and I just signed up to take a look.

It looks like twitter + images/screenshots, more or less. You can follow people, users can post text + maybe a screenshot, and there are comment threads about the posts/screenshots. I think what has the potential to be distinctive is the community that develops, rather than any particularly notable features. (That said, since on the web most features should arguably be community driven, this is probably just the core set of initial features.)

I think there is a lot of demand for a site that really facilitates developers meeting designers, but I'm not sure what format, and what interactions, make the most sense. I've been looking for a solution to this problem for a few years, and I haven't found anything really satisfying.

After some thought, this sort of seems like it's missing the point.

What I'm curious about is what the current state of cookie and user click stream sharing among analytics/advertising companies is. What proportion of my browsing history is known to any given company, and what proportion given collusion? Are there any studies on this?

A more direct way of saying it is that academics at research universities are mostly motivated to publish frequently in reputable places by grants and tenure, and nothing has forced either the way grants or tenure are given out to change from what they were pre-Internet.

(That said, innovations that allow people to write more papers or get cited more have been taken up very rapidly.)

This is a major point in legal philosophy (and presumably elsewhere).

I don't think I agree with you (for example) that an adversarial system (http://en.wikipedia.org/wiki/Adversarial_system) is inherently worse than an inquisitorial one (http://en.wikipedia.org/wiki/Inquisitorial_system).

This article seems to me to conflate two things: how to have an exploratory discussion between two opposing viewpoints and how to win an argument.

For example, in an exploratory discussion, using your opponent's terms can lead to quicker definition of those terms, and less talking past one another. On the other hand, when trying to win an argument, allowing your opponent to frame the debate (either by using their terms or addressing/refuting their central points) can be the worst possible strategy!

I suspect that exploratory discussions where both parties are genuinely disinterested are rare in most contexts.

I think that the example you give is actually a very good example of a set of context switches that people regularly deal with poorly, even though they are, as you note, very common.

How many server side web developers do you know who can output standards compliant HTML/XHTML that also works on IE6? (In fact, I suppose this is one major reason why templating is so common---so you can separate the web development experts from the web design experts.) How many web developers will have a database enforce their integrity constraints rather than having the logic in their program? How many know how to write PL/SQL? How many compilers are there now from some source language down to JavaScript in order to reduce the number of languages that a developer needs to know and control interactions between?

The danger of these sorts of context switches seems to be difficult project coordination, code redundancy, performance bugs, and logic errors.

(On a side note, it's not that I think that developers are stupid, it's that writing code well in a given language seems to require not just familiarity with that often idiosyncratic language, but it also requires continued knowledge of standard idioms, libraries, frameworks, and sources of answers to common problems.)

Screencasts seem to me like they are the right tool for explaining standard paths through complex user interfaces. This is especially true of poorly documented and/or very open ended interfaces (e.g., in my case, Emacs or Photoshop). (For example, I only realised that I could open a remote MySQL prompt through SQL-mode, using TRAMP, after watching a screencast.) That said, in the same way that people argue that good naming practices should obviate the need for many comments, I worry that people use screencasts as a crutch to avoid implementing needed usability improvements.

I think screencasts also guarantee that your thing sort of works, which adds an air of professionalism.

That said, gripes include: hard/impossible to speed up while watching, hard/impossible to bookmark a specific spot, hard/impossible to skip over bits that you already know, requires headphones/speakers/video, often hard to see what's going on, and easily goes out of date.

YHBT. YHL. HAND.

http://www.youtube.com/watch?v=SAWDYaWAVQQ

"Whatever position I took originally, I would change the position to exactly the opposite and tell the Macintosh people that I was completely wrong and they were completely right all along, and the numbers would go through the ceiling!" -John Dvorak

Regarding the last bit, this exists in a couple forms. The one I can remember at the moment is stickK (http://www.stickk.com/).

While the web seems to be alight with comparisons of Go to other languages (Python, Lisp, ...), what seems to make this post unique is the comparison of Go to a forty year old language in little common use today.

To me at least, this begs a question---where do the great trolls of the Internet go to brag about their exploits? I remember in the golden age of Slashdot one could read "TrollTalk" and see links to the best posts with the most comments trying to misguidedly correct the original troll, most mod points up or down, and so on. (I think it even had a scoring system and a best-of newsletter of sorts.) Where do the great trolls of today hang out and brag about their exploits?

I don't mean just the sort of mediocre trolls who might compare Go to Visual Basic or something, but the sort of highly specialised trolls who might start a fire storm by comparing it to Algol-68. Where does one look to find the Orson Welles of modern technology trolls? Has blogging killed the once fine art of trolling?

Actually, I've been using encfs on Dropbox, and size really isn't much of an issue. In practice, I mostly want to synchronize dot-files and other small files across computers, and broadband to Dropbox isn't terribly slow.

The real issue is conflict handling. When Dropbox detects a conflict between two versions which have become out of sync in a non-fast-forward type way, it creates two versions, the more recent version, and the old version with the host that created it and the date.

This is probably the right thing to do, but since it is doing so for the encrypted versions of the files, rather than the unencrypted versions, you end up with files like: "./j85-6kOFk3Gg0Lg1TiuqgqIz/8p2kJa8tboYSSpadfuiE9Tuq" and "./j85-6kOFk3Gg0Lg1TiuqgqIz/8p2kJa8tboYSSpadfuiE9Tuq (quiet's conflicted copy 2009-10-21)" with seemingly no way to access the conflict version through encfs.

If anyone has an idea how to fix this, I'd quite appreciate it. Otherwise, I'd really caution anyone from using encfs as a solution to encryption on Dropbox!

(TrueCrypt is a terrible solution for this as described in the posts below, due to needing to sync the whole volume every time.)