andraz

There are plenty of situations where most of RESTful idea has no meaning.

Zemanta's API is one of them. We're not dealing with resources, we're dealing with analysis of text (natural language processing, information retrieval, etc...), so except of the basic url scheme, there's not much that you can take from the RESTful idea.

> A nice simple how-many-in-stock web service would be a lot easier to maintain.

... yes, _after_ you are doing millions of transactions per month. But until you've actually built a business, there's no point in setting up a web service.

If there's a web service, there has to be a team to maintain it. This means you need to be big enough to have one team per web service.

My biggest grief is that most of the engineering meetings consist of discussions of how to get a piece of data from one system to another.

Instead of working on things that matter to the user.

I can't agree more. First you need a product that works at least for some users, then when you are expanding the user base it's time for dashboards.

Dashboard answers the question: Is my product right now working for my users?

Eventually it can also show trends in terms of resource usage, strange peaks, etc. But the primary function of dashboard is answering the question: "are my users getting the service?"

Additional note: dashboard works wonders for getting us - engineers - understand that there are real users out there right now and thus making us want to build even better product.

It is bunch of heuristics... just slightly more of them producing (in my opinion) slightly better result.

The core of the article however points at the main problem: no evaluation dataset on which we could compare these algorithms.

Great reasons to upgrade.

But they should also mention they are taking away our beloved print and quick string formatting.

This was written by Slovenian ambassador to Japan. There's quite strong opposition in Slovenia against ACTA - even the Musicians's guild is against it.

There are going to be street protests in Slovenia against ACTA on Saturday.

This can still be reversed - by EU Parliament not ratifying it and in turn giving other European countries an excuse not to ratify it.

What author suggests at the end (abolishing private corporation) is pretty absurd. But he makes a pretty good argument that the economy might be a zero-sum game if technologies can take us zero cost of production, distribution or providing of service. We're closer than ever, but still very far from that point.

If cost of replication and distribution of every product globally can be driven (close) to zero, only a few best products in each category can have economic existence. Therefore as distribution gets more efficient it's winner-takes-all market. In winner-takes-all markets wealth distribution is extremely skewed by definition.

So if technology takes us so much further that a lot of people won't be able to add more value to the economy than they will be 'taking away', the governments will have to find a way to keep balance before social unrest takes the main stage.

Some more thought should be put in this direction. Here's a milder possibility of controlling for winners-take-all phenomenon - limiting the maximum market cap of companies and then splitting them up after they reach it. This wouldn't shock the economy as much and would provide sort of balancing and competition at the top. Also stock owners wouldn't be too much worse off.

Twitter stream of what people are finding in the data: http://twitter.com/#!/search/realtime/supervizor

Zemanta, Slovenia, EUROPE

Zemanta is changing how writing process works. Text editors stayed the same for last 20 years and it's time to make them smarter. Zemanta analyzes text being written in real time and helps the author with suggestions. You can check a basic demo at http://zemanta.com/demo

We're looking for:

* frontend engineer (Javascript): Creating user interface for Zemanta's personal writing assistant While authors write their text, Zemanta analyzes it and displays suggestions inside their existing CMS. You'll be creating interfaces that integrate smoothly into existing content management systems and provide whole new way to author text. We are looking for combination of knowledge of JavaScript inside browser environment and familiarity with CS foundations like basic algorithms and data structures.

* backend research engineer: Research and development in natural language processing and information retrieval. Implementing and evaluating new algorithms in word sense disambiguation, trying out ideas to improve document-to-document search. Languages are Python, C++ and Java. Strong computer science background is a requirement, .

* backend infrastructure engineer: Scaling out the crawling infrastructure, help with the tooling for researchers (scaling on Hadoop), constantly improving deployment processes, logging and monitoring.

Zemanta’s research and engineering team is in Ljubljana, Slovenia. Our team is international. City is lately becoming a small start-up hub, so there are plethora of events related to tech, business and similar. You’ll have a hack of a time in Slovenia, we promise!

Address: jobs@zemanta.com

Think about Slovenia too... very vibrant start-up community here, English is a standard too.

Let me know if you know of this approach being used somewhere in production processing millions of web pages. I would be very interested to know how they overcome the difficulties!

I can imagine the cost/benefit of the approach is favorable for largest search engines like Google and Bing that are trying to squeeze last few percentage points of precision out of results. For everybody else, the engineering and scaling difficulties are probably too big. I'd love to be proven wrong.

Methods that do clustering on similar web pages are mostly too CPU intensive for processing larger sets (we're talking millions of web pages). They are also harder to scale from data-locality perspective, you need to figure out which pages to put together and then get the data together.

Looking at pages in isolation is much more horizontally scalable. You can take a look at Webstemmer (http://www.unixuser.org/~euske/python/webstemmer/index.html) for a method exploiting similarities.

The data is useful, but not in a way you used it.

I don't think you did a good job of disclosing or the problems that might lie in the data.

Since I've been following crunchbase both from the data perspective and from the perspective of how much resources TechCrunch is devoting to it, I can assure you it varies wildly.

Also the editorial policy has changed a lot in that period, in about 2007/2008 they started putting much more emphasis on international start-ups. So there are specific skews that you should be aware of and disclose them in the blog post.

So I think you did a nice job, but conclusions are not to be trusted at tall. Yes it might be the most well-maintained open data out there, but it does not make it in any more useful for this kind of analysis.

Take this with a grain of salt.

CrunchBase does not by any means offer stable and comprehensive picture over the years. It has been maintained with different levels of commitment, especially for entering in old data (since Crunchbase did not exist in 2000).

CrunchBase is a great resource, but doing those kinds of statistics without appropriate research of how consistent is the coverage is at least sloppy if not willful negligence.

When were those movie rated? Now or at the time of the release?

If they were rated recently then there is obvious bias there. Yes/no?

The idea that costs for a search engine go down with time is false. Bandwidth, CPU, memory and disk sizes you have available to index and retrieve information do get cheaper. And this means they get cheaper for information-producers too. So the need to increase the amount of data you index eats away the decreasing cost of hardware.

Naturally this holds true for general search engines. Vertical ones are a very different beast. And I think Peter Thiel was not talking about specialized vertical searches like patent search.

When the world gets into H.264 there's no way back but waiting for 20 years for patents to expire.

This was the last chance to at least try to keep the web video both open and free. It's a far shot, but it's at least a try. Before this move, the game seemed beyond lost, now it is only probably lost.

Flash will support WebM too, Skype is using it, so there are some chances. It's a very long way off, but now WebM has a fighting chance.

"Rather, the point is that in practice developers don't let royalties impede their implementations."

This is simply not true, MPEG-LA would go after any commercially successful project not paying royalties.

Actually when you really need to decode a blurred or mosaiced image you can do even more tricks. Especially when they are screenshots. Since you can take a pattern (digits) and blur them with all possible options of certain most popular image editing software, you can then do massive number of comparisons to see what comes out right. It's massively cpu intensive, but I am sure people that need it can do it.

Blacking out the section entirely is the only proper way, since you really want to be sure you are destroying the information in the image, not just dispersing it.

Even then, if you are removing a single digit it can be partially recovered by observing kerning statistics, etc.

exactly the same use case here... :) Maybe Amazon should go a step further and enable people to get logs aggregated together by the chosen time unit (hour, day, week), so we won't need to do round trip just to join files.

What's the point of Skype on Android if you need to be in WiFi Zone? In that case you can skype with your laptop in the first place.

... what's the name of the alternative that actually works over 3G?

wouldn't starting just 6 months ago be a bit late in any case?

Actually modern JavaScript engines depend on dynamic type discovery (via different forms of tracing) much more than on global static analysis.

Do not be fooled, right now Python 3 does _nothing_ to increase parallelism of two python codepaths being able to be executed at the same time.

It just fixes some pathological cases where GIL behaved even worse than it should.

Unladen doesn't try to fix GIL either and it is doubtful than it will be able to do so, PyPy has a greater chance of achieving that, but it's still not on the roadmap.

You do know that every temporary storage of a an object in python causes an increase of refcount which is a write? Each of those writes causes a 4kb block to be copied, and they are eventually scattered around enough to matter.

What we've done is mmaping multi-gb files into memory by separate processes. It works, but it's a cumbersome workaround which makes you do way more work than you would ever wish to be able to use multiple cores.

Brett in the comments points out that they would accept a patch doing away with GIL if it keeps C module compatibility.

However the GIL cannot be done away without breaking compatibility with existing C modules while keeping the performance up. That's impossible requirement. Doing away with GIL means doing away with refcounting, or the performance penality is unavoidable. Doing away with refcounting means major changes of third party modules C API.

The issue people have is that Python 3.0 is introducing a bunch of stuff that makes code incompatible in different ways - mostly for aesthetical reasons, but does not deliver stuff that people are calling for again and again - performance and dropping the GIL.

Oh, and for people saying "use multiple processes" have never really had a problem where the process has multiple gigabytes of state that it needs to do its work. The state that can be easily shared in multithreaded mode, but can be only shared with a great deal of manual work in multiprocess situation.

Yes, you then do distribution over multiple machines too, but that doesn't solve your problem that you cannot afford running a separate python process on each of 12 cores, each process taking 10gb of memory. Because you only have 32gb.

However complaining won't help. Maybe Unladen Swallow or PyPy eventually resolve the issue, but it doesn't look like it's on their priority list either. Well, though luck for us lazy programmers who are getting all this stuff for free.

So thank you Brett, Guido and others working relentlessly on Python. It's great. You know what bothers us, but even if you don't ever fix it, I'll still be grateful for all the hard work you did!

Patents cannot be used to prevent someone publishing research, source code, explaining how things work, etc. Infringement happens when the patented matter becomes part of commercial product, is being used commercially or sold. Not by publishing either textual description or source code. An example of this is x264 codec which is perfectly legal (to distribute and work on) even in countries that happily allow patenting everything under the sun. The infringement happens when it becomes used as part of the product or service.

Patent system's primary 'deal' with the inventor is "time-limited monopoly in return for a disclosure of how you did it".

Leaving aside the question on patentability of software and algorithms, I am sure talking to FFII, EFF and similar organizations might help gathering traction for a proper pushback.

If you need any more help from us (Zemanta), let us know!

we know a lot about these problems & solutions, so send us an email at info@zemanta.com

Andraz Tori, CTO at Zemanta

It's much more than tags - in-text exact entities, deep categorization, related news, etc :)

The exact disambiguation of mentioned concepts and entities is what is exciting and underused by web developers right now.

Andraz Tori, CTO at Zemanta