Hacker News Reader

Comment by thezoid | Hacker News Reader

Or just populate a hash with the input and see if the entries exist in the hash. If they do print it otherwise print the non matchy one.

View on HN

Exactly- hashes and sets are much more efficient than arrays.

Regarding the speed vs memory occupation, there should be comparison between linkedlist vs. hashmap. I remember that linkedlist is faster. So it depends on the language support.

You can google it more. I don't have time to find out about the Big O footprint for both now. Here is one link discussing about it:

http://stackoverflow.com/questions/7975802/when-to-use-hashm...

Linear lists are.... linear. Hash maps are constant access. N doesn't need to get very large to see orders of magnitude difference—one you'll certainly see if you switch on level0.

A linked list is not faster than a hash in looking up an element. The operation is O(N) for a linked list and O(1) for a hash.

Of course. If it's in Java, I'd definitely do that. We are using Hashmap everywhere. Since the original code looks like this, I thought Ruby does not support hash.

We usually implement code on the application level, so we don't really care about the underlying algorithms. But if we have to implement on the system level, we do care about the how to manipulate the memory and number of executions, like Big O.

The original code is like this because it's an exercise. Of course ruby has hashes.

The CTF does not give a description of the problem, but the code instead. And it persuades the new users not to change algorithm too much, just save a few steps, which implies to have minor changes of the code only.

Look at my another comment, hash may not be always better than list.

Sure a hash isn't always better than a list. In this case, it most definitely is. :)

Yes, agreed. I was sort of mislead or implied. It'd be better if CFT can give out the problem description followed by the sample code or don't give any sample code at all.

For me, it makes more sense to receive the detailed information, especially the system condition and restriction for trouble shooting and problem solving, followed by brainstorming, instead of going directly to fix code. Because we are used to the pattern to make minimum code change, especially in production. If the code should be completely changed, then we need to know the requirement and re-implement it. Sounds like we have different convention though.

The change required is a tiny one.

Unfortunately, I cannot agree with you completely.

If there is no language constraint and the system resource constraint, to the problem we have understand so far, using Java will be the fastest and easiest way without hashmap.

Load the complete file as a string (depending on how the size of the data set, up to 2^31 - 1), then using string.indexOf() function will get the best result.

The underlying algorithm for indexOf() is implemented by JVM in C code which is must fast than any other implementation.

My gut told me that it's weird to use hashmap to do string lookup. Everybody knows hashmap is used to lookup key-value pairs. The real reason for not using hashmap here are:

1. hashmap's lookup Big O is O(n), but not the build cost. if the data set size is huge, it takes long time to build the hashmap since every new element exceeded the initialCapacity being added needs a rehash

2. the underlying implementation of indexOf() will use a sort of algorithm called "automata" or something else to do a fast search within a string.

So there are lots of alternative solutions. Don't always think there is only one. I'm not in this field, and I'm not interesting to get into to it too much. But I don't think the best answer is that tiny change.

This is why I suggested to consider if you are doing application level optimization or changing system level algorithm. Building software is a lot more than code manipulation. Understanding requirement is the first step in the SDLC (Software Development Life Cycle).

Sure, you're welcome to do try it in another language, they almost encourage you to. No one said there's only one way, but for many simple problems a hash is a good solution.

The best answer is one that passes the test, and using ruby it is pretty easy to do so, but you could do it any way you like.

I'm not sure what you mean about "many simple problem". Is this one of the example for that? I'm trying to compare the two alternative solutions for this problem:

1) Hashmap 2) string index

which is better? I think about it what I'm going to do if I'm using it in my web application.

First the answer is still relying on the size of the data set. If the dictionary is huge, say up to 4G, I'd use string index vs hashmap, because the memory space is expensive. And how to break into mutiple sub strings is another performance tuning issue.

If the problem is simple enough with not too large data set, hash will be working.

When I mentioned "one way", I mean the "best way". So now you are talking about "The best answer is one that passes the test". So do you mean that all the answers which can pass the test are the best answers, or there is only one best answer which can pass the test? I don't put my personal preference on the problem solving. I'm always looking for the best solution for a particular problem under certain condition and constraints. Once we figure out the answer, coding implementation using which language does not matter that much, unless Ruby does not support the same algorithm of string.indexOf() as Java does.

Hope this discussion helps.

So do you mean that all the answers which can pass the test are the best answers

Yes, what I was trying to say was that the performance required is just that to pass the test, not more, and the dataset is a few MB here, not 4GB. This is actually really similar to a lot of problems in real life; you can spend ages trying to find a platonic solution when a simple solution works fine given the dataset and requirements (return an answer within 200ms for example). Sometimes simpler is better, and even if you can improve the solution, it won't really matter to whoever pays the bills.

There are lots of solutions though, I tried a few just out of curiosity and you can of course improve on a hashmap - the possible solutions to a problem this small are pretty similar whatever language you choose, and sometimes when a dataset is this small other more complex solutions are slower (unless you preindex).

I have no intent to choose complicated solutions over a simple issue. First, I'm always concerned about the size of the data set. The first sentence I said, if the data set is not big enough, you cannot tell too much difference. Second, I didn't insist on any solutions, so far, I proposed three of them and would like to discuss with you guys on the pros and cons under different conditions. Some people in other posts told me that the first solution using sorting + binary search actually is way faster than hashmap, which deserves more than 2000 points. I tried to verify whether it's true, but nobody answered. Some people helped to provide the Big O comparison on the hash lookup algorithm and linklist, which is also constructive. But did you learn from it that building hashmap cost more than O(1)? Lastly, I proposed the Java solution with string index, did you ever hear about that?

I don't need to spend too much time on finding the solutions, they are on top of my head. Depending on different conditions I'll use different solutions. 4GB is the upper bound of string indexing, if it's being used for web indexing, it's still not enough. If in this case it is used in document indexing for enterprise level with a few MG, it's fine for using any of them. But the difference is the score you get.

I guess eventually you will pick hashmap solution because you have the indexes built in the lookup, which makes more sense over other solutions, but you (or they) don't give the condition out in the first place. How can we discuss based on that? Looks like I pretty much wasted my time on this issue and was taught to learn that making things simple is better. Thank you.

http://www.ruby-doc.org/stdlib-2.1.0/libdoc/set/rdoc/Set.htm...

Thanks a lot for your reference. A couple days ago, I was almost convinced to use GO after I took a look about half of its examples. People said, we are not going to catch up the languages which are changed every 6 months. The same applied for Javascript and JQuery, you don't really need the latter, and there are so many JavaScript libraries. Once you change your job, you may abandon some of them. Look at my another comment. For the same reason, I'm not going to take time to learn a new language.

You dont have to use Ruby. I actually did it in Python because I dont know Ruby. Just had to change the !# and fiddle around a little with the input reading.

Thank you for your advice, I'm really new to this game and I even don't know how to use github. Don't laugh at me, because I have enough experience on coding by myself and reading/modifying other people's code. I don't have time to play coding on CTF, but I'm curious about how other people are thinking about it, especially about distributed system.

Coding is the last thing we need to concern since various languages are available to implement. System architecture design, data modeling, application performance, security and algorithms are more important.

Regarding the language itself, I understand that Ruby and Python are quick and easy, but that's not for real production systems. Lots of teenagers are using it now. Maybe you are not happy to hear like that. Here are two blog articles regarding it:

http://bingobo.info/blog/contents/do-not-rely-on-other-platf...

http://bingobo.info/blog/contents/having-a-solid-foundation-...

LOL @ "don't let other people control your tools" and then going straight into "how to set up your Windows environment" in the next post after one of your links.

Thanks for reading. Is there any conflict between the two?

The original words was "You're giving them a throat to choke". The minimum set of tools is always necessary either on Windows or Linux, especially from Open Source. Just make sure don't take the risk of your application to do re-implementation later.

Lots of teenagers are using it now.

I'm curious how people can put down vote on my comments. According to the guide line from HN, down vote is only available for people who have enough karmas.

BTW, if you are not happy to hear that, I'd like to know your opinion.

I didn't downvote, but I'm guessing it's the jab that "Ruby and Python are quick and easy, but that's not for real production systems."

There are tons of high-profile sites using ruby or python in production. Reddit is written in Python, Twitter got quite a bit of mileage out of Ruby...

There is a lot of teenagers with high enough karma around :)

I don't look down on teenagers if they are able to build websites using those languages. Maybe this type of exercise is really good or suit for them. And definitely coding is the no. 1 skill they occupy.

What I'm talking to are the hackers who are aiming for running startup companies and those who are going to grow to be software engineers.

That's exactly what I did. My fixed code is smaller than the original code and fast enough to go to level 1. It is in perl (I am not fluent in ruby).

I'd love to know that you have reference to show that with a large amount of data set, hashing is faster than linked list. I don't have the reference at hand, but I know that hashing has a higher cost from speed and memory point of view.

I don't see the original code is a huge issue per se which requires 19 levels? of improvement? I'm curious about how you guys move on though. Will appreciate if you can keep post your solution and progress.

I guess at the end of the entire program, people may learn how to send the query to multiple indexing servers in a concurrent (I prefer this than "distributed") system and then gather together of all the results. At the end of it, it shows how advanced algorithms Google search engine is used to index terabyte of data.

Is that the ultimate solution for a web of data? Take a look of the discussion: http://bit.ly/1f7xIve

I changed one line to make it work faster.

From the "best" score I was worse than it by 150 or something. Not that I care really.

Looks like one simple change to hashmap may not be the best answer. May I draw the conclusion from your case like that? The best score now went up to almost 3000. I couldn't imagine how many possible ways out there and one of them could be that excellent. Just curious.

Sorted list + binary search was 2000pts and ran much faster than the hashmap solution.

Thank you so much for your info. This is something I'm looking for from the beginning of the discussion.

http://www.algolist.net/Algorithms/Binary_search

This is my original answer! I should not sort the input list though since it's not necessary.

But is string.indexOf() will be even faster?

Reply on news.ycombinator.com