Google rejected me and now I'm building a search engine
daoudclarke.net
daoudclarke.net
In my experience the only reason you should say "I don't know" is if you're going to follow it with "but if I had to guess" or similar. Sounds like the interviewer definitely came on strong but being able to ace the psychological part of an interview is often as important or more important than the actual solution.
Edit: to clarify, your advice is good, what you said isn’t dumb. That criteria is dumb, in my opinion. I don’t want my colleagues to spend a bunch of my time guessing on an answer I could easily lookup or find on a calculator.
Companies want to hire people that do things, not question everything, regardless of whether those things ought to be questioned. You’re being hired to achieve business objectives, full stop.
In fairness, following your stated requirements; no one would ever be hired.
There's a direct contradiction such that it can never be true.
Hint: Descartes Rule of Method
conversely, it's reductive to compare it to a generic ban on saying "I don't know"
my first job was a startup I built, starting from being a waiter. There's a lot of people who don't even try, sort of reject the premise of engaging with the question.
Saying "I don't know", then looking at the interviewer, is about the least valuable interaction you can have with an interviewer, and there's a shocking # of people who do that: whether it be freezing up, some sort of implicit commentary on the question, or a form of performance art, whatever the implicit intent is, in 99% of cases, it doesn't serve you at all.
If I ask you how to fluffle nuffle the snorbknobs, you can at least come back with "I don't know, I may have misheard you: I've never heard of fluffle nuffle. Can you give me a hint?". If you're at the end of your rope with tech interviews: "Excuse me, those aren't sensical words, I'm worried you're having a stroke"
"I don't know" and silence is NPC stuff.
And part of a real job involves solving real problems. In an interview, you have a very limited time, so you have to come up with questions that test the candidate's ability to solve problems within a certain timeframe.
Another example is from my own "worst interview ever" where the guy asked me to implement a BFS and a DFS on a tree. I mentioned that I'd literally been debugging an issue with a DFS algorithm all day at $dayjob, so should be a piece of cake. I wrote out the DFS example in 10s, but then when I got to the BFS, I spaced out on how to write a for-loop. The interview literally just laughed at me, saying, "that sucks for you" so instead I simply showed him how a BFS works and his answer was, "I don't care, I want to see code."
As an interviewer, I get it. I expect you to know concepts, not be able to perform rote actions and put on a show.
Either way, if a candidate misses such a question it shouldn't be a negative. Using such a question as a filter is a bit ridiculous.
IMHO it is also the reason they can't generally make their own (good) products any more, without acquiring / copying -- that is not the type of people they hire (more artistic / risk taker types).
Well that's the opposite of what the interviewers are looking for. They DO want to work with such people.
Google didn't reject the candidate, the interview loop he got rejected him for any of the myriad of knee-jerk split-second decisions made on the candidate's character in an unnatural and stressful environment.
I've been in hundreds of interview loops at $Megacorp. You can stack the deck with preparation, but sometimes you just hit a loop that doesn't click with you.
But that's not how problem solving works. You need work your way from what you do know to get to an answer for what you don't. It's not guessing -- it's taking the background knowledge you do have and applying it to the problem at hand (and every engineer should have some background knowledge, even if it's just basic arithmetic).
That's what these kinds of questions are testing. Someone who immediately gives up is probably not going to be resilient in the face of new challenges.
I've quit several interviews because of these particular questions since they are simply wasting my time and not testing my "skills".
As a candidate, if I do not know, I say I do not know. I can offer my thoughts on "but if I were to design this from scratch" based on my experience, but only if the interviewer is interested in that.
As an interviewer, I hate when candidates pretend (or believe) that they know something when they actually don't, and just confidently make stuff up like ChatGPT.
I am sure you also go to your employers and tell them that asking super unnatural questions in a super unnatural pressure environment is also a super negative signal.
You do this right? Right?
Possibly, but I’m sure these aren’t exact quotes after 15 years and I expect they reflect how the author remembers feeling a lot more than the actual conversation.
But if that doesn't come naturally to them, "prospective interview candidates" might also consider if that's really a problem. There remain plenty of fantastic engineering opportunities that don't involve memorizing dance steps.
I've found you often need to directly ask the interviewer not to do what they're doing when they try this sort of thing. Sometimes they're just bored and want to talk because there has been a moment of silence, or because they're enamored with their interview question.
If it was trivial, there'd be no point in asking.
It's a beginning question to get you warmed up.
Remember to take you Vyvanse so that you can stay focused, and make sure to throw in some beta blockers so that you stay calm during this toxic hellscape of modern interviewing.
I haven't done a whiteboarding interview for a while, but I remember them vividly. Hot flashes, sweating, stomach churning, anytime I'd be asked a question I am definitely capable of answering, my brain would shut down and refuse to start back up again. The most apt thing I could compare it to is stage fright. Even something like simply multiplying 2's would seem impossible to me in that state of mind.
Aside from seeking professional help for dealing with anxiety, I'd recommend programmers with anxiety to avoid whiteboarding interviews [1] or at the very least let it be known ahead of time that you get stage fright.
That's my big problem (both as an interviewer and as an interviewee) with the current "best practices" in tech interviews. We're evaluating the wrong thing. This is how smooth talking, charismatic phonies breeze through and we find out in 6 months they can't code. Unfortunately by then, they've often gotten themselves promoted to Director.
That was a turning point for me, because as far as questions go it was very fair. And yet, it had the same effect. 20 interviews later I was acing far more difficult questions, being rather polished (and ahem, interview questions aren't generally very unique). So I guess I interviews at that place too early. Maybe its useful? IDK. For me personally, it seems like really good people have a track record of shipping good stuff (or doing other good things in life), and can generally explain what it is and how it works. They usually have former colleagues you can talk to as well. Some of them even have side projects you can look at. But ah when the interview mill is in full steam, its hard to individualize to the candidate. We have a feature factory sir, and what we need are some good cogs. That's how you maximize the value you achieve from a developer after all!
Unfortunately acknowledging that you are in any substantive way human is a huge turnoff to these companies.
And approaching them with the suggestion that the interview process should have at least some resemblance to a 2-way street will be seen as a huge red flag, and a sure giveaway as to what an incorrigibly self-entitled primadonna you must be for even brokering that suggestion, as well.
https://news.ycombinator.com/item?id=40850725
Rather than this clickbaity "Google rejected me" story about something that happened 15 years ago, here's a link to the actual project:
Looking back at my younger self and this person I can't help but cringe. It was a long uphill battle to be okay with rejection and I still struggle with it but I can't change my natural emotional response but I can control how I react to rejection.
I hope that OP will find his way without channelling his anger in ways that is counter-productive.
(Remember, someone building a search-engine is likely very, very familiar with SEO.)
This strikes me as fairly petty “I didn’t answer wrong, you asked me the wrong questions!”. Honestly it’s the recruiting process working as intended - folks with this type of attitude don’t make good team members in my experience.
Also > At the time “Don’t be evil” still meant something. Now it seems like their mantra is just “Be evil”.
Seems really petty. It’s a shame because we could good big tech alternatives, but building something out of spite without much perspective is unlikely to create a good alternative.
Imagine some weird employer where a narcissist has managed to capture the hiring process. They might well ask the wrong questions to torment, and in that case someone who became flustered could well say that to them without it being petty and without the hiring process working as intended. So no, I don't believe necessarily that someone with "that attitude" would always make poor team members.
> Now it seems like their mantra is just “Be evil”.
I know it's just empty rhetoric for most, when they use the word "evil"... but it's still amusing to me. How many times do you have to flippantly suggest that they're evil before you start to believe it literally, and what happens when you start to literally believe that? Is it possible to remain rational afterward?
Most people today are not properly taught to correctly recognize evil. Often as a result of this, they become evil themselves when they falsely justify evil acts which are unjustifiable.
There really isn't much ideology, or hyperbole to this. From a non-ideological perspective, Evil is any act that does not promote the long-term beneficial growth of self or others. This includes destructive acts regardless of intention or knowledge. The more destructive the act against long term growth, the more 'evil' it is.
Evil people are people who have blinded themselves through repeated acts of self-violation such as false justification, where they no longer resist doing such acts. The only way to prevent them from continuing is to stop them, and you don't argue with evil because they don't see what they are doing as wrong. They have willfully blinded themselves, and once blinded they can't see again.
This is the generally held rational view of evil. Most people today instead are taught to embrace Tolstoy's pacifism (without knowing it). Live and let live. Turn the other cheek. Its in a lot of places.
The problem with evil, is that it spreads exponentially, and consumes all until nothing is left, it turns those which it consumes, and bends them to that destructive will.
If you follow this Live and let live philosophy, Evil will grow faster than those who seek to survive, for themselves and their children, eventually it overflows and causes catastrophe.
The type of people that embrace chaos and destruction, given sufficient technology or sufficient time will cause extinction, since they turn it on themselves when no one else is left.
Illyin wrote a treatise back in the 1900s which appropriately covered this subject matter.
While there were some aspects in his writings that were not generally agreed upon (or considered correct today), he very much captured the common definition of evil known at that time, which remained consistent up through WW2 into the 60s; before sentimental moralism and nihilism introduced in the indoctrination within the public education system started watering this down for future generations.
It is such an attractive and alluring thought, that one cannot do evil unless they choose to knowingly and actively do evil, where opinions and beliefs cannot be evil (nihilism).
Obviously, to anyone rational, this last line of thought is pure fallacy (by trivial contradiction), but also commonly used as a basis for false justification...
Most people who believe they are good today, often are evil and they simply don't know it. They are products of their 'disadvantaged' environment. Destruction follows them around like bad karma in their interactions, but they are rarely touched by it themselves, they act like parasites promoting and inducing evil in all that they do.
With the advance of society, these people have also found that systems are a very efficacious way to induce people into these acts as well (great acts of evil).
As a system, any act that is contentious, or dubious can be split among several roles with only a select few knowing the actual outcomes in any reliable detail. In isolation the parts you may work on don't appear to be evil.
This can happen to anyone just doing a job, if they don't have full visibility because a business is a system.
To reiterate, to be an evil person, one simply must be willfully blind, and the actions performed must support destructive outcomes.
You can't take elements in isolation and draw a conclusion about the outcome of a undisclosed system. A system is what it does.
How long-term? What sort of clairvoyance is necessary for that? Which others? Those who hold stock in Google do well, it helps them to grow does it not? They employ about 100,000 people directly, and many more indirectly who benefit economically because Google facilitates business transactions they couldn't otherwise make.
> will cause extinction when they are not stopped.
The moment your species' fertility rate dropped below replacement, your extinction was already carved in stone. Fussing about Google and whether or not making you watch bad advertisements before watching Youtube videos is evil is pretty silly, don't you think?
Just what evil acts are they committing exactly, and how will this result in even something like a figurative extinction, do you think?
If you are interested in the answers to these questions, I'd kindly refer you to the original treatise I mentioned. It will cover these answers and more in far better detail.
You would be mistaken to think that employment, stock valuation, and that business transactions couldn't occur otherwise, are necessarily good things or in fact actually true. It would be an over generalization and other fallacies. They get their money for operations regardless through preferential loans. Stock valuation is very much floated on option structures which are just another indirection.
https://www.amazon.com/Resistance-Evil-Force-Ivan-Ilyin/dp/1...
> They facilitate business transactions they couldn't otherwise make.
It sounds like you may have drunk their kool-aid. This is almost verbatim what their marketing department puts out. They lie often, and recently were caught in some of those lies. Not just to customers, but to the US Government as well. For example, they have claimed that pagerank and Authoritative Score are not collected or used, but their leaked internal API documentation shows otherwise.
The stock market isn't a good example for you, because Valuation is transitory, and what drives the company is in fact the debt that they hold and their preferential treatment by banks which print money from nothing. The advertising market sector withered and died when google took over (funded by government grant).
Regarding business transactions that they couldn't otherwise make, compared to what? If google left the market, there would be a vacuum because they are a monopoly and they got there through Lax antitrust oversight, and deceptive business practices. For there to be winners, there had to be losers.
Prior to Google; Radio, Newspapers, and Magazines, largely thrived on ad revenue which is now no longer available. They all have to kiss the ring.
There is also a fair argument that google actively engages in enabling ad-fraud on a much greater scale than previous structures. They claim otherwise, but they've also removed metrics to ensure no one else can contradict them. Uber cut their entire ad spend in 2016 amid controversy (origin of ads), and they found the metrics didn't line up with their actual sales following that change.
> The moment your species fertility rate dropped below replacement ...
The fertility rate is a lagging indicator. There is nothing rational that proves that it will continue to go down once it gets below a threshold. Generally there are reasons why it reduces from the norm, and when those reasons are addressed then the rate would no longer be depressed and it would go back up.
> Fussing about Google and whether or not making you watch bad advertisements
I don't recall mentioning anything about youtube. I'm more concerned with real harms.
> Just what evil acts are they committing exactly
To adequately answer this rationally, I'd need to reference a few books from subject matter experts which are not common knowledge today at a bare minimum to keep this response to a sufficiently short length. I'll reference them at the bottom.
The short gist is, both as kids (developmentally), and in times of isolation or extreme duress (torture), the mind enters a vulnerable hypnotic state where permanent changes can be induced, sometimes very subtly (without your knowledge).
This has been proven in the 50s through a structured approach of distorting reflected appraisal, and other methods, the former is the distortion of the biological mechanism all humanity uses to pass culture to our offspring.
Google structures their results and other aspects of their services to use this mechanism to induce bias for their own short-term gain. It largely happens below a conscious threshold, and there is little if any defense if you don't know about it.
To the uneducated (in this area) this sounds on its face fantastical to most people largely because media has associatively primed us to think of these things like the silly trances we see in movies, but it is real and rational science backed from the 1950s (POWs and torture). This has not been refuted, and its known that rational thought is the first thing to go in these torture structures (which need not be physical).
Additionally, there is an uncanny valley where psychological reversals happen when you think someone is trying to manipulate you. Its an irrational emotional response and adaptive. Anything that we might view as a communication of meaning is open to distorted reflected appraisal, it is a hard-wired vulnerability.
When distorted reflected appraisal is not consistent (and it never can be, unlike actual reflected appraisal), weird things happen to people's psychology.
Some people disassociate (more like zombies/automata), others become psychotic (and potentially violent), only a few people have higher tolerances and seem to be unaffected (but they are still somewhat changed, hollowed out).
Google is using this mechanism for their gain, and enabling businesses to do the same through their platforms (pay for play). In addition to this, they also are removing history (not segmenting spam from legitimate material; removing it all), they lie, and they also make weapons (anything a weapon depends on is part of that weapon system). All of these things are evil because they all promote outcomes of stagnation or destruction.
> figurative extinction... How might it happen for real?
Our society is past ecological overshoot. A review of Malthus law of population shows technology can allow overcoming ecological limits to growth, but its primary dependency is organization, and rational thought.
If people are subtly induced into either of the two states mentioned (through mental coercion), its just a matter of time before systems fall into ruin and fail, and many of the feedback system's in place (including our perception) fail, or encourage brittle changes which induce more failures.
Sieving purchasing power in the economy for example makes the general person's life so dystopian that you end up with something like the plot of "In Time".
People will cease having children because it costs too much. When the exposure to torture systems becomes so great, and life so insufferable, some will seek to end their own lifes, or their childrens (as happened in the 1500/1600s regarding the natives under Pizarro iirc), some will become psychotic in their madness and act out violently against innocents before being killed themselves, some will simply wither and eventually die.
This is documented historically in a book called "The Wealth and Poverty of Nations", in the chapters of the effects of the Spanish Inquisition, and the expansion to South America.
Inevitably, all paths lead to what? (slavery, then death, which is destruction). Meeting the definition.
The world is the culmination of aggregate action. Evil acts beget more evil acts, and we are at a point in society and technological development where corrective counter organization can no longer occur. Evil overflows, and unfortunately, the lessons learned by East Germany's Stasi were put to horrifying use by all other supposedly democratic governments without the knowledge (or education) of those they claim to represent.
References:
Dr. Robert Epstein (Editor in Chief at Psychology TOday) has a good talk on what google is doing. https://www.youtube.com/watch?v=3GE3HoJMEMw
Robert Lifton "Thought Reform, Psychology of Totalism" (in depth PoW stories of Mao during Korean Conflict).
John Meerloo "Rape of the Mind" (includes both the details of this subject under Mao and Nazi torture)
I do agree that the spite aspect could have been reduced or removed. In fact, I couldn't really see the point the article makes: they had a bad interview, a 15 year gap, then they're building yet another search engine... to contest Google?
(If I were conducting the interview, I might explicitly explain that I'm going to ask them a few questions with the goal of seeing how they perform when they reach the limits of their crystallized knowledge and have to problem-solve from first principles/common sense — but I would still do it.)
In my interviews I now throw out these dumb google style questions, not to see how they think, but to see how they react to a silly question. The person I’d like to hire, when asked “how would you calculate how many ping pong balls fit in a 747?” they would answer “I wouldn’t.” Only one so far has given me an answer like that and has been delightful to work with.
That may or not be correct but it often shows me how willing a candidate is to jump through hoops they think they need to to get a job. Maybe that was the original intent of these questions. I have absolutely no desire to see how someone thinks their way through a silly thought experiment. My interview process should already have shown me how they think.
so i wonder if there is any better way to test for this.
when i was doing an internship once the boss asked me to take care of some task, and i said no, because i was going for my lunch break now. later that day i got berated by my supervisor for denying the bosses request. it was a pretty frustrating experience, but i was to young to defend myself.
on the other hand, in china, where people are typically very subservient at work, i once hired a student, who, after about 3 months of working with me had the courage to argue a technical question with me and stand his ground defending his opinion. he had never worked for a foreigner before, and he certainly would never act like that with a chinese superior. when he started he barely even spoke english (he could read and write it though) so i felt pretty proud that he was able to make that change while working with me.
the point of that story is that the initial response in an interview may not be an indicator of future behavior because the behavior can be changed by how i treat my subordinates, welcoming critical responses and encouraging different opinions.
Sounds more like a bored sadist who's throwing away talent entertaining himself instead of focusing on finding suitable candidates.
Reminds me of the stories about Steve Jobs 'joking' that people are fired when they were trapped in an elevator with him.
On the other hand, a) this guy does have a point - it’s very very strange that you can have all sorts of interesting, highly relevant expertise and tech interviews don’t care at all, they just ask you generic stuff that has nothing to do with the job, b) he’s allowed to complain c) you’re complaining about him complaining.
I see a sort of weird logic all the time (usually around interviews or dating) where someone is complaining on the internet about how the system is broken and they are subsequently accused of some moral failing (being “petty”). I don’t think you know anything about this guys attitude or how this guy works on a team - you don’t know him!
I guess the lesson is “don’t complain on the internet”, which I suppose I agree with.
I guess I’m complaining about people complaining about complaining, so I’m part of the problem, too. But seriously, let’s engage with each other in good faith and not just assume someone’s a bad person because they’ve been hurt by rejection.
Edit: grammar
Comparing this line here with what TFA is saying, the lack of empathy is astounding.
Now if powers of 3 or 5 or 6 or 7 were asked... Eff them...
The actual link from it says the rankings are, like everywhere else:
> To train a learning to rank model. No matter how many queries are manually curated, most user queries will be organic because of the natural diversity of user queries. Curation is still important for these results since it impacts the machine learning model that will be trained on the curated rankings.
so this not true in the long term.
I remarked that in the circumstances I'd need to know, that I'd google it and check the documentation to make sure I got it right.
The interviewer (who I later found out was the founder/CEO) absolutely laid into me for that answer, saying if he wanted people to google that a "thousand Indians graduating in computer science every day" could google it.
I tried to argue that I was looking to be employed for my problem solving skills and experience rather than rote knowledge, but he was really angry. He literally said to be verbatim, "Let me give you some interview advice, NEVER tell an interviewer you'd google something". He also made a mildly off-colour remark that if he "wanted someone just to google, [he] could hire one of thousands of fresh graduates coming out of India".
It was an experience so bad that it inspired me to create a glassdoor account just to leave negative feedback, something I've never done before or since. The recruiter was absolutely pissed, and still doesn't provide me leads, which is kind of annoying since he's the most active C#/.Net recruiter in my area.
But my point is that some people have absoultely atrocious interview manners. Interviews are a two-way street and I discovered that there was absoultely no way I'd want to work with them. (Even when I just thought they were a team lead rather than the CEO it was enough to put me off.)
The minute that he showed aggression or anger, I 100% would have just walked out. Life is too short for that nonsense.
Not every programmer is skilled to be an interviewer. Not every manager is skilled to be a hiring manager.
Had somewhat similar scenario. Company's internal headhunters had reached out to me once already before and I did few interview rounds with them and said no. Year later they reached out to me again and had to go thru tech interview again.. during that I did help(sleep) on python repl and mentioned why; since I haven't used sleeps on my own code I wanted I make sure that if sleep will yield cpu time or not. Mood of the interview changed at that point and got rejected by not having enough skills in Python.
Another case; One of the interviewers was late to the meeting and started to shout profanities cuz my Audio Quality was poor. And it was - thanks Sony XM's but the way he acted on the call really gave lasting impression on their "company culture"
the CSV question was easy to look up, and with time anyone could have solved it. failing it because it took to long was frustrating. the scope question would have been difficult to answer without experience, and even looking it up would not have been easy without knowing what to search for.
If we start with the assumption that (for the most part) interviews are a useful HR tool to hire people with, supposing we have a skilled manager to give the interviews (haha!), they will ask a series of questions, and otherwise engage in conversation which will elucidate whether or not the candidate should be hired. Presumably they are relying on the answers given, but they might also be relying on non-verbal clues... body language, facial expressions, who knows maybe even pheromones. Whatever the correct "answers" are, what if we train a candidate to give those answers without actually understanding them? What if he rehearses it? What if he can even do the body language and facial expressions?
We've just cheated the process. Candidates are incentivized to cheat the process, and you can vilify them all you like, but if they can manage the trick they can (at least temporarily) receive a paycheck which all of us seem to need. No matter how difficult it is to do this, it's likely some have managed to perfect that trick.
Furthermore, no one gets a bachelor's in "assessing interview performances". There are no degree programs for it. No training bootcamps for it. I've never worked anywhere that they send the hiring managers away to some seminar specifically about this. So even if it were possible to assess the performance, the people doing the assessing likely aren't very good at it.
If magically someone developed some brain scanner that gave perfect, empirically verifiable answers in a "hire/don't hire" format, what are the chances that a hiring manager could do even 60% of what the machine says? Flipping a coin should get 50%, if we limited the candidates to a matched set of hires-don't-hires, right? Would the hiring managers even do as well as random chance? Or are there some personality defects that have some of them do worse even than that?
Interviews are more in the realm of superstition than sound practice. They're polygraphs without the polygraph machines.
Tangent: I like .NET as a platform, but I get the impression that a lot of .NET shops tend to be toxic in this particular way.
.NET attracts bigcorps — and I don't really that they're toxic. Working in a big enterprise environment is actually fine most of the time.
But because .NET attracts bigcorps, .NET also attracts development agencies that mostly want to work with bigcorps — i.e. agencies whose sales process is designed around attracting and retaining solely enterprise customers. These agencies market to middle-managers' needs to check checkboxes and satisfy scrum tasks; and then they skate indefinitely in their contracts on a basis of "shoddy work in bounded time" and infinite make-work extensions.
These "enterprise agencies" exist to deliver internal political value for the people hiring them, rather than delivering any business value for the company as a whole. (As such, they mostly get hired by bigcorps that are themselves dysfunctional in some way. But there's enough of those to keep quite a lot of these agencies in business.)
In agencies like this, I find that the only people who stay working there, are either burn-outs trying to keep their heads down and take home a paycheck, or some flavor of awful people.
If you want to avoid this kind of experience in the future, I'd highly suggest either focusing your search for enterprise-y language shops on actual enterprises rather than agencies — or marketing yourself for your talents in less enterprise-y languages, to shift your appeal more toward SMB employers.
If he remembers that max signed int is ~2 billion, than easier to divide 4 billion by 2. 2b/1b/500m/250m/127m/64m - got 6 divisions, 32-6=26.
If you think that max int is irrelevant to the position - it is so relevant, I can't even describe, this number is everywhere, from database design to js-wasm (limited by 32-bit), from deep-learning (where some libraries still limited to 32-bit buffers) to networking (hello ipv4)
Figure that problem out first (something novel and useful), then start marketing yourself.
Right now you just gave us a story we've all lived (academic hazing) without any plan of action -- so 2010.
Sorry this page does not exist =(
https://cc.bingj.com/cache.aspx?d=4652446581392&w=-V-8V9bl07...
Kagi is great but more options would be good too.
OP's product is clearly at a very early stage. OP's post is also pretty opinionated.
Hard to say which impact on product it will have - but as long we have more options for search engines, this will be one out of many options.
if you need a job or financial aid kindly contact us now via email : shalomagency247@outlook.com
Thanks.
It's also easy to read this as "interviewer hand-held a candidate through a problem".
There's nothing in there about the candidate.
Anyway, I don't think this is the kind of attention the author would want. Nobody even talked about the search engine yet.
For instance, if you search for 'Trump', the top links are
```
1. http://www.trump.de — found via Mwmbl -- Trump
2. https://itep.org/md/ — found via Mwmbl -- Trump Tax Proposals Would Provide Richest One Percent in Maryland with 69.7 Percent of the State’s Tax Cuts Earlier this year, the Trump administration r…
3. https://is.gd/mUHYTg — found via Mwmbl --- Trump embraces QAnon conspiracy because ‘they like me’ After skirting the issue for weeks, President Donald Trump offered an embrace Wednesday of the fri…
4. http://dict.cn/trump — found via Mwmbl -- trump是什么意思_trump在线翻译_英语_读音_用法_例句_海词词典
```
Surely there are millions of results more relevant to the phrase 'Trump' than trump.de. The other links aren't better. A random article from 2017? Another one from 2020. A Chinese dictionary definition of 'Trump'?
I get that search is hard, but what's going on here? You can try any phrase, and you just get weird results.
I'm wondering the same thing. Google gives me _exactly_ what I want without me having to add keywords or cajole it. All of these other search engines give me such weird irrelevant results. If I search "python reverse string" on YaCy's demo peer, the third result is the ArchWiki page on ... MATLAB.
I really wish I knew what to do to help the situation here because distributed p2p search engines seem so cool. But then again, Google wouldn't be so dominant if it were so easy.
Well, if you really want to know, you could try taking the HTTP responses for the page you expect to be highly ranked, and the page that's actually highly-ranked, and applying various common ranking heuristics to them, to figure out what the result-ranking algorithm is actually doing.
For any search engine who hasn't had a bunch of competitive pressure forcing them to improve, the ranking algorithm is very likely something incredibly simple and standard — e.g. tf-idf across the whole HTTP-result corpus.
So I'd guess that the results you tend to see in your tests, are because one of those "standard" algorithms ends up doing something dumb for the ranking pairs you care about.
Naive / biased statements such of these cause me to lend less credence to author's other points.
The officially stated goal of Hamas is the destruction of Israel.
War is never pleasant or desirable. That said, Israel has every right to defend itself and to fight for the return of its people.
Your position is that Israel should lay down and die. Any attempt to do otherwise is then disingenuously labeled as “genocide.”
The argument you are making is ridiculous. Fortunately, most people recognize that.
They’ve killed innocent civilians[2], health workers[3], and journalists[4] all in record numbers. They have not shied away from war crimes and humanitarian abuses [5].
Israel has every right to defend itself, but this has gone so far beyond that. There is no action that justifies this brutal of a reaction.
[1] https://www.reuters.com/world/middle-east/israeli-troops-kil...
[2] https://www.hrw.org/news/2024/04/04/gaza-israeli-strike-kill...
[3] https://www.doctorswithoutborders.org/latest/seven-months-re...
[4] https://cpj.org/2024/07/journalist-casualties-in-the-israel-...
[5] https://www.amnesty.org/en/location/middle-east-and-north-af...
If they are performing their duties adequately, then that's absolutely wrong.
It's also utterly stupid. Employees who think about what they are doing are far, far more valuable than mindless drones.
Yeah, but you still reserve the right to not crawl sites (or to remove them from your index), yes? So there's still the opportunity to do evil.
I'm still waiting for a "raw" search spidering provider. One that:
1. runs a web-spidering cluster — one that's only smart enough to know what robots.txt is, to know how to follow links in HTML pages, and to obey response caching-policy headers;
2. captures the spidering process losslessly, as e.g. HAR transcript files;
3. packs those HAR transcript files, a few million at a time, into tar.xz.tar files (i.e. grab a "chunk" of N HAR files; group them into subdirs by request Host header; archive each subdir, and compress those archives independently; then archive all the compressed archives without compression) — and then uploads these semi-random-access archives to a CDN or private BitTorrent tracker (or any other data delivery system that enables clients to only retrieve the blocks/byte-ranges of files they're interested in);
4. generate a TOC for the semi-random-access files, as a stream of tuples (signed archive URL, chunk byte-range, hostname, compressed URL-list); push these to a managed reliable message queue on an IaaS, publishing each entry to both an all-hostnames topic, and a per-hostname topic. (I say an IaaS, as this allows consumers to set up their own consumer-groups on these topics within their own IaaS project, and then pay the costs of message retention in these consumer-groups themselves.)
5. Also buffer these TOC-entry streams into files (e.g. Parquet files), one archive series per topic; and host these alongside the HAR archives. Prune TOC topic stream entries if (entries are at least N days old AND the entries have been successfully "offlined" into a hosted TOC-stream archive.)
---
This "web-spidering-firehose data-lake as-a-Service" architecture, would enable pretty much anyone to build whatever arbitrary search index they want downstream of it, containing as much or as little of the web as they want — where each consumer only needs to do as much work as is required to fetch and parse the HARs of the domains they've decided they care about indexing something under.
This architecture would also be "temporal" (akin to a temporal RDBMS table) — as a consumer of this service, you wouldn't see "the current version" of a scraped URL, but rather all previous attempts to scrape that URL, and what happened each time. (This would mean that no website could ever censor the dataset retroactively by adding a robots.txt "Disallow *" after scrapes have already happened. Their robots.txt config would prevent further scraping, but previous scraping would be retained.)
And in fact, in this architecture, the HTTP interaction to retrieve /robots.txt for a domain, would produce a HAR transcript that would get archived like any other. Domains restricted from crawling by robots.txt, would still get regular HAR transcripts recorded of the result of checking that their /robots.txt still restricts crawling. (Reducing over these /robots.txt HAR transcripts is how a consumer-indexer would determine whether they should currently be showing/hiding a domain in their built index.)
I'm not sure you would like the results of what you suggest - if you are really going to crawl everything indiscriminately, you will end up with a lot of rubbish. Just check out Common Crawl if you want to get an idea of what it would look like.
It has a lot of rubbish, sure, but the reason that that matters with Common Crawl, is that Common Crawl isn't a continuous stream; it's rather a big monthly 100TB incremental deliverable, that makes up part of an even larger multi-petabyte whole dataset; where "using the Common Crawl dataset" mostly means relying on one of a few IaaS providers who've grabbed the whole thing and unpacked it into their serverless-data-warehouse cluster that you can run map-reduce jobs against.
A given consumer of this hypothetical web-scraping-results "firehose via a data lake" API, meanwhile, wouldn't need to drink from the entire firehose in order to "follow" live data. For many purposes, they could instead just drink from the much-lower-pressure URLs queue, to discover what has been scraped; and then schedule fetching just those things [or rather, the domain-and-time-bucketed archive-chunks that contain just those things].
Which, for many consumers, might end up a low-bandwidth-enough affair that the data could be delivered to them over the regular public Internet, without needing them to "move compute to data."
Consumers might still need a copy of the entire dataset to backfill their indexing system initially — and this might still require doing the "colocate to the IaaS cluster where the dataset is, and run a map-reduce job" thing — but that'd be a one-time bootstrapping process, not a periodic job that needs to be reliable.
(In fact, since it's so rare, the scraping-service provider could even take responsibility for running these jobs themselves, as a sort of single-shot PaaS. "Subscribe to the firehose and we'll help you to do a one-time map-reduce over our dataset to backfill your index, all costs on us. Just define a job using this here SDK and upload it to our dashboard; it'll be queued to run on our infra; and when it's done, you'll get emailed a link to an object-store snapshot of the ephemeral data warehouse the job populated.")
---
Also, to be clear, I wasn't intending to describe an infrastructure whose output is directly able to be used as the index of a search engine. It'd be quite useless for that, just as Common Crawl would be. Such a dataset still needs curation.
It's just that, as with Common Crawl, the curation step should rightfully be the (direct, B2B) consumer's responsibility — because there are many different use-cases such data can be put to, that require different curation strategies:
• general whole-web search engines (obviously)
• site-specific search engines
• vertical-specific search engines (think: Google Scholar; FrogFind)
• format-specific global aggregators (e.g. a PubSubHubbub gateway that pre-discovers RSS feeds; a Matrix server that discovers and suggests other Matrix servers; or that old idea of an "Internet Yellow Pages" built out of people's VCard-RDF-microformat contact data embedded in XHTML — but now extended to the proprietary pseudo-microformats of various "about me" landing-page services)
• "see previous versions" services like the Wayback Machine (taking advantage of the immutability of the historical HARs in the data stream)
• a Shodan-like deep-web "discover what doesn't want to be discovered" service, surfacing websites with Disallow * robots.txt rules
• web analytics (like you can do with Common Crawl, but live, using scalable OLTP methods)
• continuous updating of ML models with "up to date" knowledge of the world (at least, once we figure out how to continuously train ML models)
There's really a lot you can do with what's essentially a periodic high-level packet dump of the result of poking every URL you can find, as often as it is willing to let you.
You should try and avoid competition but taking on a failing monopoly like Google is a great thing to try and strive for.