Hacker News Reader

Comment by dspillett | Hacker News Reader

You might want to rate limit that if you don't magically have free unlimited bandwidth quota.

I hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.

While I'm in general not a fan of wildcard certificates, yours doesn't work for the bare domain: https://i.imgur.com/uKVK5xd.png

I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.

Hmm, curious behavior. I'm not getting that in any of the browsers I can run. Maybe some safari-specific interpretation?

I don't trust letsencrypt and I don't want to give them or anyone else a list of which subdomains I use.

I can at least reproduce it in Chrome, for what it's worth. Perhaps you have the redirect to www cached?

On an Apple or another OS?

I can reproduce it even just with curl on Linux.

The certificate is only for "*.marginalia.nu", which simply doesn't cover "marginalia.nu". It should give the same error on any platform and browser, unless their SSL implementation is broken.

Some browsers try to be smart and insert www automatically though.

I would recommend getting certificate which includes root domain and wildcard subdomains to cover all bases.

The cert I am seeing [1] does include both the wildcard subdomain AND the apex domain

[1] https://crt.sh/?id=6125359537

That's not the certificate I get served though (this one is from 5 february, the one served is from 25 january) do you actually get this certificate on https://marginalia.nu?

I misunderstood what URL we were talking about. What I am seeing:

https://marginalia.nu = https://crt.sh/?id=6046506678 (includes wildcard but NOT apex)

https://search.marginalia.nu = https://crt.sh/?id=6125359537 (includes both wildcard and apex)

Ah, I see what's happening. search.marginalia.nu goes through cloudflare which does its creepy MITM thing and re-encrypts on its end; the others do not.

Why does marginalia.nu use Cloudflare? Have you been hit by DDOSs? I can't think how a search engine would benefit from Cloudflare caching things.

Yeah, It's 100% about bot mitigation. I got hit pretty bad by some botnet a while back, saw 10-15 searches/second. Can't have nice things on the Internet I guess.

Don't really understand their motive either. Maybe they thought I was cloud hosted or used some expensive API to do searches and were attempting to rack up big bills or something.

:-|

I wish I had a good solution to this. Cloudflare to mitigate DDOS attacks has somewhat of a “baby with the bathwater” vibe (considering how much of a pain it is if you can't pass the CAPTCHA, or if you're on Tor), but I can't think of an alternative.

:-| indeed.

I've considered having a naked endpoint with a rate limiter that, when it hits some ceiling (dunno, sustained load of 2 RPS or so), offers the alternatives of going through an unlimited cloudflared domain, or waiting until the bots give up. But that might be annoying too.

You could try to push forward the frontier of accessible CAPTCHAs. You just need some tasks that basically all humans can do, but barely and bots can, that don't rely on vision, hearing, emotional response or cultural knowledge. Should be easy, right?

It's simple, your cert points to *.domain.tld, domain.tld doesn't have a subdomain, <anything>.domain.tld does. You should expand it to include both domain.tld and *.domain.tld.

It seems like a bad cert is being served at the redirect; the LE cert at https://search.marginalia.nu DOES have both the wildcard and apex domain

See https://news.ycombinator.com/item?id=30301015

> .domain.tld

Oof, I'm not sure if I would do that. Since "" stands for all sub domains.

There may be some type of caching problem. When I visit https://marginalia.nu directly, I'm served with this cert, which does not include the apex domain: https://crt.sh/?id=6046506678

When I visit https://search.marginalia.nu I'm served with this different cert, which does include both wildcard and apex: https://crt.sh/?id=6125359537

Free unmetered bandwidth is pretty much the norm unless you have fallen for some “cloud” scam.

edit: s/unlimited/unmetered/

Both old-school webhosting and modern cloud hosting commonly have data caps. Unlimited data is common only on mid-to-high-end VPS or dedicated servers and unlimited bandwidth doesn't really exist.

>unlimited bandwidth doesn't really exist.

True, when there is no deliberate cap stated, it's unmetered, not unlimited, meaning you can use what however much fits through the pipes (usually best effort instead of guaranteed bandwidth), and that's the natural limit. A 1 Gbit/s link for example will get you close to 300 TiB in each direction in a 30 day month if you are able to saturate that link 24/7, while a 100MBit/s link will get you a tenth so "only" close to 30TiB.

Free unlimited bandwidth always has a catch somewhere, that this use could trip on, unless the service is fairly expensive.

There is often a “fair use policy”, and even without anything like that you need to consider the potential effect on the stuff you are actually running the service for (while a search engine is pulling random data as fast as they can, that transfer is competing against other network load). I have a couple of inexpensive hosted servers with genuinely unlimited bandwidth, but only at 100 and 250mbps respectively, better rates cost a fair chunk more without some sort of cap (usually of the form “after some-TB-or-some-tens-of-TB we'll throttle you to 100mpbs until your next billing month”.

Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable.

> Unlimited throughput on gbit/more lines is commonly available, but never both cheap and reliable

It’s been a long time since you could call OVH “unreliable”, Hetzner chugs along just fine too.

I think you might just be stuck in 2004, reliable unmetered gbit+ has been cheap for years.

I think the concern here is about congested peerings to certain providers, not uptime.

For example Deutsche Telekom: They don't accept free direct peerings. Sending data to them indirectly frequently runs into congestion issues. For a while Hetzner offered a paid option for better connectivity to DTAG. It looks like Hetzner added a direct peering in 2020.

https://web.archive.org/web/20200205014018/https://wiki.hetz...

https://www.hetzner.com/news/03-20-dtag/

> reliable unmetered gbit+ has been cheap for years

I suppose that depends on what you call cheap, and whether you include “guaranteed”/“dedicated” in the definition of reliable for bandwidth.

I'm looking at things mainly from a personal projects & packups PoV, but demanding at least two drives for RAID1+ rather than the cheapest units. If you have money-making projects then the lines between cheap/inexpensive/reasonable might move.

> OVH / Hetzner

I have Hetzner down as “inexpensive” rather than cheap, assuming you count the older hardware in their auction lines (with their new-kit offerings being “reasonable”).

OHV is cheap if you are considering their lower-spec offerings, but those are at limited rates (100mbit usually in the case of Kimsufi branded services, 250mbit for SYS) and IIRC that rate is not guaranteed (while not a massively oversold resource like you'll see in many VPS/shared/similar hosting arrangements, there are enough machines sharing a larger resource that if many try saturate their allocation at the same time they'll hit congestion even within the DC rather than just when your traffic touches external peering).

(Don't take the above as a criticism of Kimsufi/SYS - they are honest & open about the limits of the services they sell (older hardware, limited max network throughput) - I get what I pay for, which is no less than promised, and I've found them to be reliable over the years I've had services with them)

> I suppose that depends on what you call cheap, and whether you include “guaranteed”/“dedicated” in the definition of reliable for bandwidth.

Then there are no cheap options. Even frivolously expensive “cloud” providers don’t give you this.

The reality is that there are very few customers that actually have a real need for “guaranteed”/“dedicated” bandwidth.

This isn't physically possible. I think you're confusing (a) an ISP saying "we'll give you all the bandwidth we get (from our tier 1 networks / backbone links)" with (b) genuinely unlimited bandwidth, which would suppose a infinitely large carbon fibre cable subsuming the universe, or something along those lines.

(a) may not be that much, in reality, depending on how their network is set up, and how much other traffic there is at any given moment. It also depends where you're routing to, whether e.g. (i) your neighbour or (ii) a DC across the world (neither is clearly better, it all depends). You have to think of it through the lens of physics and information theory, rather than law and business models.

This is pointlessly pedantic. In this context the term is being used to refer to unmetered and uncapped bandwidth and that usage is well understood.

I'd agree if the upper bound of unmetered and uncapped bandwidth were so far in excess of the possible upper bound for this application that it were silly to consider - but, like I addressed in my comment, it's very much not.

If he were only talking about meters or caps, then that would therefore make his point a bit of a non sequitur in the context of an argument about whether tarpitting/blackholing a connection is risky for the serving party.

(FWIW, at past companies I've not-infrequently encountered network saturation even without an artificial cap from whomever we were peering with.)

Genuine unmetered/unlimited bandwidth claims also include a port speed. And the really good claims will include a 'minimum sustained' rate (with well connected peers), which lets you complain when you can't get decent throughput because all the servers on the 10G switch share a t1 out to the local internet exchange.

Please tell me what provider has free unlimited bandwidth.

I’ve heard this false notion so many times on HN and every time I check it out, the provider says clearly that they don’t allow for unlimited bandwidth.

So please tell me the provider. I’ll pay for a month of service for 10 Gbps unlimited service and then I’ll saturate that line up and down stream for an entire month.

And we’ll just see what happens, ok?

> Please tell me what provider has free unlimited bandwidth.

Unmetered rather than unlimited of course, but OVH, Online.net, Hetzner. These are rather well-known providers.

I haven't checked all of them for the exact language but Hetzner explicitly says "All dedicated root servers have unlimited traffic" for example.

Of course, they do have a limit for servers with 10 Gbps connections probably because of people like you who like to push the limits beyond what is reasonable.

Yep, those. Also FDCservers, Nforce, psychz, worldstream and countless of others.

I have extensive experience maxing out unmetered 10gbit lines with these providers 24/7. It’s been years and years since a host last tried to FUP me, bandwidth is just really cheap now.

Someone will always pull the plug when you hit the undocumented limit.

Big providers like OVH and Hetzner (and many others) offer truly unmetered bandwidth. You can max out your line nonstop for as long as you’d like.

Yep, the cheap dedicated servers I have are with Kimsufi and SoYouStart (both OVH brands), though the lines are limited to 100mbit and 250mbit respectively and IIRC neither is guaranteed (something like X servers sharing a gbit+ connection, where Xrate > TotalLineRate) so I can usually* but not always get 100mbit out of kimsufi. I've not really pushed the SYS machine's resources yet, it is a recent addition to my dominion.

A search engine pulling data from /dev/urandom as in the post that started this discussion is likely to impact other uses of the 100mbit link from Kimsufi. Worse if bad luck means multiple hits of that sort at the same time.

If you’re too small to get negotiated rates, you can still get kimsufi and SYS kit with guaranteed gbit from resellers.

Check out https://discord.com/invite/7Gv8tdM

There are really good deals available, all of these still come with a decent margin for the reseller.

  Core i3-2310 8GB RAM 2x2TB HDD 1Gbps Unmetered €16
  Core i5-2300 16GB RAM 2x2TB HDD 1Gbps Unmetered €24
  E3-1225 16GB RAM 2x2TB HDD 1Gbps Unmetered €35 (iGPU enabled)
  W3520 16GB RAM 2x2TB HDD 1Gbps Unmetered €40

It’s almost worth burning my terabyte quota on Linode to try this out :)

Reply on news.ycombinator.com