Hacker News Reader

Comment by samat | Hacker News Reader

One important implication is that collateral freedom techniques [1] using Amazon S3 will no longer work.

To put it simply, right now I could put some stuff not liked by Russian or Chinese government (maybe entire website) and give a direct s3 link to https:// s3 .amazonaws.com/mywebsite/index.html. Because it's https — there is no way man in the middle knows what people read on s3.amazonaws.com. With this change — dictators see my domain name and block requests to it right away.

I don't know if they did it on purpose or just forgot about those who are less fortunate in regards to access to information, but this is a sad development.

This censorship circumvention technique is actively used in the wild and loosing Amazon is no good.

1 https://en.wikipedia.org/wiki/Collateral_freedom

View on HN

If there is anyone from Amazon caring about freedom of speech and censorship — please contact me at s@samat.me, I'd love to give you more perspective on this.

Hey Samat, I am pretty sure that AWS knows exactly what they're doing. They don't want to lose money by hosting objectionable content, and then lose customers to Aliyun or Russian cloud providers.

Does Aliyun support path-based file storage? That could be handy!

Not sure but what I heard their API is the exact copy or the S3 API, so you can switch from one to other without any effort.

I tested that when writing the S3 implementation of a Go key-value wrapper [1] and back then "Alibaba Cloud Object Storage Service (OSS)" did not support path-style addressing.

If you're looking for a similarly robust and scalable alternative, Google Cloud Storage is accessible via S3 API when enabling that in the bucket's configuration and it supports path-style access (at least back when I tested the different S3-compatible services).

[1] https://github.com/philippgille/gokv

They did it not to make blocking in Russia and China easier, but to make their deployment cheaper and faster. Basically with v2 protocol your TCP packets go straight to the server where data is stored without going through one giant proxy. In another words, they do IP routing now instead of HTTP proxying.

https://en.wikipedia.org/wiki/Domain_fronting#Disabling Interestingly a different although related trick (that of domain fronting) has been blocked last year "by both Google and Amazon.... in part due to pressure from the Russian government over Telegram domain fronting activity using both of the cloud providers' services."

Follow the money :/

The Russian market is tiny, so that logic leads me to the country south-east of Russia. The one with all the new consumers.

This is an interesting perspective.

Just as a counter argument, one of the things we tried to do at a previous employer was data exfiltration protection. This meant using outbound proxies from our networks to reach pre-approved urls and we don't want to mitm the TLS connections. This leaves a bit of a problem, because we don't want to whitelist all of s3, the defeats the purpose, so we had to mandate using the bucket.s3 uri style, which is a bit of a pain for clients that use the direct s3 link style, but then we could whitelist buckets we control.

I don't want to say this use case is more important, but I can see the merits of standardizing on the subdomain style, and that this might be a common ask of amazon.

Exfiltration protection is pointless. It's a great way to waste money and annoy your employees.

This is not a helpful comment. Could you provide some examples of why it's a waste of money?

When there is a will, there is a way.

Any output device is an output device. A VGA interface. An HDMI interface. A Scroll lock keyboard light. A hard drive interface. A speaker. All you need to do is send the signal down one wire and you could tap into that wire and copy all the data to another system.

Copying files from one folder into another could do the job.

> ... A scroll lock keyboard light ...

Oh, this was a fascinating read! Thanks for encouraging my perusal.

http://staff.ustc.edu.cn/~zhangwm/Paper/2018_10.pdf

I brought it up because I had heard that the original iPODs ROM was extracted using the "click" sound and a microphone. I can't find any reference to it now...

There is also research about using modem lights (even in the background of a room) to figure out what people are doing on their dialup internet connection. Those RX and TX LEDs are actually blinking at your data transmission rate.

Here's the iPod story: http://www.ipodlinux.org/stories/piezo/

Generalising further, signal, channel, reciever.

Not the commenter, but I think the point is that there are simply too many ways to bypass protections. If you want someone to be able to view data, it is impossible to prevent them from exfiltrating it. In many ways it is similar to the analog hole problem with DRM.

You can make it harder to do on accident, or to prevent someone from doing it for convenience (e.g. someone copying data to an insecure location to have easier access to do their job), but you can't stop a malicious actor from getting the data.

They could tunnel over DNS, they could use a camera phone and record the data on the screen, etc. The possibilities are endless.

https://en.wikipedia.org/wiki/Analog_hole

> You can make it harder to do on accident

That's a major point of exfiltration prevention, both because accidents are a real problem and because reducing the opportunity for accidents makes it easier to establish that intentional exfiltration is intentional, which makes the ability to impose serious consequences for it greater (especially against privilege insiders with key contractual benefits that can only be taken away for cause.)

Technical safeguards aren't standalone, they integrate with social safeguards.

>> You can make it harder to do on accident >That's a major point of exfiltration prevention

But that’s not how it is advertised. Usually they claim is to catch hackers and mal intending employees.

Advertising is 100% bullshit, both in this space and elsewhere. I'm not sure why even well-paid people in positions of corporate power don't seem to get it. It's similar to Gell-Mann amnesia.

It's hard to prevent data exfiltration by an attacker with physical access to your premises, but it's much easier to prevent exfiltration (especially of large quantities of data) by compromised devices.

I read a great paper talking about this problem once, which made the point that the only actual measure of data security is the bandwidth of possible side-channels vs useful size of the data.

Do you happen to have any pointers about this? I would be very interested.

The parent was probably talking about this paper: https://csyhua.github.io/csyhua/hua-ipdps2018.pdf

Here are a few more (somewhat) related to this topic:

1.Joe Grand, “Advanced Hardware Hacking Techniques”, Defcon 12 http://www.grandideastudio.com/files/security/hardware/advan...

2.Josh Jaffe, “Differential Power Analysis”, Summer School on Cryptographic Hardware http://www.dice.ucl.ac.be/crypto/ecrypt-scard/jaffe.pdfhttp:...

3.S. Mangard, E. Oswald, T. Popp, “Power Analysis Attacks -Revealing the Secrets of Smartcards” http://www.dpabook.org/

4.Dan J. Bernstein, ''Cache-timing attacks on AES'', http://cr.yp.to/papers.html#cachetiming, 2005.

5.D. Brumley, D. Boneh, “Remote Timing Attacks are Practical” http://crypto.stanford.edu/~dabo/papers/ssl-timing.pdf

6.P. Kocher, "Design and Validation Strategies for Obtaining Assurance in Countermeasures to Power Analysis and Related Attacks", NIST Physical Security Testing Workshop -Honolulu, Sept. 26, 2005 http://csrc.nist.gov/cryptval/physec/papers/physecpaper09.pd...

7.E. Oswald, K. Schramm, “An Efficient Masking Scheme for AES Software Implementations” www.iaik.tugraz.at/research/sca-lab/publications/pdf/Oswald2006AnEfficientMasking.pdf

8.Cryptography Research, Inc. Patents and Licensing http://www.cryptography.com/technology/dpa/licensing.html

Thank you very much for the material!

> You can make it harder to do on accident, or to prevent someone from doing it for convenience

Why are you discounting these as valuable use cases?

In my experience, they're far more common than the determinedly-malicious actor. And they're far, FAR more common than the malicious actor who also (a) knows that the exfil monitor is there, and (b) has the technical prowess to circumvent it. (The analog hole is only _trivially_ usable for certain kinds of data.)

(I am continually frustrated by the number of people who claim that protection is worthless if it's potentially circumventable. In most situations, covering 90% of attacks is still worthwhile.)

> I am continually frustrated by the number of people who claim that protection is worthless if it's potentially circumventable. In most situations, covering 90% of attacks is still worthwhile

People are saying that because it's misguided and potentially harmful.

Doing so is security theater, where the solution is scoped down to something incomplete but easier, and then everyone walks away happy they solved 90% of the smaller problem they chose to attempt.

Particularly with things like data exfiltration, this is potentially harmful because then you've organizationally blinded yourself.

Nobody wants to poke holes in their own solution, and so they stop looking.

But, hey, we're catching the odd employee accidentally sharing confidential documents via OneDrive.

Fast forward a year, and an entire DB gets transferred out via an unknown vector, nobody finds out about it for a couple months, and it's all "Oh! How did this happen? We had monitoring in place."

Go big, or run the risk of putting blinders on yourself.

The way I'm reading this, your attitude seems to be "if you can't stop targeted nation state actions you might as well not bother with network security and just run unencrypted wifi everywhere."

Network security is a balancing act between prevention, detection, needed user and network capabilities and cost. If I have unlimited money or no limitations on hindering network usage I can make a 100% secure network - it's not even that expensive, just unplug it all.

>covering 90% of attacks

It doesn't cover most attacks. That's why it's so misleading. It mainly just protection against incompetent people from accidentally sending out data.

Which is what like 80% of data leaks?

I agree. Locking my front door is trivially circumventable. It is pretty easy to pick up a rock and break a window. Or use a heavy instrument to break down the door. Or heck, a car could just go crashing through the wall. But that lock is a pretty good deterrent from casual abuse. It requires crossing a psychological barrier into the explicitly illegal and malicious realm.

That's a sensible analogy on the surface, but the difference is that having to lock the door doesn't

- end up forcing you to lock it from the inside, and then crawl out the window

- have your friend who is visiting request a door-opening-token 24hr in advance through a JIRA ticket

- cause the power to go out once it's locked, also for security reasons

- force you to replace the keys with 'special' plastic ones from a new third-party vendor

- leave you stranded outside for a few hours because the door-opening system is having an outage

Those are the kind of trade-offs that will be made, not simply the act of locking the door.

It's also the reason why people mail sensitive data to their private mailboxes and that no matter how hard some companies try, they can't make their employees stop using Excel as their primary work tool.

'arcbyte over at https://news.ycombinator.com/item?id=19827012 does have a point - a lot of potential exfil risk is caused by companies doing their best to make it as difficult as possible for their employees to do their jobs.

> there are simply too many ways to bypass protections

This is not a good general principle, since it can be easily applied in contexts that (I would predict) many same individuals would vehemently disagree with. For example:

- Personal privacy is pointless, there are simply too many ways for governments/corporations/fellow citizens to find things out about you

- Strong taxation enforcement by governments is pointless, there are simply too many avenues for legal tax avoidance and illegal tax evasion

- Nuclear arms control is pointless, the knowledge of how to make a bomb and enrich uranium is widely available (I mean, if NK could pull it off, how hard could it be?)

Maybe data exfiltration prevention isn't a good policy, but I think you need a more nuanced argument than 'there are ways around it'.

All these things work statistically, preventing a certain part of incidents.

They are not a guarantee that an incident cannot happen, though. They can only lower the rate at which incidents (privacy violations, tax evasion, nuclear proliferation) occur.

Same with exfiltration.

Within hours of setting up DLP, I had someone complaining I had broken their workflow. That workflow apparently involved emailing credit card numbers to an external personal mailbox "for security".

It was allowed to go on simply because noone knew about it. Could a skilled attacker spread a card card number across three lines and get past the system? Absolutely. Is exfiltration protection pointless? Absolutely not. Once you scale a certain number of users, you'll find someone somewhere that completely ignores training (which they did have) and decides they don't see the problem with something like this. And you won't know about it until you put a suitable system in.

This proves my point perfectly.

Instead of investing in i.proved tools and productivity for your workers so they dont have to do stupid shit you dont want them to do, you instead made it harder for everyone to do their job.

I'm not saying your business is going to crash and burn. I'm saying you will never be as successful as you could have been. You're literally wasting resources, leaving needs unfulfilled, and giving up ground to your competitors.

Wasting resources by preventing people from sending CC# in clear text to a personal email? That person is lucky they didn’t get fired for doing it

since your employer is willing to invest in this, wouldn't a custom proxy solve your problem? just whitelist the s3 buckets you care about, and have people access s3 through the proxy.

This is based on using a http proxy, it's just the proxy whitelists the domains to connect towards. As for a proxy that can mitm tls connections, I'm not a big fan of this approach, as you ned to add a cert into the trust store of all the machines, which if compromised tends to be a bad day across the board.

But that isn't really a counter argument.. if you provide the ability to use both formats, your use case would still work (only provide access to the custom subdomain you control).

you're correct, it's just my experience with this is that certain libraries expected the url format we couldn't accept to be working, and didn't provide an alternative. So having more flexibility in the api can work against you at times is all.

Google Reader served a similar purpose. People used its social features for communication since (the thinking went) governments weren't going to block Google.

Teenagers use Google Docs to chat in environments where popular IM applications are blocked, such as schools and libraries.

It's not really the same threat model as people living under dictatorships, but it might just work.

Now that you mention it…

[√] absolute dependence on authorities for food, shelter, clothing, transportation, money.

[√] curfews often in effect for you and your social circle, especially if suspected of deviance.

[√] 24/7 electronic or in-person monitoring is possible and largely accepted.

[√] social circle often molded by authorities.

[√] not allowed to vote or generally exercise political agency (and when allowed it's dismissed).

[√] not allowed to leave your workplace or home without permission from authorities.

[√] possible to flee and seek asylum but it means leaving everything behind for an uncertain future.

[√] indoctrination is so effective you're extremely likely to continue the system when allowed to be an authority.

Good thing it's a benevolent regime.

You're neglecting a key point: primary and secondary education are the province of legal minors. Full legal rights of majors do not apply.

Not that there aren't problems with both P/S education and higher education or public discoure and media generally, though your analysis misses a few key salient aspects and presents numerous red herrings.

J.S. Mill affords a longer view you may appreciate:

https://old.reddit.com/r/dredmorbius/comments/6x7u6a/on_the_...

IMO the even bigger problem is that this literally breaks HTTPS.

AWS S3 will only provide SSL validation if your bucket name happens to not contain "."

Which is a practice encouraged by AWS. [1]

So anyone that has www.example.com as the bucket name can no longer use HTTPS.

[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/website-host...

We have exactly this problem. I would appreciate if somebody explained how we should fix this. We need HTTPS and we have buckets with dots in their names

A simple answer might be "time to move to another static hosting solution".

If static hosting is the purpose, just put cloud front in front if it.

s3 isn't just used for static hosting. And if you have terabytes of data in a bucket that happens to have a dot in it (that may have been created a long time ago). Your options appear to be not using https, or spending a _lot_ of time and money moving to a new bucket or a different storage system. It seems to me that if Amazon is going to do this, they should at least provide a way to rename buckets without having to copy all of the objects.

It obviously depends on how many files we are talking about but copying files to a new bucket in the same region will not cost that much. You could definitely make the case to AWS that you don't want to pay since they are removing a feature and you might get a concession.

$0.005 / 1,000 copy requests...

ref: https://blog.cloudability.com/aws-s3-understanding-cloud-sto...

Also you will likely want to use some sort of parallel operation. I used this eons ago: https://github.com/mishudark/s3-parallel-put

> Your options appear to be not using https, or spending a _lot_ of time and money moving to a new bucket or a different storage system.

The only way it would be a lot of money to move to a new bucket is if the bucket is hardcoded everywhere. Moving data from one bucket to another is not expensive, and a configuration change to a referenced URL should be cheap, too.

You have until September 2020 to fix it?

More seriously; it's going to be a nasty migration for anyone who needs to get rid of the dots in their domains. At minimum you need to create a new bucket, migrate all of your data and permissions, migrate all references to the bucket, ...

I mean, if Amazon wanted to create a jobs program for developers across the world, this wouldn't be a bad plan. :(

They do say "We recommend that you do not use periods (".") in bucket names when using virtual hosted–style buckets." in the "bucket restrictions": https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestri...

Still, seems kinda lazy from their part, they could just generate a custom cert.

Did Amazon invent the rule the cert wildcards only match one level?

I don't believe they did, no.

Collateral freedom doesn't work in China. China has already blocked or throttled (hard to tell which, since GFW doesn't announce it) connections to AWS S3 for years.

S3 is hardly the only example of collateral freedom in China. There are many other cases where the concept works.

One counter example is GreatFire, who use GitHub for their wiki [1].

[1] https://github.com/greatfire/wiki

Funny you mention this one particularly - they didn't really like it [1].

Thinking about it the other way round, how likely would have Amazon been the target of similar attacks?

1: https://arstechnica.com/information-technology/2015/04/meet-...

Pretty sure Signal and Psiphon use it successfully. Yes it's throttled, but it's usable most of the time.

They probably got an ultimatum from Chinese authorities to either stop allowing this or get blocked entirely.

They just blocked wikipedia last week.. no one is too big to get shutdown in China

Very few people use Signal or Psiphon in China just FYI

Definitely worth checking out https://ipfs.io/. Even for those who don't or can't run IPFS peers on their own devices, IPFS gateways can fill much of the same purpose you listed above. Additionally, the same content should be viewable through _any_ gateway. Meaning if a Gateway provider ever amazoned you, you simply make the requests through a new gateway.

Yes, but restrictive governments will have no problem with blocking access to the ipfs.io domain via DNS and by blocking its IP addresses, whereas using the same method for blocking all access to AWS or google cloud is too costly as it will result in collateral damage at home. (Well China can block access to AWS located outside of China because there are AWS Regions in China)

With ipfs anyone can operate an http relay to access the network from any arbitrary IP and/or distribute endpoint IPs to populate the daemons dht if run locally.

Use DNS over HTTP. Firefox is very easily configurable (network.trr.bootstrapAddress, network.trr.mode, etc) so that if you pick the right bootstrap provider and DNS over HTTP provider you'll never send an unencrypted DNS query (including no SNIs) and it will fail completely rather than reverting to your OS's DNS Client if it cannot be resolved via the DNS over HTTP channel you define.

Because the S3 buckets are virtual-hosted they share IPs so there is deniability if you can hide the DNS/SNI.

Yes but https SNI still exists.

TLS v1.3 finally addresses this, https://blog.cloudflare.com/encrypted-sni/

I expect this will only work until the government in question is sufficiently angered that they just outright block the entire AWS infrastructure. Or whoever else supports ESNI.

But the only reason domain fronting works in the first place is because people think that large web hosting providers are too large to block.

If a hypothetical tyrantical government was willing to block all of Amazon S3 this change doesn't affect anything.

If it impacted Amazon’s (or whoever is targeted) bottom line then I would expect they would be open to dropping domain fronting support. But I admit I don’t know this for sure - time will tell.

China has blocked GitHub and Akamai before. https://www.latimes.com/business/technology/la-fi-tn-great-f...

But the alternative, which is to just not force this virtual host change in the first place, similarly might have gotten AWS blocked from those countries anyway

We need this in Chrome, badly.

TLS 1.3 was completed, published as RFC 8446 but eSNI is still a work in progress.

You need TLS 1.3 because in prior versions the certificate is transmitted plaintext, but eSNI itself is not part of TLS 1.3 and is still actively being worked on as https://datatracker.ietf.org/doc/draft-ietf-tls-esni/

This isn't a general-case solution (because you can no longer just give someone a link), but, can't you send "s3.amazonaws.com" or really any other bucket name in the SNI and give the full bucket name in the Host header inside the encrypted channel? Or does S3 block SNI/Host mismatches?

They will possibly block mismatches (I believe they do for cloudfront etc. now), but also if the point of moving to sub domains is sharding, there's no guarantee the bucket you want is behind the faked hostname you connected to.

This currently works, but considering their crackdown on domain fronting last year, I don't expect it to work for much longer.

Did you mean use DNS over HTTPS?

This is similar to domain fronting, which many providers are no longer allowing either.

Would encrypted SNI fix this?

[1] https://blog.cloudflare.com/encrypted-sni/

Yes, nothing man in the middle can do to detect the final domain being connected to.

"collateral freedom" is a failed concept. Many years ago people use Gmail to communicate, and they argue that Chinese government won't dare to block such important and neutral service.

There are like <1% websites in China relies on S3 to deliver static files. Blocking AWS as a whole has happened before. There is simply no freedom was "collateral". Freedom has to be fought hard and eared.

Use Cloudflare or any free CDN service?

Edit: Why am I getting downvoted, it's a legit answer, CDN hides your origin.

Cloudflare is just a disaster for bypassing government censorship. If some website is blocked in my country and I try to access it with Tor or VPN then I better hope it is not behind Cloudflare, because Cloudflare just gives me endless Google captcha instead of the desired website.

That assumes the website actually wants to block abusive traffic. A freedom site won’t.

It's captchas from cloudflare because their country is routinely used for malicious activity.

They do have a Captchas Effectively Off setting, but you are right that it still could trigger: https://support.cloudflare.com/hc/en-us/articles/200170096-H...

I think this is exactly what happened.

"right now I could put some stuff not liked by Russian or Chinese government (maybe entire website) and give a direct s3 link to https:// s3 .amazonaws.com/mywebsite/index.html. Because it's https — there is no way man in the middle knows what people read on s3.amazonaws.com."

Chinese government will just ban the whole s3.amazonaws.com domain. Same as facebook.com, youtube.com, google.com, gmail.com, wikipedia...

However letting them banning sub-domains will actually make S3 a useable service in China. It's a huge step forward.

You could say the same about Dragonfly.

You cannot solve a political problem with a technical solution.

It's better to approach issues from all sides.

You are correct, but you forget that a technological solution like this one can help bringing around the actual social change you are looking for. It is kinda difficult to bring a social change when your major communication and information distribution methods are gutted.

That's true, but technology can be part of a political solution.

In particular, a political solution requires that people be able to communicate (in order to work together), and technology can be a component of that.

Sure you can. Weapons research is a very common counterexample; people have been solving political problems with technical solutions ranging from sharpening spear-heads to achieving nuclear chain reactions.

(Of course "who is politically right" and "who has the most technical expertise on their side" are at best tenuously related, but that's a different and longstanding problem. If you believe you're politically right and you have technical expertise on your side, use it.)

Are there any non-violent technical solutions? I think we all know that's what that person really meant.

Radio Free Europe is a technical, non-violent solution to a political problem.

Viagra solved tiger poaching political problem.

Encryption, medicine, transportation, food production etc.

Using encryption to hide from a repressive regime may just make you a target.

Is there anything lost if you've already felt that you're a target?

Isn't there everything to gain from encouraging everyone to use encryption so that there are too many targets to process?

Encryption also solves the political problem of "I need to communicate plans with my allies across long communications links without my enemies knowing what they are," which can often be opposed to violence (notify people of an opposing military action, evacuate armies or civilians, coordinate a plan to surrender without showing weakness, coordinate a plan to demand the other side surrender by showing so much strength they won't fight, etc.). Much early encryption research was for governments who were already targets to hide data from other governments.

If encryption is used by most people you can't use it to identify suspicious activity.

Electronic voting machines. Digital signatures on passports. Long-distance communication, whether by telegraph, radio, or satellite. Norman Borlaug's wheat hack. Machine translation. The counting machines that powered the Holocaust (as mentioned, something being a political problem solvable by technical means does not mean it should be solved). Forensic analysis of DNA and fingerprints. Eurovision. Irrigation. Aqueducts. The printing press. I feel like there are many things....

Electronic voting isn't a solution, it's an attack. https://www.youtube.com/watch?v=w3_0x6oaDmI

Bitcoin

What was the Manhattan Project?

Reply on news.ycombinator.com