Reading the Silk Road configuration | Hacker News Reader

Reading the Silk Road configuration

blog.erratasec.com

247 pointsphilip1209153 comments

So - how to do it correctly? The onion server should not be able to access the internet directly and should never know it's own IP-address. So putting it inside a virtual machine/container and redirecting outgoing traffic through Tor (on another VM) should fix that.

As for the backend servers - if all machines are connected through OpenVPN with TLS-Auth they can use private IP space and are not accessible from outside. The OpenVPN tunnels are configured outside of the onion VM and Tor VM and the backend machines are also in a VM that pipes all outgoing traffic (except OpenVPN traffic) to the internet through Tor.

This setup should at least omit the problem of leaking public IP addresses on compromise or did I overlook something? Let's ignore VM outbreak exploits (ksplice/unattenend-upgrades should help here if it's not the NSA).

With TLS-Auth and a default DROP firewall rule all machines should not even appear in scans on the internet. So any outside contact to any machines is only possible if you know the OpenVPN TLS-Key + Certificates. An intruder would only see local IP addresses even if he manages to compromise a backend server. All created outgoing traffic to find the machines would be routed through Tor.

As the Tor onion VM is separate from the webserver only a Tor remote exploit or a VM outbreak exploit would be critical. Maybe encryption can help here.

Another point is using dedicated hardware with Full Disk Encryption and just enter the passphrase via the provider console. There are cold boot attacks but VPS servers allow to dump a memory image while running - that's not so easy with dedicated hardware.

Say you have your own hardware, what would you need to access the display and keyboard securely remotely over the network? Are there such devices available? I imagine you can have the OS disable its display and keyboard input after boot for added security, and lock down the boot process, but it still seems like it's hard to do this securely. I'm trying to think of a scenario where you can have:

1) Full Disk Encryption

2) Two physical machines owned by you, perhaps stored in some basement

3) Be able to boot them back on without physical access (perhaps this is simply a Bad Idea?)

4) One machine with two NICs running Tor, exposing only Tor to one of the NICs

5) The other machine running a VM host with 1 VM for each of your services. The host is connected to the NIC of the first machine, thus only has access to the internet through Tor

> Full Disk Encryption

[…]

> Be able to boot them back on without physical access (perhaps this is simply a Bad Idea?)

This is exactly the problem which our project solves:

http://www.recompile.se/mandos

Introduction:

http://www.recompile.se/mandos/man/intro.8mandos

Summary:

The computers run a small client program in the initial RAM disk environment which will communicate with a server over a network. All network communication is encrypted using TLS. The clients are identified by the server using an OpenPGP key; each client has one unique to it. The server sends the clients an encrypted password. The encrypted password is decrypted by the clients using the same OpenPGP key, and the password is then used to unlock the root file system, whereupon the computers can continue booting normally.

Mandos is pretty cool. Debian and Ubuntu can also embed dropbear into the initramfs so you can ssh in and unlock the disk.

Dropbear (SSH access) still requires you to manually log in and unlock the disks; a primary feature of Mandos is that it can boot completely unattended (unless configured otherwise).

Yup, I've used Mandos, just stating that there are other options if you don't want automatic.

Mandos can be configured to be not automatic by setting the “approved_by_default” option for a client to “False” in the clients.conf file; See mandos-clients.conf(5): (http://www.recompile.se/mandos/man/mandos-clients.conf.5).

You also need to adjust the “approval_delay” setting.

My chrome browser thinks your site isn't "trusted". Gogle's not a fan?

It’s just the CAcert certificate which doesn’t exist in Chrome. Either ignore it or install the CAcert root certificate from http://www.cacert.org/?id=3

"Say you have your own hardware, what would you need to access the display and keyboard securely remotely over the network?"

https://en.wikipedia.org/wiki/KVM_switch#KVM_over_IP_.28IPKV...

Thanks, that's what I was thinking of but I couldn't think of the name.

> The onion server should not be able to access the internet directly and should never know it's own IP-address.

I may be overly pedantic here but it should be "public IP address" as there is no TCP/IP without an IP address. The address may be in the private ranges, though.

Yes. I wrote that in a hurry. I'm surprised the comment is on the top. This was purely theoretical speculation. I don't run a hidden service.

If I were doing this, I would not use a VM for it. Running on bare metal has a significantly smaller (by several orders of magnitude) risk profile and is much simpler to do right. I'd have three boxes: webserver, Tor server, external exposure box. The webserver would connect solely to the Tor server, and would expose absolutely nothing other than HTTP. The Tor server would run the hidden service pieces and would expose nothing to either side, but would connect to the webserver and tunnel things accordingly, and would send its Tor connections through the external box. The external box would have a simple forwarder to tunnel connections to the Tor server.

There are always going to be bugs, but this would very strongly isolate each of the risky components. Feeling even more paranoid? Put a strong network filter in place between each of the components to make sure that only the specific subset of TCP that's actually in use makes it through; that'll prevent network stack-level issues.

There are some jigsaw pieces missing here.

If the nginx configuration for port 443 did indeed not restrict access to [star].php, then that means that index.php would have been accessible to the Internet at large (although HTML elements with other suffixes - e.g. .jpg, .css, .js - would not have been served).

If the CAPTCHA element's URL also ended in .php, then it's not beyond the realms of possibility that Tarbell could type the IP address, followed by /index.php and end up seeing a screwed-up version of the SR home page, with the CAPTCHA as he describes in his testimony.

The log file entries cited are for port 80, whereas the SR webserver ran on port 443.

If the defence already have all the log files, they should grep for 199.170.71.133 in the 443 logs and/or search for a group of log file entries with simultaneous successful serves of anything ending in .php, with "permission denied" failures for things not ending in .php

Incidentally, the May 3, 2013 webserver IP leak referred to in footnote 5 to Tarbell's testimony syncs up nicely with the date of this thread on Reddit: https://www.reddit.com/r/SilkRoad/comments/1dmznd/should_we_...

Credit to Michael Koziarski for the Reddit link: http://krebsonsecurity.com/2014/10/silk-road-lawyers-poke-ho...

Am I being extremely thick here?

The evidence submitted by the FBI [1] shows the phpmyadmin page running on address 192.168.1.24. Are they claiming to have connected over the internet to a public facing RFC1918 address and it was routable?

(Saying that, I'm now realising that I could knock up a similar screenshot showing me connecting to any "real-world" IP address anyway; it seems to prove nothing.

[1] http://krebsonsecurity.com/wp-content/uploads/2014/10/70-8.p...

Interesting that they are: 1) using IceWeasel and 2) searching the login page for "downloads."

Looks to me like they're just using Kali. Not that surprising, for someone doing that type of work.

Just a guess, but I imagine that's a clone of the original server (using the server image they took) running on a local network.

It's no longer a clone if they've made config changes... o_0

The way I read that was that the phpmyadmin screenshot was not supplied by the FBI, rather it came from Krebs for the purpose of demonstration.

I don't think it came from Krebs. The pdf is labeled like the rest of the evidence submitted.

Looking at it again, you're right. Its definitely from the case files.

Talk about burying the lede! Even though he starts off by saying the data confirms the "parallel construction" theory, he proceeds to poke a hole in the defense's claim - that someone wouldn't have been able to access the login page (I am too tired to determine whether or not he's correct about this), and then proposes a possible mechanism (scanning the entire Internet for pages containing a term) which, although it would technically constitute "parallel construction" in that it wasn't mentioned by the prosecution, need not have anything to do with the NSA and could have been done (with enough time) by anyone.

If he's right that the quoted configuration file doesn't do what the defense said it did as part of their accusations of lying, they've seriously messed up, no matter what the truth is.

> The second problem is that some of the details are impossible, such as seeing the IP address in the "packet headers".

If the frontend server in Germany is reverse proxying through to the backend server in Iceland, then sure, a user is not going to see the Icelandic server's IP in the source IP field of the packets. But I don't see this as definitive proof of the FBI's assertion being a flat out lie. The IP could easily have been exposed in the packet body.

What happens if you visited the captcha URL with a HTTP/1.0 request without Host header? If the resulting URL generated any self-referential links, what did they use as the hostname? If the Host header is available the norm is to use this, but if not then the script may use the server's FQDN or IP address. If it sent a 301/302 redirect in the HTTP response headers, then that _must_ contain a hostname according to the RFC (it shouldn't be relative), so what was used there? There's nothing in the nginx config that rewrote such response headers.

What happens if you make malformed requests to the captcha URL? Do you get an error page with the IP address embedded, or something that references an object hosted on the IP?

These are just two possibilities, and yes, neither would lead to the IP being exposed in the 'packet headers'. But it's very feasible for it to be exposed in the packet body, so it seems silly to hang the entire argument on the basis that one word is correct, without considering the alternatives.

In both the cases you site would there be a trail of those requests in the server log files? If so they could have been (or will be) pointed to by the prosecution. Since the prosecution has an image of the server they should be able to demonstrate the point if needed. If they can't then I think it casts doubt on their story.

You'll see the requests in nginx's access logs, but it might be very hard to isolate them. These would not show the response data (only the length and the response code).

But you're right, I'd expect that whatever tactic they used, the prosecution should be able to demonstrate in far more depth how the IP leaked.

It was no image - it was a tarball with some highly suspect mtimes.

> The IP could easily have been exposed in the packet body.

Which would make the FBI's assertion that they found the IP address in the packet headers a flat out lie...

I know for a fact that there was a source disclosure MySQL injection bug on many of the pages in SR.

There was also other MySQL Injection bugs. You could even look through the SR forum archive and find people talking about how the search field at one point was exploitable by the standard "' or 'a'='a" and was disclosing customer's names and addresses.

I don't know how the US legal system works - but is there any chance of the DPR actually walking away a free man if the FBI are lying?

Maybe. Though I don't know what other evidence they have. It might allow him to plead to lesser charges. Long but give an overview of this point: http://legal-dictionary.thefreedictionary.com/Fruit+of+a+poi...

It would be a big step forward. The government would be unable to use anything that it obtained from the server in their prosecution. This may make it impossible to prove that he ran Silk Road. I think he has several other charges though, including multiple counts of solicitation to commit murder. Those may well involve evidence from outside the server that would be unaffected by this.

Given this, the odds of DPR ever seeing the light of day, even if they win this suppression motion, are quite small. He is looking at multiple life sentences even without Silk Road specific charges. But his lawyers are going through piece by piece, hoping to convince prosecutors that it will be difficult enough to get convictions that they offer him a substantial, but less than life, prison sentence in exchange for a plea. Even then we're talking about decades in prison.

I think he has several other charges though, including multiple counts of solicitation to commit murder.

Nope, most were dropped before the indictment. Only one (in Maryland) remains: http://freeross.org/correction-of-our-report-on-the-indictme...

Still, one is probably enough for a decades-long or life sentence.

This could also be a HUGE step for all anti-parallel construction cases out there. Not only will it embolden other lawyers to look for details in their cases that might point to parallel construction, therefore leading to more such discoveries, but the government should also freak out that their parallel-constructed cases (I assume they have at least hundreds we don't know about, possibly thousands) would get invalidated, and might actually stop getting information this way.

> the government ... might actually stop getting information this [parallel construction] way

lololol. They'll just smooth the practice over with some new laws which, like every other blatantly unconstitutional law currently in effect, won't be overturned by the courts because doing so would upset the status quo.

The only way it'll ever stop is when the fuckers are finally bankrupt, both economically and socially.

There is a very very very slim chance.... I wrote a bit about this last year when he was first arrested.

http://msansnom.tumblr.com/post/63069733245/fruit-of-the-poi...

No. It's federal, they seriously don't like him, he's never walking. At some point a judge will come up with a bullshit ruling that covers the feds' collective ass, and he'll die in a cage.

All evidence from the server, and anything obtained subsequently because of the server evidence, would be inadmissible (i.e., it could not be presented to the jury). The burden is on the prosecution to prove DPR guilty beyond a reasonable doubt, and so their job gets a lot harder. Probably before it gets to trial, the prosecution could be sufficiently concerned about their ability to convict that they let DPR plead guilty to some lesser charge.

I thought it was a foregone conclusion that the NSA uses its massive surveillance network to do traffic analysis on all entrance and exit nodes.

What we're seeing here is called "parallel construction". The FBI was given this illegally obtained surveillance data, made the arrest, and then needed to make up a lie as to how they really found him.

Fun, failing to count how many times he said "I am an expert", I assume thats the basic tactic to survival in security consultant space but still.

I would say you assume incorrectly, and this guy is just a tool.

Yeah, that pretty much undermined his speech..

I thought he pressed that point a little as well but assuming his lack of writing skills disqualifies his security opinion is a bit of a far jump to me.

The original charges against Ulbricht, as well as his defense attorney's responses, both include this kind of "I am an expert" language. This is most likely a convention of legal writing, not technical writing.

Not quite, Silk Road document is using very neutral and "objective" kind of speech, c.f. "Based on my training and experience, Silk Road has emerged as the most sophisticated and extensive criminal marketplace on the Internet today." [0]

[0] http://computersweden.idg.se/polopoly_fs/1.526338.1380735946...

This is actually a legal convention, not arrogance. Look at criminal complaints for other computer crimes (child pornography, hacking, etc.) and the language is identical.

Hm. "As an expert I can attest DPR sucked at opsec"¹ is a bit unconventional for a legal convention. ;)

¹) Not an exact quote, I know, I glued two sentences together and cut some parts.

Nearly all infosec experts can agree that DPR was awful at OPSEC, and expert testimony is a real legal thing. :)

Yeah, I was only a bit curious about the wording of the statement.

The Iceland/Reykjavik bits are pretty interesting[1].

I wonder if they just said here you go? It sure seems like it.

[1] http://krebsonsecurity.com/wp-content/uploads/2014/10/70-4.p...

Is it time to have special judges/jury who understand the technical details of such cases? I can't seem to even think as to how a judge/jury with no technical expertise would understand the fine print of the details being given out by the lawyers.

I think expert witnesses serve that purpose. There have been complicated technical details in jury trials before the internet and this seems to be an adequate way to solve the problem.

The problem with expert witnesses is that the skill required to tell which expert is more credible is the same skill that each expert claims to represent best. And then you have the outright liars who have made their careers being expert liars in court, like Dr. Michael West, who are not backed up by the medical community, but by prosecutors.

It's interesting to parallel to medical cases. Ordinarily, a jury uses their own judgment to decide if someone was negligent measured against ordinary standards of care. But in a medical malpractice case, they decide instead whether someone failed to adhere to accepted medical practice. That gives the medical community a lot of power, because they get to set the standard of practice, and testimony about a specific doctor's actions is always filtered through a medical expert witness.

To date, engineers have not been given similar consideration, probably because they're not independent and their employers would force them to abuse the privilege. E.g. automotive engineers would say "oh, it's totally acceptable to have that exploding gas tank" and software engineers would say "oh, it's totally okay for that to lose all your data."

Barring special consideration like that, it comes down to the expert witnesses. The jury will decide based on which expert they find more credible / whose lawyers do the best job of presenting their expert.

Can someone who knows things about things address this item in the errattasec article:

> BTW: one plausible way of having discovered the server is to scan the entire Internet for SSL certificates, then correlate information in those certificates with the information found going across the Tor onion connection.

Would this be considered parallel construction, or would this be a legitimate way to attempt to figure out who was involved in the Silk Road, and is it plausibly the way the FBI might have zeroed in on the server?

parallel construction is retroactively finding an alternate, legal/unclassified explanation for an evidence trail, but one which is inherently incorrect; if the ssl cert scanning or other repeatable method was the route taken, nsa probably wouldn't want to tip their hand. it's probably worth noting that the only examples of parallel construction that we have confirmation of were drug cases

Yes I know what it is, but what I'm curious about is would that tactic fit the definition. Scanning IP addresses for publicly served SSL certs and comparing them with one served over TOR isn't obviously an unwarranted search is it?

If the original TOR cert was discovered in an unlawful search, and then they did a scan of the public internet to find it again in a legal search, that's parallel construction.

And let's not forget: NSA already have an almost-irresistibly useful database designed for exactly this purpose (selecting and correlating on attributes of SIGINT-captured SSL/TLS sessions, such as certificates - and they could easily just put a selector on the CommonName or the certificate fingerprint).

You don't have to be the NSA to make a database like that, but it helps. I could build a database broadly like that for certificates/ciphersuites/other metadata myself with active scanning and zmap (and it might make a good weekend project, to examine and contrast RC4 proliferation amongst TLS-encrypted web and mail servers) - but they have a near-realtime-updating passively-constructed one. If the FBI asked them for help, they'd definitely use that.

It's hard enough to explain this kind of stuff to one intelligent businessperson--there's just so much background knowledge that goes into understanding this. Explaining this to 12 jurors, half of whom will likely be of below average intelligence, is a pretty much impossible task. Sadly I think this case will be won on the charisma of the lawyers rather than facts which will go over most jurors' heads.

This may have something to do with discovering the real IP.

https://www.reddit.com/r/SilkRoad/comments/1dmznd/should_we_...

How much time would it take to browse all public IP addresses?

Depends on your definition of 'browse'. But possibly 45 minutes https://zmap.io.

This is strictly for IPv4 addresses

Note that in one of his talks Morgan Marquis-Boire said he had to change IP some times because he was blocked generating too much traffic while scanning.

But he managed to map a lot of secret Trojan servers used by govs.

Someone recently did this and posted screenshots of what was available on the public internet (including a control system for a water turbine, IIRC).

You have a link?

If you scroll down past all the photos of chilli peppers, this guy was posting interesting things he found during the VNC scan: https://twitter.com/viss/media

(Would be nice if there was a gallery, as infinite scrolling uses a ton of browser memory, but I'm not aware of one...)

This is the relevant tweet: https://twitter.com/Viss/status/499980463646048257

Why are you asking?

This was discussed in the original linked article http://blog.erratasec.com/2014/10/reading-silk-road-configur...

before the mods changed the link. Here's a vote for changing the HN code so it says "link changed from $X to $Y" at the top of the page when they do that.

Isn't it contempt of court to bear false witness; don't people caught doing that get charged and put in prison? So, which person was in the witness stand and will USA prove to be a democratic nation under the rule-of-law and charge them?

The government (executive) lies to other branches of the government (judiciary) with impunity. Once in a while, a judge will get angry about it, but prosecutors and expert witnesses issue whopping lies into the official record every day.

Aren't USA judges voted in? Why do the people vote for judges that will sit quiet whilst someone provably lies in court?

We found them by simple looking at the IP that the packets of the captcha were served from. The captcha was served over tor, based on the evidence that's impossible. Either you lied or you need to add further evidence. If you can't/won't add further evidence then as the court sees it you lied and are in contempt and will be punished in line with minimum sentencing??

This is of the order of a "he robbed me at home at exactly 2pm", "your submitted cell-phone evidence says you were at work from 1pm until 4pm".

If the defendant is convicted on some other counts can they, or anyone else, do anything or is the allowance of officers of the FBI to lie in open court somehow embedded in the USA constitution.

As a foreigner to the USA this sort of thing just undermines the entire foreign policy rhetoric of bringing democracy to the world. Bring some damned democracy to the USA first: government by the people for the people, my arse.

Why do the people vote for judges that will sit quiet whilst someone provably lies in court?

Most voters don't have information about such incidents or enough time to crawl through court records looking for that information. Actually, in judicial elections, most voters don't even have anyone else to vote for.

Someone else can correct me if I'm wrong, but judges are not voted in. At least, I know that supreme court justices are not and I assume other positions are done the same way. US judges are elected by politicians (Supreme Court justices being elected by the president, although Congress can veto that decision).

Any time you have two opposite sides in court, one says yes the other says no, one of them is lying. So, in addition to a regular guilty / not-guilty verdict, do you want to also see an automatic contempt of court verdict for the losing party?

If they provably and knowingly lied it seems reasonable. Here there seems like ~0% chance that the FBI expert in computer forensics (which they'd need to be to be running such an investigation, surely) can't possibly know that he's not lying?

If you can't tell the truth in court then keep your mouth shut or you'll be punished accordingly seems like the exact message a country intending to operate under the rule-of-law should be promoting.

> Aren't USA judges voted in?

This case is being heard by a federal court, the United States District Court for the Southern District of New York. Federal judges are appointed by the president with the consent of the Senate.

Perjury requires that you are knowingly lying.

Some judges are appointed, some are elected. AFAIK (IANAL/YMMV) appointment is more common at the state and federal levels, while elections are more common at county and municipal levels.

IMO electing judges and sheriffs makes negative sense, but that's what we have going on here.

I wish they'd have made the FBI's story a little more believable. I mean, it seems slightly more legitimate than what you see on TV and movies about hacking.

I have a feeling that they don't actually care. They know that judges and juries can't handle these kinds of technical details.

That's what experts are for.

I don't do forensics. But I am a reverse-engineer and I am familiar with the techniques: more familiar than Tarbell, it seems. (That's really his name? Tarballs from Tarbell? My goodness.) Tarbell's declaration reads to me more like a textbook demonstration of (bad) parallel construction in action.

They could have done it legitimately, without compromising the server and potentially tainting the evidence any way they wanted: DPR indeed made a few rookie mistakes that would potentially provide for that. But the logs don't seem to actually have evidence supporting that, which is very unusual and at this time not explained? The declarations filed so far do not really seem to support that either, which is very odd and strongly suggests that we don't have the whole picture here: and we really should.

(Of course, we don't have the whole image, so we don't have the whole picture here. BTW: They used tar, not dd or ddfldd? Boo.)

So, I had to indulge myself...

http://www.collegehumor.com/video/6905757/the-worst-hacking-...

...I've got to be honest, a few of those are better than the FBI story!

Ouch on the downvotes, it's called sarcasm guys!

Why not link the original Krebs article? http://krebsonsecurity.com/2014/10/silk-road-lawyers-poke-ho...

Ok, we changed to that from http://blog.erratasec.com/2014/10/reading-silk-road-configur..., which points to it.

oh HN, the articles are completely different with one general and the other technically detailed.

Please, change it back. These are completely different articles, and most of comments here are about the first one.

Seriously how is this constructive? We let people up vote and comment on an article and then swap it out to something else leaving all the comments and up votes?

HackerNews, you frustrate me.

Also the new one is not accessible due to server overload.

Ok, done.

Thanks for all the work you do to make this place a little bit better every day.

You're welcome! I appreciate your saying so.

Thank you!

I don't know why you (HN administration) need to do these URL switches. Isn't their room on the internet to add a line "Editor (dang): original source appears to be https://example.com/original-source" and let people decide for themselves which source to use whilst maintaining the context for the discussion?

This seems like a mistake. The original posted article wasn't blogspam. It commented on the krebs article. Adding detailed commentary to a source has a long and valid history.

Maybe we could have a whitelist, where non-blogspam blogs get a 'pass' on being edited. In this case, the Errata Sec blog provides much more value than some crappy blog spam and stands on it own. (Errata sec provides a desired value-add, you might say.)

That would have been the far better choice. I don't know how this article got to #1 on HN (see my other comment).

The FBI owes me 1 bitcoin.

I am a foremost Internet expert on this sort of thing..........As an expert in such topics as sniffing passwords.........As an expert, I know the Tarbell declaration is gibberish. As an expert reading the configuration and logs........

HA!

It is somewhat comical, but he's not lying. At least he's not hiding the fact that he's arguing from authority (his own).

Arguments from authority might be a bad thing on internet fora where everyone is assumed to have infinite time (and ability) to dig into details. In society as a whole, and especially in the courts, however, these arguments are a useful short-cut.

There is nothing inherently wrong with making judgments based on one's expertise. That's pretty much what expertise is good for.

I was quoting from the original article http://blog.erratasec.com/2014/10/reading-silk-road-configur...

Which has since been changed to http://krebsonsecurity.com/2014/10/silk-road-lawyers-poke-ho...

It may help to rewrite those in your head to, "as an experienced systems administrator who knows how these particular config files work" or perhaps "how config files work in general".

These are all questions of basic fact and they are all easily testable.

What do courts do when this situation comes up? Do they play warring experts, when (at least) one side definitely wants to perform a test, because they are confident that their interpretation is correct?

(ed: 'questions of basic fact' like, whether a server with this configuration is hittable from non-allowed IPs)

That's the judge's job to figure out what the truth is. If needed I've heard a court can hire their own impartial expert.

"myphpadmin"? "200 error code"? "As an expert[...]" ... no comment.

Reply on news.ycombinator.com