Hacker News Reader

Why do you need so much password cracking power? Do you forget it that often?

lol :)

I'm inclined towards the hypothesis of parent being a pentester and buying for their freelance business, or company.

What motherboard can you buy for under 50 with 6 PCIeX16 (power) slots? The cheap "bitcoin" motherboard that go for about 50-75$(old intel express chipset) won't work in this case, not sure about the 1070 but the 980 (non-ti) and the Titan X do not want to work with any PCIe slot that isn't a true PCIeX16 power slot. The PCIe x1-4 to x6 riser converter boards do not work, even the "good" ones with a SATA power connector and an actual power PCB for some reason, my guess is that there is some actual sensing going on in the card itself.

Unless you buy something like this http://www.amazon.com/ASRock-Motherboard-H81-PRO-BTC/dp/B00H... which which is i think the only non-server motherboard which is commonly available you aren't getting more than 4-4 PCIe slots, and even to get 4 (PCIe16) you need to spend about 200$ on a motherboard these days and the H81 chipset mobo's don't want to work with modern nVIDIA cards (or the other way around) with all but the main PCIe16 slot.

Pretty much to get more than 2 PCIx16 slots these days you have to buy a good gaming Z170/Z97/X99 motherboard, that ain't cheap, nor are the CPU's that go into them. While the AMD CPU's might be a bit cheaper again the motherboards won't be.

P.S. my personal password cracking setup at home is 4 5970's you could get them for about 100$ even a couple of years ago (today they are much cheaper 2 for 130 or b/o: http://goo.gl/yplISa) and for most hashes they go 40-50% of the perofmrnace of a Titan X (I do have 2 Titan X's in my personal gaming rig which i can use in a pinch). I got them setup on an ASROCK motherboard (h61 iirc) that was designed for bitcoin mining so it comes with 8 PCIE slots and powered converter ribbons connect the cards do it all of this sits in a al cheapo rackmount case of a 20 year old compaq server.

But if you are looking for more scalable compute instance AWS CUDA instances are probably a better solution and I've been using them more and more lately.

I have been wondering about the AWS GPU instances. They are something like $2.60 / hour. At what point do they make sense monetarily? You can build a machine like described here for ~$5000 total. Thats not that many hours of AWS time. So if anyone is doing any long running cracking, or machine learning, does it ever make sense? I am truly curious about what people's experience has been.

g2.8xlarge $0.7414 per Hour (linux pricing) it makes sense unless you are doing it for a "hobby", you just roll costs into each project which makes it much easier from an accounting POV :P

If you buy Spot instances you can get them for ~85% discount off normal pricing. Your spot instances can be pre-empted, of course, but for password cracking this shouldn't really matter much.

>At what point do they make sense monetarily?

For password cracking? Never, especially considering you can just go on insidepro or something and rent access to someones cluster.

Insidepro is a great resource, but you usually aren't going to get a VAT invoice to either bill your client for or deduct in your taxes ;)

AWS is simple, it's not overly expensive for most password cracking ventures (e.g. password quality audit that certain organization like to do on their AD/LDAP) and more importantly you aren't giving away potentially very sensitive information to people that by default cannot be trusted.

Inside pro is ok if you are going mostly non-legit / off the books to begin with, it's also ok for basic research on non-real world data, but never use it for actual real world data not to mention B2B services.

Cloud is a no-go if you're handling sensitive client data (like all of their passwords)

Huh, interesting. I know Amazon, at least, has provisions for HIPAA, [1] which seems pretty strict. Having to notify newspapers in case of breach and some pretty stiff fines. I know a bunch of places do credit card processing on AWS. PCI is pretty strict as well.

Is the big concern physical access? Or is there a subtle hack to escape the VM you're running in? I'm just having a hard time seeing the difference between a vm connected to the internet at the office, a colo server and a aws instance. I guess you must be referring to minimizing surface area, not a proven attack.

It's tempting to trust your team, and you need to. If you're doing real, regular audits of access controls and have some pretty robust physical security, that could be a win. Seems like you'd have to be a pretty big shop to do security "right".

[1] https://aws.amazon.com/compliance/hipaa-compliance/

Compliance is not security.

A sophisticated attacker with access to Amazon's infrastructure can read anything they like out of the memory of the machine your instance is on without you or anyone else ever knowing it.

If you are worried about actors with that capability you probably want to airgap your entire operation.

Doesn't need to be airgapped, just defended, and not in the cloud.

If the client is someone like Juniper or Cisco (which, if you're a pentest firm buying 8x gpu servers for cracking hashes, would be a reasonable customer) nation state actors would absolutely love to have access to your dataset. And they're already in AWS's datacentres.

> Unless you buy something like this

That is exactly the motherboard that I was thinking of - I just rounded down a little too aggressively (iirc you can get them for $69 on NewEgg).

> not sure about the 1070 but the 980 (non-ti) and the Titan X do not want to work with any PCIe slot

I'd completely forgotten about this - thanks. When I initially started setting out a new rig build I was looking at the 970 which would work.

Here is pcpartpicker filtered with at least 4x PCIex16:

http://pcpartpicker.com/products/motherboard/#h=4,7&sort=a8

It brings this board up:

http://pcpartpicker.com/product/3YH48d/gigabyte-motherboard-...

Think that works?

With your cracking setup I can definitely see the attraction of building multiple cheaper boxes rather than one larger one. This is definitely something I need to spend more time researching (which i'd intended)

I'm using AWS at the moment - I fire up spot instances but have noticed that there has been a lot of demand recently and prices spike a lot. I've written some basic scripts to bring instances back up, continue tasks, send jobs, shutdown on complete, etc. I'm spending a lot on AWS at the moment which is why i'm looking at putting a new rig together (my old rig broke down a year ago and is 6+ years old)

I don't know if the 970 will work, don't have one, IIRC the GTX770 doesn't want to work either, but I don't remember if i tested the 770/780ti's i had lying around from older gaming rigs.

The AM3+ mobo's might works, just make sure to get one that actually has been rated for 4-way xfire since I've noticed that quite a few of the mid range ones(regardless of being Intel or AMD) don't actually support more than 2 PCIEx16 power ones at the same time (none of them come with 48 PCIE lanes to begin with).

According to wikipedia: 990FX Four physical PCIe 2.0 ×16 slots @ x8 electrical which can be combined to create two PCIe 2.0×16 slots @ x16 electrical, one PCIe 2.0×4 slot and two PCIe 2.0×1 slots, the chipset provides a total of 38 PCIe 2.0 lanes and 4 PCIe 2.0 for A-Link Express III solely in the Northbridge

Looks like you'll be getting either 2 x16 electrical with the rest disabled when a card is inserted (if the OEM decided to invest in some extra hardware) or and 4x8 electrical. x8 electrical is usually enough for (most) Nvidia cards to work, not sure about the 1070/1080, but you probably want to confirm it.

You can ignore the PCIe land bandwidth since it's not a factor for (most) compute applications, but the electrical stuff is sadly a pitfall, this gen could even be potentially worse since allot of the OEM's dropped to a single power connector (6pin on 1070, and 6/8pin on 1080 non OC cards) which might indicate they are planning to draw more power from the PCIe bus. The fact that Nvidia kinda also dropped support for 3 and 4 way SLI (which now requires an "enthusiast" code that is written to the bios of your card) could also introduce some issues if OEM's pretty much bet that you'll only plug the card into an X16 electrical slot even in SLI so the cards might either not work or not be stable if they are connected to an X8 one (x1/x4 shouldn't work at all unless nvidia did a complete 1080 from the previous gen).

P.S. if you are building everything from the ground up and not using old hardware that you already own I would look at old (like 2nd gen Core based Xeon, or old Opteron) servers with lots of PCie slots, you can usually find them for a couple 100's $ on ebay.

Thanks again - i've been digging into the specs of those boards and it does look like you have to go pretty far down that list to get to a board that can support 4x PCI-E v3 x16 simultaneously so you were right with your first comment that it is $250+

HP DL380's which can take 2 cards for $200 - pretty good.

"the Titan X do not want to work with any PCIe slot that isn't a true PCIeX16 power slot."

You should try shorting the presence detection contacts, as described in the middle section of my post:

http://blog.zorinaq.com/?e=42

That was years ago I did it, but it never failed in making any x16 card work in an x1 PCIe slot.

I'm using these adapters that give me a PCIe x16 mechanical connection as well as the extra juice via a molex/sata power conector: http://www.frozencpu.com/images/products/main/cab-1907.jpg So i don't think I need short the detection contacts to ground i think because the PCIex16 mechanical slot is does it for me (the AMD cards work without any modification).

The Titan X doesn't want to work with these, I never paid it much thought since i wasn't going to fill the machine i have with Titan X's anyhow, I just wondered if i can use them when i retire them (i filled with 5970's since i got 4 of those for about 300$ a about a year ago).

I will investigate this further if at any point I'll retire the Titan X's and decide to replace my some of the AMD cards, atm I'm still using the Titan's for gaming, and I really don't want to build a mixed GPU rig since it won't give me any boost to performance unless i somehow manually batch my hashes since iirc hashcat doesn't know how to address multi vendor GPU's my itself.

In my experience many PCIe extenders—even those extending x1 to x16 with a separate power plug—do not short the presence detect pins.

Also, I don't think the fact AMD cards work does not mean shorting won't be required for other cards. The PCIe initialization process is complex enough that there could be a variety of reasons why AMD cards can be down-plugged but not your Titan X.

In short (pun intended) I would highly recommend you to try shorting if you feel confident in doing it.

You may want to look at hybrid/liquid cooling with an X99 motherboard e.g. 4 cards x 16 PCI-e lanes http://www.newegg.com/Product/Product.aspx?Item=N82E16813132...

(tho you'd have to cut holes in the case to bolt radiator fans onto)

The problem is then you are spending about 2000$ on your "chassis"(case, cpu, motherboard, memory, cooling etc.) if you are in that situation then getting the more expensive cards (1080/1080ti or even the Titan w/e they'll name it now) is usually the only way to go because your chassis is now a major factor of the final price so you'll need to squeeze a much performance from it as possible.

Yes, I can confirm this is the case (pun intended). I've been speccing out GPU machines for molecular simulation, and any way you slice it, the actual GPUs account for less than 50% of your total cost, as long as you're looking at GTX class cards.

The other issue with 8 GPU machines, which tend to use dual CPU motherboards to have enough PCIe lanes, is that you pay a serious performance penalty for communication between GPUs attached to different CPUs (over QPI). It's so bad that I've seen several standalone 8 GPU systems using FDR Infiniband (with GPUdirect RDMA) to communicate internally on the machine. Which is of course bloody expensive.

If it suits your workload, having two separate 4 GPU machines is better and cheaper than a single 8 GPU machine.

Can you actually get full performance out of that? I thought the bottleneck these days is the CPU which maxes out at 40 lanes at the high end. This only gets you 1x16 and 3x8 even if all the slots support 16 lanes.

Thanks for posting this, I was looking at this type of build with CUDA before pascal was announced.