How does it do under SMHasher (a library to test hash quality)?
2,967 karma · joined January 27, 2008
FWIW, I help maintain FlatBuffers for Go and Python. I'm happy to answer any questions I can.
Look for people who can learn the structure of a problem when there's no guidebook (or textbook) available. If they can do this, then they will likely succeed in tackling the unknown.
This 'autodidactic tenacity' can be learned inside or outside of school.
Knowing how to independently learn means you will invest in yourself for the rest of your life. Attending classes and doing homework do not, themselves, teach you how to do that.
Pardon the essay, but I've written a tool in this space before.
Back in 2011, I wrote a textual steganography library and command-line application, called Plainsight: https://github.com/rw/plainsight
Additionally, @workmajj and I wrote TweetFS using Plainsight. It lets you recursively pack up directories and post them as an encoded linked list of Tweets to Twitter: https://github.com/rw/tweetfs
Plainsight uses each byte of the input message to generate tokens. Bits are used to decide how to traverse the token tree, weighted by frequency. The drawbacks are 1) verbosity and 2) incorrect grammar.
One of the lessons of writing Plainsight is that spam can be used to contain secret messages. Send enough gibberish to enough people, with your intended recipient included, and you'll look like a spammer--not a spy.
-- Example 1 (regular text) Type your message to encode:
echo 'Meet at Union Square at noon. The password is FuriousGreen.' > cleartext
cat cleartext | plainsight -m encipher -f sherlock.txt > ciphertext
cat ciphertext | fold -s
which was the case, of a light. And, his hand. "BALLARAT." only applicant?"
decline be walking we do, the point of the little man in a strange, her
husband's hand, going said road, path but you do know what I have heard of you,
I found myself to get away from home and for the ventilator little cold night,
and I he had left my friend Sherlock of our visitor and he had an idea was not
to abuse step I of you, I knew what I was then the first signs it is the
daughter, at least a fellow-countryman. had come. as I have already explained,
the garden. what you can see a of importance. your hair. a picture upon of the
money which had brought a you have a little good deal in way: out to my wife
and hurry." made your hair. a charge me a series events, and excuse no sign his
note-book has come away and in my old Sherlock was already down to do with the
twisted
cat ciphertext | plainsight -m decipher -f sherlock.txt > deciphered
cat deciphered
Meet at Union Square at noon. The password is FuriousGreen.
$ dd if=/dev/urandom of=/dev/stdout bs=1 count=10 | plainsight -m encipher -f 1984.txt
10+0 records in
10+0 records out
10 bytes (10 B) copied, 9e-05 s, 111 kB/s
Adding models:
Model: 1984.txt added in 0.89s (context == 2)
input is "<stdin>", output is "<stdout>"
enciphering: 100%|#####################################################################################################################################################################|474.67 B/s | Time: 0:00:00
which is a war is real, the proles used mind on the telescreen. He could see through all right to. You have read what said. 'Yes,' only in the Ministry
One serious use case is to seed the generator with a spam email corpus. This lets you generate messages that look like spam. Example:
wget https://spamassassin.apache.org/publiccorpus/20030228_spam.tar.bz2
tar -jxvf 20030228_spam.tar.bz2
cat spam/0* > spam-corpus.txt
echo "The Magic Words are Squeamish Ossifrage" | plainsight -m encipher -f spam-corpus.txt > spam_ciphertext
$ cat spam_ciphertext
(8.11.6/8.11.6) 3 (Normal) Internet can send e-mails until to transfer 26 10 [127.0.0.1] also include address from the most logical, mail business for your Car have a many our portals ESMTP Thu, 29 1.0 this letter on internet, <a style=3D"color: 0px; text/plain; cellspacing=3D"0" how quoted-printable about receiving you would like width=3D"15%" width=3D"15%" border="0" width="511" Date: Tue, 27 Thu, 19 26 because zzzz@localhost.spamassassin.taint.org for
$ cat spam_ciphertext | plainsight -m decipher -f spam-corpus.txt
Adding models:
Model: spam-corpus.txt added in 2.57s (context == 2)
input is "<stdin>", output is "<stdout>"
deciphering: 100%|#####################################################################################################################################################################|543.84 B/s | Time: 0:00:00
The Magic Words are Squeamish Ossifrage wget https://spamassassin.apache.org/publiccorpus/20030228_spam.tar.bz2
tar -jxvf 20030228_spam.tar.bz2
cat spam/0* > spam-corpus.txt
echo "The Magic Words are Squeamish Ossifrage" | plainsight -m encipher -f spam-corpus.txt > spam_ciphertext
$ cat spam_ciphertext
(8.11.6/8.11.6) 3 (Normal) Internet can send e-mails until to transfer 26 10 [127.0.0.1] also include address from the most logical, mail business for your Car have a many our portals ESMTP Thu, 29 1.0 this letter on internet, <a style=3D"color: 0px; text/plain; cellspacing=3D"0" how quoted-printable about receiving you would like width=3D"15%" width=3D"15%" border="0" width="511" Date: Tue, 27 Thu, 19 26 because zzzz@localhost.spamassassin.taint.org for
$ cat spam_ciphertext | plainsight -m decipher -f spam-corpus.txt
Adding models:
Model: spam-corpus.txt added in 2.57s (context == 2)
input is "<stdin>", output is "<stdout>"
deciphering: 100%|#####################################################################################################################################################################|543.84 B/s | Time: 0:00:00
The Magic Words are Squeamish OssifrageOne of the use cases of TweetFS is to use Twitter as a 'dead drop'. You'll generate a lot of tweets by doing that, but there's no harm done.
I don't think I hijacked your thread. The other comments also discuss textual steganography.
Additionally, @workmajj and I wrote TweetFS using Plainsight. It lets you recursively pack up directories and post them as an encoded linked list of Tweets to Twitter: https://github.com/rw/tweetfs
I presented Plainsight at Hack'n'Tell NYC in 2011 and a video was recorded: http://bit.ly/pecGgW
Plainsight uses each byte of the input message to generate tokens. Bits are used to decide how to traverse the token tree, weighted by frequency. The drawbacks are 1) verbosity and 2) incorrect grammar.
One of the lessons of writing Plainsight is that spam can be used to contain secret messages. Send enough gibberish to enough people, with your intended recipient included, and you'll look like a spammer--not a spy.
I also wrote a fuzzing tool, called Shag, to find edge cases, e.g. for single-byte inputs: https://github.com/rw/shag/blob/master/shag.rb
-- Example 1 (regular text)
Type your message to encode:
echo 'Meet at Union Square at noon. The password is FuriousGreen.' > cleartext
cat cleartext | plainsight -m encipher -f sherlock.txt > ciphertext
cat ciphertext | fold -s
which was the case, of a light. And, his hand. "BALLARAT." only applicant?"
decline be walking we do, the point of the little man in a strange, her
husband's hand, going said road, path but you do know what I have heard of you,
I found myself to get away from home and for the ventilator little cold night,
and I he had left my friend Sherlock of our visitor and he had an idea was not
to abuse step I of you, I knew what I was then the first signs it is the
daughter, at least a fellow-countryman. had come. as I have already explained,
the garden. what you can see a of importance. your hair. a picture upon of the
money which had brought a you have a little good deal in way: out to my wife
and hurry." made your hair. a charge me a series events, and excuse no sign his
note-book has come away and in my old Sherlock was already down to do with the
twisted
cat ciphertext | plainsight -m decipher -f sherlock.txt > deciphered
cat deciphered
Meet at Union Square at noon. The password is FuriousGreen.
$ dd if=/dev/urandom of=/dev/stdout bs=1 count=10 | plainsight -m encipher -f 1984.txt
10+0 records in
10+0 records out
10 bytes (10 B) copied, 9e-05 s, 111 kB/s
Adding models:
Model: 1984.txt added in 0.89s (context == 2)
input is "<stdin>", output is "<stdout>"
enciphering: 100%|#####################################################################################################################################################################|474.67 B/s | Time: 0:00:00
which is a war is real, the proles used mind on the telescreen. He could see through all right to. You have read what said. 'Yes,' only in the MinistryFront-end developer
About us: QLabs is a tech incubator based in Noho, NY. We’re a small group of hackers who rapidly prototype new products every 10 weeks, then test them in the market for long-term potential. Our primary focus is the consumer web, though we experiment across a broad spectrum (e.g. hardware).
We have corporate backing, which means competitive salaries, corporate perks, and full benefits.
Our management structure is extremely flat: we succeed or fail as a team.
Our office is arguably the coolest place to work in New York: It's a 22,000 sq. exposed-brick loft, which we share with Codecademy, Turntable.fm and MoviePass.
We’re about to graduate our first product, Huntsy, and are proving an exciting new model for incubating startups.
----
About you: We are searching for a developer with front-end expertise to join our NYC team.
We are constantly vetting and integrating the latest technologies into our development stack, so you may be the person we’re looking for if:
You love building interfaces that are sexy, sleek and simple;
You have an opinion on backbone.js and/or ember.js*;
You’re familiar with CSS3, HTML5 and OO Javascript;
You're familiar with Ruby on Rails and/or have MVC experience.
[*] Our current project uses ember.js for the entire frontend.
My favorite is here: http://bleeoo.com/videos/31c92a10-c228-012e-7578-12313d09290...
The main site has many more.
> He looked at me for a long time, "A computer is telling you what to do on the job? What does the manager do?"
> "The computer is the manager. Manna, manager, get it?"
> "You mean that a computer is telling you what to do all day?", he asked.
> "Yeah."
FULLTIME NYC
Executive In Residence
QLabs is a recently formed incubator in New York City with funding from the VC arm of a publicly traded company. Our charter is to rapidly build innovative and revenue-generating businesses. Essentially, we prototype startups by building and launching an MVP every 6-8 weeks.
We are currently looking for an Executive In Residence. Your responsibilities will include (but are not limited to) creating and executing on: marketing, PR, BD and social media strategies. You'll be an equal member of the team, informing product-level discussions throughout project ideation and development (especially as pertains to market fit and revenue potential).
Major bonus points if you have hands-on experience at a startup, and know how to grow users and revenue from the ground up.
email rw@rwinslow.com to get in touch!
QLabs
We're a new type of in-house incubator: we prototype startups by building and launching an MVP every 6-8 weeks. This is like startup boot-camp. Come hack with us on amazing projects, learn everything there is to know about building and launching products and starting companies, all without the painful early-startup salary.
We take an aggressive approach to learning and inspiration. Example: six weeks ago, our team spent the day at MIT's Media Lab, where we learned about biomechatronics, fluid interfaces, and more. Even though our primary tool is Rails, half of us just learned nodejs for a two-week 'sprint project'.
The lab started in February of this year, and now has five awesome developers (hailing from Columbia, Disney Imagineering, ITP, NYC Resistor, and more). We're looking to fill position #6.
Show us your projects: we love to see an entrepreneurial streak and a diversity of experience.
Email me at rw@rwinslow.com if this appeals to you!
QLabs
We're a new type of in-house incubator: we prototype startups by building and launching an MVP every 6-8 weeks. This is like startup boot-camp. Come hack with us on amazing projects, learn everything there is to know about building and launching products and starting companies, all without the painful early-startup salary.
We take an aggressive approach to learning and inspiration. Two weeks ago, our team spent an entire day visiting MIT's Media Lab, where we learned about biomechatronics, fluid interfaces, and more. Even though our primary tool is Rails, half of us just learned nodejs for a two-week 'sprint project'.
The lab started in February of this year, and now has five awesome developers (hailing from Columbia, ITP, NYC Resistor, and more). We're looking to fill position #6.
Show us your projects: we love to see an entrepreneurial streak and a diversity of experience.
Email me at rw@rwinslow.com if this appeals to you!
QLabs
We're a new type of in-house incubator: we prototype startups by building and launching an MVP every 6-8 weeks.
This is like startup boot-camp. Come hack with us on amazing projects, learn everything there is to know about building and launching products and starting companies, all without the painful early-startup salary.
One of our upcoming field trips is an all-day visit to MIT's Media Lab.
The lab started in February of this year, and now has five awesome developers (hailing from Columbia, ITP, NYC Resistor, and more). We're looking to fill positions #6 and #7.
Show us your projects: we love to see an entrepreneurial streak and a diversity of experience.
Our one steadfast requirement is that we definitely need you to know Rails.
Email me at rw@rwinslow.com if this appeals to you!