Hacker News Reader

cmccabeon The New AWS Command-Line Interface

Hadoop is a framework, not a library, so user applications need to link against the jars they need. That implies using CLASSPATH to locate them. Whether or not there is 1 jar or 100, the fact that there's no standard place to install jars in Java is a problem.

Hadoop jars are hundreds of megabytes, and we have multiple daemons. Duplicating all those jars in each daemon would multiply the size of the installation many times over. That's also a nontrivial amount of memory to be giving up because jars can no longer be shared in the page cache.

Some of these problems could be mitigated by making Hadoop a library rather than a framework (as Google's MR is), or by pruning unnecessary dependencies.

cmccabeon The New AWS Command-Line Interface

If you don't want the pip dependencies to "pollute the rest of the system," you can just use pip install with the --root option.

Java's CLASSPATH causes enormous pain for end-users. Just read the Hadoop mailing list. The fact that Java doesn't have a sane default for where to put anything or how to manage dependencies is a huge flaw.

Back when I was doing AWS, I just used the C binaries (I forget what they were called) to transfer things to or from S3. I just wanted to avoid installing hundreds of megs of dependencies. We paid money to transfer our AMIs around, after all! Still, a more full-featured tool will no doubt come in handy in some scenarios.

cmccabeon Linus: bool is dangerous in C if you don't understand it

In the example you gave, any compiler from the last 10 years would have given a warning message when converting the pointer to an int. Ignoring error messages is extremely, extremely bad practice. If you are still doing that, I suggest auditing your code right now to see if there are more problems such as mismatched printf arguments and so forth.

bool is a bad thing in C and C++, because of the implicit conversion rules. In C++, pointers implicitly convert to bool in every context-- there is never any warning message. This effectively reduces the compiler's ability to typecheck your program, since so many variables are pointers already, and C++ will happily stuff them into any bool argument to a function.

It seems like the C99 _Bool type implements the same implicit type conversion brain damage. That makes it a step backwards in terms of type safety, not forwards. That's right, using the old-fashioned, fuddy-duddy, plain old int and 0 and 1 gives you better type checking than the shiny new C++ feature.

As Linux points out, the worse typechecking comes with a side order of compatibility problems. And it is not any more efficient or readable.

cmccabeon The Decline And Fall Of Flowtab, A Startup Story

The problem was that this idea was a feature of a point-of-sale (POS) system, not a product.

cmccabeon A flexible E Ink screen that could be in the 2014 Kindle

There are a bunch of e-ink readers out there already that do well with PDFs. I have an e-reader made by PocketBook (http://en.wikipedia.org/wiki/PocketBook_International) which is 10.5 inches long and does PDF very well. Amazon also made a bigger Kindle called the DX. That one wasn't quite as good as PocketBook at doing PDFs, in my opinion, but it offered all the other Kindle features.

It will be nice if Sony makes a 13.3" e-reader. Hopefully the software won't be terrible...

cmccabeon San Francisco Real Estate Exuberance

Did you read the article? SF has been expensive for a long time. When adjusted for inflation, this isn't even the most expensive that it's been.

cmccabeon San Francisco Real Estate Exuberance

You know, there is more in the bay area than San Francisco. If you try living in Berkeley or Oakland for a few months, you will find that it isn't "a remote, suburban sprawl-y part of the Bay Area." It's only a half hour away by BART from all those shiny SOMA offices. And there is San Mateo and all the other towns on the penninsula.

cmccabeon Guido van Rossum Deconstructing Twisted's Deferreds

You can always pass around pointers to immutable data structures which only implement read interfaces.

cmccabeon Mark Zuckerberg’s Facebook page was hacked by an unemployed web developer

Is anyone here arguing in favor of colonialism? This is equivalent to comparing people who disagree with you to Hitler. Thanks for your "extensive academic thought," which I note you did not even bother to summarize here. I think this thread is over.

P.S. While I do think some of the statements made by cliveowen were insensitive, I also think that teaching people English (or another world language) in addition to their native tounges would help break down barriers between people, and lead to a better world overall.

cmccabeon How I Made My Ruby Project 10x Faster

Yeah, it sounds like he should just use a statically typed language. (yes, I am aware that types != contracts, but they're similar enough that my statement still stands)

cmccabeon Guido van Rossum Deconstructing Twisted's Deferreds

Now that we are on the subject, what is one sane way to drastically reduce the need for locks and semaphores? Use actors like Erlang does. A class with a thread and a queue attached to it. Then avoid sharing data instead send messages. You'll be hit with a serious performance issue because of memory copying, because well, there is no free lunch.

In Golang, you can avoid the serious performance issue by sending a pointer over a channel and treating that as a transfer of ownership of the data.

cmccabeon 40 Days Without Booze

You're posting drunk, aren't you?

cmccabeon Why is FreeBSD deprecating GCC in favor of Clang/LLVM?

Most of gcc is still written in C. Only a tiny amount of new code is in C++. In my opinion, that's the worst of both worlds-- inconsistent code, plus C++ overheads.

I work with some fairly recent version of both compilers-- gcc 4.7.2 and clang 3.2. I also know a developer on the LLVM team. clang is way ahead of gcc in many ways, and the gcc devs know it.

We still ship software built with gcc, rather than clang, because gcc is still better at optimization. (gcc also supports more obscure targets, though that's not our biggest concern).

We also make use of llvm's ability to dynamically recompile code on the fly in Cloudera Impala. More information here: http://blog.cloudera.com/blog/2013/02/inside-cloudera-impala...

This would not be possible with gcc no matter how much work we did (license issues-- Impala is Apache 2.0, gcc is GPLv3).

cmccabeon Why is FreeBSD deprecating GCC in favor of Clang/LLVM?

Well, I like the clang error messages better than gcc's, and I use recent versions. And to refer to the major architectural differences between the two as something that could be bridged by a single "patch" is a little silly.

gcc is still better at optimization, though. For now.

[edit: wow, people are really confused here. Guys, LLVM was designed from the start to have a clean separation between passes. You can compile some other language to the LLVM bytecode and then just use the code generator, for example. It even includes a JIT! In contrast, gcc is a huge monolith. The projects are written in different languages, too... C++ versus C.

tl;dr is that when you say that "the only difference is the license," you sound like someone saying "the only difference between Ruby and Java is the file extension." Pointy haired.]

cmccabeon Perl 5.18.1. Yes, it's alive

OK, I admit it was a cheap shot. But I'm not the only one in the thread making it. And the quote is definitely bizarre. Can you explain how it's related to Perl?

I agree that Perl has done a very good job with backwards compatibility. I have run many ancient Perl scripts that all work fine, and yes, CPAN is huge. I never liked the language, but that for other reasons, not worth going into here. Feel free to mod me down or whatever, just speaking what is on my mind.

cmccabeon Go for System Administrators

Maven isn't standardized. There are still a lot of projects that use Ant + Grail or Gradle. There's also OSGI, which is sort of yet another way to manage dependencies in Java.

Maven doesn't prevent versioning issues from cropping up, either. One common example is where dependency A pulls in library C 1.0, but dependency B pulls in library C 2.0. They often conflict, and then your software fails at runtime (fun fun!) There are a bunch of hadoop bugs where exactly this happened, although I'm too lazy to look up the JIRAs right now.

I feel like if you want a stable version of something, you should use yum, apt-get, or your package manager of choice. I don't think the programming language should try to do the package manager's job. It never seems to end well.

So far, I haven't seen actual harm caused by a lack of library version numbers, and there seems to have been a lot of good done (people using and testing latest versions, green master policy, etc).

cmccabeon Go for System Administrators

You can't install a java tool-- any java tool-- just by a file copy. At minimum, you need to install the proper version of the JVM (different software may need different versions)-- that's at least 50 MB and maybe 100 these days. Then you need a directory with all the jars which the tool depends on-- figure another 50-100 MB or so for that. Finally, you need a shell script to set up the CLASSPATH, set up any JVM arguments you might need (increased heap size, anyone?) and actually locate the binaries.

In Go, you just copy the binary to the machine and run it.

cmccabeon Go for System Administrators

Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly.

- The Story of Mel.

Real Users hate Real Programmers - FreeBSD fortune

cmccabeon Go for System Administrators

Yes, you do have to keep your master clean. The version tagging thing is only intended to be used when you start depending on a new language feature not available in older versions of Go. For example, if your new code needs something that's only in Go 1.1, you could create a 1.0 tag right before making that change. But people using new versions of Go will get whatever is in master, not whatever is at that tag.

Yes, you can always clone explicitly into a path, but then we're not talking about the module system any more, are we? We're just talking about you copying around files locally. It's misleading to state this like it's the recommended or usual way to do things.

cmccabeon Perl 5.18.1. Yes, it's alive

Yeah, it's pretty bizarre.

Was slashdot still relevant in 2005? Was perl? I feel like they both peaked right around the turn of the century.

cmccabeon Go for System Administrators

I think what Go's subprocess stuff needs most is better examples. There aren't a lot of examples out there yet. For example, I saw one guy on StackOverflow use a complicated Reader thread to print a subprocess' output to stdout, when all he had to do was set Command#Stdout to os.stdout.

I hope that I'm not assuming too much here (maybe you really did need bufio for your code), but in a lot of cases you don't need it. Like if you want to supply stdin to a process, just use Command#Stdin = strings.NewReader("what I want to send to stdin"). For getting the command output, it's often enough to just call Command#CombinedOutput and cast to a string-- no bufio required unless you're doing something fancy.

I wrote some utility code that might or might not be helpful: https://github.com/cmccabe/test2/blob/master/subprocess.go

cmccabeon Go for System Administrators

One thing that annoyed me about both Python and Rubyis that the way things in the standard library worked changed from release to release in subtle, incompatible ways. I remember getting burned by a change in how iterating over characters in a string worked in Ruby in the 1.8.x versions. Sorry-- I don't remember the details, it's been a few years. The point is that if you want to run in on a different PC than you developed the code on, or upgrade, these are buried landmines which are going to blow your leg off sooner or later. If you're working on an open source project, often the first sign you get that your Python code doesn't work on older or newer versions is a user reporting a mysterious RuntimeException.

virtualenv can sort of solve a lot of these issues, if you have complete control of all computers running your software, and if you can find a Python version that works for all the packages you need. But for a lot of open source projects, these assumptions may not be true. And then you have the issue of stagnation. It sucks to be stuck on Python 2.4.x in 2013. But how do you make a business case for a potentially very risky upgrade? This is the real reason why initiatives like Python 3.x and Perl 6.x have been DOA, in my opinion.

cmccabeon Injectable oxygen keeps people alive without breathing

Blood can be blue-- if you're a hermit crab.

cmccabeon Lavabit abruptly shuts down

I've never been to China to see anything for myself...

Yep, I can tell. I can tell you haven't read about it either.

cmccabeon Lavabit abruptly shuts down

The safeguards for foreigners are much less than those for US citizens. For example, the NSA doesn't need to go through FISA court to spy on non-citizens. So if you store your data on a non-US server, you're probably just making it easier for them to get to it, not harder.

Not to mention, most countries will pretty much cooperate with the US when it comes to intelligence. The only ones that might not are countries like Russia or China that have their own military-industrial complexes, which are just as eager to get at your data and a lot less scrupulous about using it.

cmccabeon Android AOSP maintainer quits

Uh, what? The only GPL code in Android is the kernel.

cmccabeon Scala vs. Go TCP Benchmark

Yeah, I always seem to forget the exact details of Nagle (probably because every project I've worked on just turns it off). Write-write-read is the killer, I guess-- just doing a small write followed by a small read, or vice versa, should not be affected by Nagle. So the results I got make sense.

Re: MacOS, I do know that some of the Go developers use Macs as their primary desktops. So I don't think they neglect it, but given that they're targeting the server space, it makes sense to optimize Linux more.

I still haven't seen any really good explanation of these results. I don't buy the argument that the JVM is providing the advantage here. The main thing that the JVM is able to do is dynamically recompile code, and this shouldn't be a CPU-bound task.

cmccabeon Scala vs. Go TCP Benchmark

EDIT. OK, so. I ran this benchmark myself on an 8-core Xeon running Linux. 2.13 GHz, CentOS 6.2. Kernel was 2.6.32-220.el6.x86_64. 50 gigs of RAM.

I got somewhere between 2.0 and 2.2 "milliseconds per ping" for Scala 2.9, and somewhere between 3.5 and 3.7 for Go 1.1. This is not the 10x difference that the authors reported, but it is something. The difference may be due in part to the different platform and hardware I am using.

Contrary to what I wrote earlier, I noticed that GOMAXPROCS=8 did seem to be slower than GOMAXPROCS=4 here. I got around 4 "milliseconds per ping" with GOMAXPROCS=8. Using a mutex and explicit condition variable shaved off maybe 0.2 milliseconds on average (very rough estimation).

Again contrary to what I wrote earlier, Nagle on versus off didn't seem to matter in the Go code. I still think you should always have it off for a test like this, but on my setup I did not see a difference.

I still don't think this benchmark is showing what they think it is. I have a hunch that this is more of a scheduler benchmark than a TCP benchmark at all. I think I'd have to haul out vtune to get any further, and I'm getting kind of tired (after midnight here).

cmccabeon Strict Posix Nonconformance

I suppose it depends on how good you are at coming up with random file names. You could also use O_EXCL | O_CREAT to fail if the file already existed. The more I think about it, though, the worse this whole interface is starting to smell.