Hacker News Reader

Comment by dchest | Hacker News Reader

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

Spawning a process, even if its just vfork() followed by exec(), is very very expensive compared to spawning a thread, and even then most apps created threads on startup and then just cache them for the lifetime of the application. Spawning a process 20 times a second honestly isn't terrible, but even hundreds per second can be noticeably slow IIRC.

I thought Linux treated forks and processes the same, where as the former is just a special case?

`fork` is how you create a new process on Linux. Threads, which GP was talking about, are different, especially Go threads, which are not the same as OS threads.

I believe GP was referring to the `clone` system call, which both fork and pthread_create use under the covers. The difference is what happens to the memory-resident data- the forked/child program shares the address space of the parent until either writes to a page, in which case the page is copied such that each process gets its own unique copy. This is known as copy-on-write (COW). Pthreads, on the other hand, outright share the process's address space and must implement their own synchronization via locks or whatever.

Obviously, the performance of fork vs. pthread_create could differ dramatically depending on what the program does.

Goroutines are a layer of abstraction above this. They might run on different threads concurrently- the Go runtime controls what happens here and may differ on various architectures / OSes. If you break into a running Go program on Linux with gdb, there's definitely a bunch of pthreads running, maybe for goroutines, and probably for garbage collection, and other stuff. (If you want to actually debug go code, you should of course use something like delve).

http://man7.org/linux/man-pages/man2/clone.2.html

Process isolation is good for security.

Parsing text data in ad-hoc, non-standardized, not documented, not defined format is really bad for security.

Just spawning a process creates as many security problems as it solves.

If it was done right, it would look like Chrome architecture, where untrusted, isolated processes can do dangerous work but communicate with trusted process via well defined IPC protocol.

Yes, but there is no need to spawn them all the time.

A parser service daemon, or a pool of them can be used instead, getting requests from the main application process.

> It's beneficial for security

... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Java has always been 'use memory to increase speed...sometimes'. You can tune it some, sure, but that's what it's known to do.

Last time I checked, I still couldn't control the max heap free ratio, because apparently that option/flag just doesn't work with Java 8's default GC.

Back when I tried running a jruby application on 800mb ram, it bloated, then started throwing "OutOfMemoryError"s and "Insufficient Class Space" or something similar. Apparently jruby was generating too many new types at runtime to accommodate rails framework. Garbage collector was pretty garbage at it's job back in 2011.

AFAIR JRuby creates a Java class for each Ruby method. Classes were part of the Permanent Generation, so sometimes this generation was too small escpecially when using Rails. It was enough to just increase the size of this generation with -XX:MaxPermSize (-Xmx doesn't increase the permanent generation's size).

The Permanent Generation was named that way since objects in there were never collected. For most applications this isn't really a problem. Running JRuby+Rails just allocated a lot of classes in this generation, so the default size was too small. But still, the permanent generation was quite small compared to the heap size.

I wouldn't really call the GC bad because of this, IMHO they were already quite good back then. And in Java 8 the permanent generation was replaced with the Metaspace, objects in there can be free'd and the Metaspace can be expanded at runtime so it's less likely to get these OOM errors for the permanent generation.

That was my experience as well. Increase maxpermsize and reboot every night.

That's mostly just a Java thing, though. Java has a weird relationship with memory.

Reply on news.ycombinator.com