Old server: two Xeon E5450 chips, 3.0 GHz, 8 cores total, 24 GB RAM.
New server: one Xeon E5-2690 chip, 2.9 GHz, 8 cores total, 32 GB RAM.
195 karma · joined October 9, 2006
New server: one Xeon E5-2690 chip, 2.9 GHz, 8 cores total, 32 GB RAM.
pid = fork();
if(pid == 0){
close(1);
dup(pipefds[1]);
exec(...);
}You are wrong to suggest that we grow to like anything life forces us to do. If you have an analytical mind, a job flipping burgers or pressing buttons as a tester is never going to satisfy you. You might be able to change the job to something you like more (e.g. write test scripts) but that's a whole other topic.
Some additional things I have learnt that may be useful for you are: (1) There is no one thing predestined to be your calling in life, there are several things you may like. (2) If you really want something, you'll usually get it eventually or find something that you like better.
It is useful to set a long term goal based on what you know you like so far and at least initially, while you are still in college, it is useful to state it broadly but don't tie yourself to a specific way of getting to that goal.
For example, I know I enjoy coding, technology, business and teaching. There are several combinations of these that could end up being my calling. The specific image I may have in mind is to work as an engineer / technical entrepreneur now and become a I think you have already honed in on the basics: "You have to actually do it". If it seems like there are too many things to try and they all look appealing, you just need to understand yourself better and self observation will help you there.
You are wrong to suggest that we grow to like anything life forces us to do. If you have an analytical mind, a job flipping burgers or pressing buttons as a tester is never going to satisfy you. You might be able to change the job to something you like more (e.g. write test scripts) but that's a whole other topic.
Some additional things I have learnt that may be useful for you are: (1) There is no one thing predestined to be your calling in life, there are several things you may like. (2) If you really want something, you'll usually get it eventually or find something that you like better.
It is useful to set a long term goal based on what you know you like so far and at least initially, while you are still in college, it is useful to state it broadly but don't tie yourself to a specific way of getting to that goal. NEW END.
New: 3.0 GHz Core whatever, 12 GB RAM, 64-bit FreeBSD 7.1.
5000 per second with --innodb_flush_log_at_trx_commit=0, which only writes the log to disk once per second.
The MySQL documentation claims that this configuration does crash-recovery correctly, though you may lose the last second's worth of transactions.
Same setup but --innodb_flush_log_at_trx_commit=0, a million transactions in 200 seconds, or 5000/second.
I don't know if InnoDB wrote its B-Tree to disk during the 200 seconds. Same performance even with InnoDB's buffer pool size set to 200 KB with --innodb_buffer_pool_size=200000.
The MySQL documentation claims that this configuration does crash-recovery correctly, though you may lose the last second's worth of transactions.
Such a feature would anyway be hard or impossible to use as part of a design to get fast writes and crash recovery. Crash recovery usually depends on constraints on the order writes were applied to the disk surface -- for example that all the log blocks were on the surface before any of the B-Tree blocks. Or (for FFS) that an i-node initialization goes to the surface before the new directory entry during a creat(). Drives that just provide write caching don't guarantee any ordering (much of the point of write-caching is to change the order of writes), and don't tell the o/s which writes have actually completed. So the write-order invariants that crash recovery depends on won't hold with write-caching. That's why tagged command queuing is popular in high-end systems: TCQ lets the drive re-order concurrent writes, but tells the o/s when each completes, so for example a DB can wait for the log writes to reach the surface before starting the B-Tree writes.
In our case, perhaps a pure log-structured DB could use a disk write-cache. Crash recovery could scan the whole disk (or some guess about the tail of the log) looking for records that were written, and use the largest complete prefix of the log. But we would not be able to use the disk for anything with a more traditional crash recovery design -- for example we probably could not store our log in a file system! Perhaps we could tell the disk to write-cache our data, but not the file system's meta-data. On the other hand perhaps we'd want to write the log to the raw disk anyway, since we don't want to be slowed down by the file system adding block numbers to the i-node whenever we append the log.
You might imagine that the disk would write-cache only an amount of data that it could write to the surface with the energy stored in its capacitors after it detected a power failure. But this is not the way disks work. Typical disk specs explicitly say that the contents of the write-cache may be lost if the power fails.
You may be thinking of "tagged queuing", in which the o/s can issue concurrent operations to the disk, and the disk chooses the order in which to apply them to the surface, and tells the o/s as each completes so the DB knows which transaction can now continue. That's a good idea if there are concurrent transactions and the DB is basically doing writes to random disk positions. In the log-append case we're talking about, tagged queuing is only going to make a difference if we hand lots of appends to the disk at the same time. In that specialized situation it's somewhat faster to issue a single big disk write. You need to defer log flushes in either case to get good performance.
The huge win is if you can append many transactions to the log in each rotation. To do that you have to gather up many updates per disk operation. So deferred logging is critical.
I suspect the reason PostgreSQL doesn't really support delayed log flush is that they are thinking about ACID transactions, where you really need the data to be on disk immediately. A more technical issue is that the log data must be on disk before the corresponding permanent data (otherwise crash recovery will break), and I suspect postgresql.conf's "fsync" option has the effect of not fsync()ing the log at all, which indeed would cause permanent corruption after a crash.