Can you give some more details on the language/libraries that you are using? I am just getting started on a 6DOF game...
I am planning to use C++ to code up the engine. So the easiest option is to use Boost.statechart to implement a Hierarchical Finite State machine based AI. I can always switch to a hand coded state machine later on if the performance of boost is not good. See an old article on Halo 2 AI: http://www.gamasutra.com/gdc2005/features/20050311/isla_01.s...
Havok Behaviour seems to be a game middleware covering the same area. So it might be an option if you are targeting PC (See licensing details of the same) even if it is not open source. I am not considering it as most of the features of it seem to be targeting the FPS game type which is a bad fit for a 6DOF game.
Apart from the high level AI functionality that can be implemented using Boost.Statechart, there is also the need for good pathing. For a 6DOF game reduces to a problem of generating non intersecting B-Splines a between source and target. Space is mostly empty ;) This is relatively simpler compared to the pathing algorithm needed for an FPS. Off the top of my head, I would use something like Boost Graph library to do an A* search on a Nav nodes as the simplest solution for an FPS game. You should be able to provide various heuristics using multiple vistors to customize the A* searches. Another option is to use a dedicated pathing library like Path Engine ( http://www.pathengine.com/ ) or Havok AI ( http://havok.com ). Unfortunately both are paid libraries. A Nav Mesh based pathing provided by Path engine or Havok AI will be much better than the Nav Node system, but with adequate coding and good level design, it should not too different from a nav node system. Eg: Left 4 Dead 1 & 2 seem to be using a Nav Mesh based system versus Alien swarm which seem to use a Nav Nodes based system for pathing.
Planning of AI is a hard problem without having the constraint of being real time. I would instead use strategically placed volume triggers (Ghost/Phantom object of your physics library) to customize the AI NPC behaviour (triggering AI HFSM transitions) or to trigger a preset sequence of scripted actions (Cinematics, sort of). Combine that with a decent nav nodes system and some randomness, we get sufficiantly 'intelligent' behaviour. This, however, becomes a data heavy approach to game level design. A full planning engine becomes a code heavy approach. The trick seem to be to strike a balance between the two :D