It seems to be pretty heavy on object allocation, eg:

    BABYLON.Vector3.prototype.add = function (otherVector) {
      return new BABYLON.Vector3(this.x + otherVector.x, this.y + otherVector.y, this.z + otherVector.z);
    };

I do a lot of WebGL work and I'm constantly fretting about avoiding allocations. For instance, compare to gl-matrix's operator:

    vec3.add = function(out, a, b) {
      out[0] = a[0] + b[0];
      out[1] = a[1] + b[1];
      out[2] = a[2] + b[2];
      return out;
    };

A little more awkward, but you're only allocating the vector once and then modifying in place wherever possible. THREE.js takes as similar approach with THREE.Vector3.