Collision detection absolutely affects performance, if you can run with 100 objects naively, but you can run with 10000 objects by being efficient, your performance has been affected. Computational power is never enough!

How are you storing your objects in the grid container, is it actually a container or just an array with an index pointing to the object in some kind of object array? It's possible to do the grid without actually creating it as a datastructure, you just "hash" the point you are at:

  px = (int)(p.x-grid_min.x)*grid_delta.x;
  py = (int)(p.y-grid_min.y)*grid_delta.y;
  hash = px + py * grid_width
  cell_list[hash] += [p.index]

Where p is your object and grid_delta is the cell size. The cell_list could be implemented a couple ways, but if you think of it as a hash table with each entry as a list of objects in the cell, then all you need to do to get potential colliders is calculate the hash of an object and get back all the indices.

This is a technique we are using to speed up neighbor checks in some 3D particle simulation code, it works quite well on the GPU but its simple and you already have a grid in your code.

Another option would be a quadtree, I haven't looked, are there not JS implementations already out there?