You're correct. There is one object in all of mathematics that isn't defined (you've got to start from somewhere), it is just assumed that humans implicitly understand this concept, and it's the concept of a set (denoting a collection of objects).

Along with it, certain properties of this object are assumed (among them the ability to choose an element of a non empty set - this is typically called the axiom of choice). For a full list you can take a look at http://mathworld.wolfram.com/Zermelo-FraenkelAxioms.html