Agency is where OTA acts as an agent between customer and lodging supplier. The OTA takes a fee and/or percent cut of the booking. Usually the customer pays the hotel directly upon checkin, and the hotel in turn pays the OTA.
Booking.com uses the Agency model and they dominate Europe. Expedia is strong in North America and struggles to get a foothold in other regions.
Expedia now uses the agency model and merchant model. Europeans understandably do not prefer to pay for the hotel stay at the time of booking.
There are more nuances, but that is basically how it works.