There are several light (compared to pandas) xlsx/xls libraries for Python, but none that I know of have a simple read-to-dict API.
openpyxl has good xlsx support and is easy to use. This seems to be an abstraction layer on top of both. Unfortunately this will use a lot of RAM working with large files.
yes. if you are only going to be working with CSV files then this is overkill
It is true. I usually use it in tandem with flask / fast-api to parse both: CSV and XLSX. In the near future I will add XLS (yes, someone still uses it) and also the open document format.