Very interesting analogy, thank you! When I initially learned about linear regression, i learned that to capture non-linearities, instead of choosing a more complex, non-linear hypothesis function, I can just come up with "arbitrary" features for my data set. Basic example: house price calculation. Obvious features are square_meters, age, n_rooms, ... But I can make even this linear model learn complex connections by transforming or combining these input features and add them as additional inputs, such as n_rooms * age, or log(square_meters) or whatever.

What you're explaining sounds very similar. Is it, or am I understanding it wrong? (Idk why it's so hard for me to understand this attention thing...)