In a recent discussion, the question was asked (paraphrased for clarity):
"How does music theory apply to songwriting or improv"
In an attempt to clarify this, I came up with the following, bordering-on-nonsensical response. It is not quite perfect, but close... More importantly, it made me laugh and I hope it does you. Here goes:
Think of a key as a family.
Each chord represents a child of that family. The parents are the scale. So each child, has pieces of the parents ... therefore a similarity ... But, unique in and of themselves. No two are the same. Some, get along REALLY well with certain members of the family. Some not so much, but none of them truly hate one another. This key, is like the Brady Bunch!
A song is like an episode.
The other 12 keys are cousins to the Brady's. Some are more closely related than others. Some members of these closely related cousin families, actually look a LOT like one of the Brady's ... and get along really quite well. These cousins come on family trips, in fact, strangers think they are one of the kids. Sometimes these cousins will invite their favourite Brady kid over for a sleepover. Now the focus of the 'episode' is on THAT cousin's family and their house ... maybe just for a momentary part ... or it could be for the rest of the episode. They could even go to other houses from there ... but they are usually brought there by one of these 'friendly' cousins (pivot chord) who is welcomed.
These more distant other cousins and their families are more removed from the Brady's ... Some, to the point where they bear zero resemblance. Several of these distant relatives actually have a dislike, even a hatred, for the Bradys. These characters will not work at all with most (if not ALL) of them. So they stay to themselves and live in their closer family circles.
The moral of this fanciful tale:
Just like people, certain notes and groupings just plain work.
Others just do not.
Theory, is the writing down and the formalizing of all this. Like drawing out a family tree hierarchy chart.
No musician is truly making progressions up as they go. Even if it seems like - or SAY they are. The chords are grouped, whether we like it or not, naturally by your ears. Because they just flat out work. Over centuries, scholars have written all these relationships down. Made groupings and formalized the presentation of all this information. Now they teach it universally, to any who wish to know.
When more advanced musicians compose or improvise (to use our analogy), they know which family members get along... So armed with this knowledge, they can skip comfortably through the family tree. Recruiting the best member for the task at hand. The more time they spend getting to know the different family members, the more comfortable they are inviting them over for the BBQ or perhaps unload the cube van! Some players never bother going beyond their small tight 'family' circle. It is comfortable, feels right, they know who to call and life is good. It just works.
So your understanding of the dynamics of the musical family tree, really relies on how "social" you choose to be.