When I write, I follow different rules for two different kinds of modifiers: adjectives and "adjectivals" on the one hand; adverbs and "adverbials" on the other.
I'm certain I follow a (third?) set of rules for a third category -- sentence modifiers -- but I still don't consciously understand what sentence modifiers are, so I can't take that thought any further. (Katharine's explanation is at the end of this post.)
I'm going to steer clear of sentence modifiers for the time being.
Grammar books, including at least some linguistically-informed grammar books, tell us that adjectives must be put next to the words they modify, but adverbs can go all over the place.
The black cat is sitting on the roof.
The cat is sitting black on the roof.
The cat is sitting on the roof black.
Adverbs are different:
The black cat is sitting happily on the roof.
The black cat is happily sitting on the roof.
The black cat is sitting on the roof happily.
Happily, the black cat is sitting on the roof.
The same principle holds for adjective & adverb phrases & clauses:
The cat that is black is sitting on the roof. (adjective clause)
The cat is sitting that is black on the roof.
The cat is that is black sitting on the roof.
The cat is sitting on the roof that is black.
Adverb clauses can move around:
The cat is sitting on the roof because she likes high places.
Because she likes high places, the cat is sitting on the roof.
The cat, because she likes high places, is sitting on the roof.
And even, in some cases:
The cat is, because she likes high places, sitting on the roof.
(I wouldn't write that sentence, but I'm pretty sure I've seen the occasional adverb clause dropped inside a 2-word verb.)
According to grammar books - at least according to the ones I'm reading - the words "because she likes high places" are an adverbial clause modifying the verb "is sitting."
I find that explanation confusing, but I don't find the rule confusing. I follow the rule automatically and unconsciously, and I always have. I also follow, automatically and unconsciously, the rule that says adjective clauses must go beside the nouns they modify (though dangling participles are something of a temptation, which I think is interesting.) Importantly, if we're talking about English teachers imposing artificial, made-up rules they learned in books upon captive students, I didn't learn either rule from a book.
I learned these rules from talking and reading. I'm not just a native speaker of English. I'm a native writer.
Katharine on sentence modifiers
I'm starting to think maybe the reason I find adverbials confusing is that the category grammar books call "adverbials" includes the category linguists call "sentence modifiers." I don't know.
In any event, here is Katharine on sentence modifiers and Groucho Marx, and this explanation does make perfect sense to me, which is a great relief!:
It's true that modifiers are generally placed next to the things they modify. But sometimes it's the entire sentence that is being modified, in which case the modifier can go at the beginning or at the end.
In the ordinary interpretation of "I shot an elephant in my pajamas" (before the "How he got there, I don't know" clarification), "In my pajamas" is a sentential modifier. That is, it most obviously characterizes the circumstances of the elephant shooting. In the bizarre interpretation (which becomes obvious only after the clarification), "in my pajamas" is a modifier of object noun "elephant." As such, it cannot be moved to the beginning of the sentence. Thus, "In my pajamas, I shot an elephant." is unambiguous.
The tradeoff is between stylistic concerns (e.g. FedupMom's) and concerns about clarity. Depending on the overall context, sentential modifiers placed at the ends of sentences can be misinterpreted as verb phrase or object noun modifiers, which both tend to go at the ends of verb phrases. Since the end of a verb phrase is often also the end of the main sentence, you often can't tell what an end-of-sentence modifier is modifying from word order alone.
With Dick Cavett, I discussed sex. (unambiguous)
I discussed sex with Dick Cavett. (ambiguous; example from Steven Pinker).