I think the OP formulated it very well. And I never did like the world buffs. It is fun to compare and check raidstats, it doesn't define your skills, but it's fun. With world buffs those stats are a bit more of a "gamble" since you might die on many encounters without doing much wrong.
And the argument that if we don't want buffs, don't use them then. That simply just removes any fun with raidstats :) since those that won't use the buffs will just drop down like rocks in water. If they work, people are pretty much obligated to take them or get kicked/loot banned etc from the guild ^^
World buffs gets a big nay from me.