Where the shoulder seam should sit
Short answer: On the edge of the shoulder, where the arm begins. Set inside that and the garment pulls. Set outside it and the shoulder looks wider.
The simplest fit test
Put a finger on the bony point where shoulder meets arm. That is where the seam should sit. This single check tells you more about fit than chest width does, because everything else hangs from the shoulder.
When the seam sits too far in
The garment pulls backwards, tightens across the shoulder blades and binds when you reach forward. A larger size does not fix it if the cut is too narrow, only a different cut does. See bamboo t-shirts for men.
When the seam sits too far out
The sleeve hangs on the upper arm and the shoulder line disappears. That is intentional on oversize and drop shoulder styles but a fault on an ordinary cut. See bamboo t-shirts for women.
Raglan is an exception
Raglan sleeves have no shoulder seam. Here the seam runs diagonally from neck to armpit, and you judge fit by how the sleeve falls instead. See sweat clothing.
Frequently asked questions
Can a shoulder seam be moved
Yes, but it is an expensive alteration. Choosing a different cut is easier.
Does the rule apply to shirts too
Yes, and even more strictly, because shirt fabric does not stretch.
What about oversize styles
There the seam sits on the arm on purpose. Judge the sleeve length instead.