"R" is usually just for fighters, most shuttles and transports may be excluded depending on the game (which one are you playing and which version?).
You will have to target them manually using T and Y to scroll the targets or use button 2 for selecting the target in your sights.

I believe the E key may target the nearest enemy attacking you, while the R key just targets the nearest enemy.

R targets nearest enemy fighter.
E targets nearest enemy anything attacking you.

Okay I pressed 'E' many times and ended up ramming into an enemy transport without ever targeting the unit.

Seriously how do you actually target enemy units? I couldn't give a damn how NOT to target them.

"E" works only for fighters too.

Any ship that isn't a fighter or saved in the presets (shift + relative F-keys to assign, they vary between games, IIRC TF uses F5, F6 and F7) has to be targeted manually either by scrolling targets with the Y and T keys, if a main mission objective there is the O key in some games, or by putting it in your sights and tap joystick button 2 or the right mouse button.

Is this the Windows version of X-wing? It seems like they left some of the code where shuttles can target you from TIE Fighter. You have to target it manually, using joystick button 2, or scroll through targets with the T key.