To disable DLC delete the corresponding folder in the following directory:

..\GOG Games\BioShock Infinite\DLC

You do not have to delete the folder actually. You can just rename it or move it elsewhere in case you want to play the DLCs later. This way you do not have to download and reinstall the whole game once again.

Thank you! By getting the complete edition, I thought I was going to have a “complete” experience. But forcing all kinds of abilities on me in the beginning to make the game easier is anything but having a "complete" experience. That was one of the dumbest things I've ever seen developers do.