I figured this out now. I was sotflocked and had to restart because I didn't have enough stamina (yellow bar), so I couldn't jump high enough with the fairy.
SPOILER ALERT (PACIFIST MODE)
This happened because I had tried to find the "pacifist route" at first, but had then defeated the Dirt King (I think that was because the pacifist route itself was "softlocked" as I had used a key in the wrong place and was therefore missing the key to go through the pacifist passage). I dion't know how I missed the stamina upgrade though, possibly a bug in the game, or an earlier version of it? Maybe the upgrade gets removed from the map when you take the wrong door.
I then did a normal game defeating all the bosses and have now started on the pacifist route again. I think the game is meant to be played this way (you should do the normal route first so you get to explore all the special abilities which you won't have on the pacifist route - pacifist is harder, so it's better to do it with some experience under your belt). Now the pacifist route feels pretty much like being on rails; since going through the secret passage in the cave, two of the boss fights have simply been removed, I didn't have to choose anything. Let's see if this continues.
One thing that had me confused about pacifist mode, is that you have to kill some slugs to get the key to the secret passage, which kind of opposes the concept of pacifist. So from where I am now in the game (not finished yet), it seems that pacifist really only applies to the bosses, not the small monsters.
