I saw a couple of threads about this problem and players were suggesting to load a previous save and kill every siren before you actually destroy the nest. The problem is I'm way ahead in the game now and wouldnt like going back 15 hours to clear this site. Is there another solution to this problem?

It's a bug. There are a specific number of harpies that must all be killed. The problem with this site is that one (or more) of the harpies can become trapped underneath the terrain so you can't see them, but they still exist and so the script doesn't fire to bring the people back. I don't know of any cure that works, hence it's always prudent to save before you enter into any battle.

I've had some success meditating at sites to get monsters / bandits to spawn in where they should be.

Found a solution. It requires opening the console thou.

"You can enable the console in one of two ways - changing a config file or by using a mod.

After enabling it, load your save, go to the ruined inn, open the console, and run the following command -

addfact(sk75_sirens_dead)

After a few seconds the liberation sequence should play."
-from Zingmar.info

In 1.31 version bug still exists.

What MixiumDraGoM wrote -> works.

Zingmars described on blog with more details.

Based on that I recorded on youtube how to apply this fix (on PC only). Unfortunately can't post links so just search for:

Ruinned inn - bug and fix (Witcher 3, Skellige)

Hope someone will find it usefull. Cheers.