Sure, of course you can reallocate some of your points after building a library,but it's not worth the micromanaging.

As to "the point", there is none, that's just how those buildings function. Such restrictions make for a more interesting game.

When you put the slider down to zero all the static bonuses will still be applied to the same skill.

You can allocate magic bonuses however you like (from shrines, temples, etc) but not research bonuses.

The buildings are not supposed to be identical, that's why. +1 research is, intentionally, less valuable than +1 mana, because it is less versatile. If you have no mana allocated to research, the +1 research is materially inferior because of that lack of versatility.

Just look at the various farming, production, and happiness buildings. They can effectively be treated all as +production - farming buildings let you change farmers to workers, and happiness buildings let you change rebels to workers. Wouldn't it be a bland and lame game if they were all just explicitly called +production bonuses and you didn't have to worry about how to manage things?