I did some testing with songs and discovered some interesting things:

When you use a song, the game increments a counter. If you don't use a song for a round, the counter resets. This means you need to keep singing or the song's effectiveness will start over.

If two party members are singing the same song, the counter will increase by two, but you will only get the effect once. With the Song of Healing (and probably also Destruction), the first song will be the one that takes effect, but the second song will still increase the counter before the effect is calculated.

If different songs are used, they all share the same counter. This means that you can combine different songs and the effect will be greater than the sum of the parts.

For example (assuming first round):
Song of Healing: 5% HP recovery
Song of Healing x2: 7.5% HP recovery
Song of Healing + Song of Protection: 7.5% HP recovery plus damage reduction.

For purposes of this post, I will consider the counter (c) to start at 1 and increase with every song to a maximum of 8. (This means it is 2 for the first song.) Here is how the songs work:

Song of Healing: restores 5% * (max HP) * c / 2 to the party. Tackle can prevent the healing, but will not prevent the counter from increasing. Zero is treated as a full heal.
Song of Destruction: Does 5% * (max HP) * c / 2 to the enemies. Is not prevented by Tackle. Zero is treated as 1.
Song of Protection: Reduces all damage received for the round. I think the amount of reduction is something like c/16. (According to the Japanese Elminage 3 wiki, it maxes out at about half damage received in that game.) Comes out at the start of the round, not the end.
Song of Hope: May cure status ailments. Can cure paralysis but not petrification. (Does Tackle prevent this?)

I am sure about Song of Healing's behavior; less so about the others.

Anyway, here is what I am calling the Choral strategy. Sing the Songs of Healing, Protection, and Destruction at the same time. This will quickly increase the counter, reducing the damage you take, healing the damage, and winning the battle on the 6 round (assuming enemies don't heal).

One final note: Tackle will prevent turn recovery, even if that turn recovery is negative.

Song of Destruction - what type of damage is that and can it be resisted (totally or partially)? Does it works against all types of enemies (including ghosts)?

Amount of healing I get from Song of Healing fits your formula. It is great if it scales as you say, as I was about to regret choosing it when it went all way down at around Lv. 3-4.

I haven't used Song of Destruction seriously, but my best guess is that the damage has no type and either is unresistable or can only be resisted (partially) by Song of Hope.

By the way, I believe Song of Healing triggers before Mark of Ruin's damage.

Actually, I think you didn't quite get the one counter-intuitive part of the formula. Namely, c starts at 1, not 0, and will be 2 by the time the first song effect occurs.
Also, the game actually takes MaxHP / 20 (rounded down) and then multiplies by c. This means a character with 40 Max HP will be healed twice as much as someone with only 39.

Depends how to look at it - counter increasing before or after using song. "After" sounds more intuitive. How many songs were sung? Indeed, I was fooled by "%" when removed it. it will be:
[(C + 2) * 0.5 * [MaxHP / 20]],
square brackets stand for rounding down.

So same as your formula, I just count Songs differently, like invisible buff. You start with no buff, and each song gives you 1 stack of buff (or applies 1 stack of debuff to enemy).

Meanwhile I partially "deciphered" spell table for base spell damage and healing. I wonder why they don't list this stuff in manual :)

Offensive Spells (Mage):
Balad: 8d1 (1-8).
Balados: 8d1 (1-8).
Glass: 2d8 (8-16).
Mabalad: 6d6 (6-36).
Ziakal: 6d6+20 (26-56).
Maglass: 3d12+6 (18-42).
Rabalad: 8d7 (7-56).
Argeiss: 3d8+8 (16-32).
Raglass: 4d14+12 (26-68).
Ziakalad: 8d6+45 (51-93).
Enterook Mista: 10d10 (10-100).

Offensive Spells (Cleric):
Elnam: 4d15+12 (27-72).

Some unknown offensive spell:
Holmic Gate: 10d10 (10-100)

Healing Spells:
Feiria: 4d2 (2-8).
Feirima: 4d2+15 (17-23).
Rafeireed: 10d10+35 (45-135).
Rafelima: 4d2+16 (18-24).

Actually, the counter must increase before the song is sung. If you have two characters singing Song of Healing in the first round, the first song will be the one that takes effect, but the healing will be as if it were the second song.

For spell damage, I decided to try editing it and observed a few results:
An attack spell that does negative damage is absorbed.
A healing spell that would do negative healing has no effect. (Unless the caster has Hand of Kindness, of course.)

I did some tests with Rafaireed set to restore exactly 90 HP (instead of a random amount), and I discovered three things:
1. My figure of +3 per Piety bonus (16. 17. 18. 20. 22) seems to be correct.
2. A healing spell that heals exactly zero is a full heal (just like with Song of Healing).
3. A Servant's bonus to potion healing introduces some randomness. The premade Servant was healing about 11-18 with with a potion that would normally heal 0 (thanks to an edit).