There are four different levels: Game, Contest, Test, and Challenge.
The only Game you will ever see is on Emerald Island, on the hill behind the Temple of the Moon. You need 25 points in the appropriate attribute to win (Just use the Day of the Gods pedestal and you'll get those attribute points automatically), and you get 3 skill points for it.
Contests are the brass braziers that you've probably found in Harmondale, Erathia, Tularea, etc. You need 50 points in the appropriate attribute to win, and you get 5 skill points for winning.
Tests are stone pillars with red-orange flames. 100 attribute points wins you 7 skill points.
Challenges are stone pillars with white flames. 200 attribute points wins you 10 skill points.
Note that once you've won, say, a Contest of Might, you cannot gain skill points from any other Contest of Might. If Harmondale and Erathia happen to have contests of the same attribute, there is no use taking advantage of both.