Does the GOG edition of Myst: Masterpiece Edition install or require the program QuickTime?
(I seem to remember Myst using it, but I may be wrong...)

I didn't install Quicktime for Riven, and I didn't have any issue. I have VLC instead.

Yes and you can get the newest version of Quicktime here:
http://www.apple.com/quicktime/download/

If you're still experiencing video problems, try this:

In QuickTimes control panel change the video mode to "Safe mode (GDI only)"
You find QuickTime's control panel in your system's Control Panel => QuickTime (on 64-bit Windows you must click the "View x86 Control Panel Icons" button on the left side of the Control Panel window before the QuickTime icon appears)

If this doesn't fix your problem, you can try using an older version of QuickTime (uninstall the current one before)
GOG provides a download for QuickTime 6.52.
To set "Safe Mode" in QuickTime 6 look in the Video Settings section in the drop-down menu of QT.

Thank you for the info! ^_^

You really don't need QuickTime. ScummVM has support for Mohawk Engine games. You will need to install the nightly build, as Myst support isn't compiled in the stable version. But it works well.