Avadon 1 keeps track of your current difficulty (the one you are currently using on Game Settings) and the minimum difficulty used during the playthrough (used for all difficulty related medals in addition to the current difficulty). So if you start playing on Torment, decrease to Normal or Hard, and come back to Torment, then you are not eligible for the Torment medal anymore. If at any point you decrease to Casual, then you are not eligible for any difficulty related medals anymore.

It seems that the game only registers changes to the minimum difficulty used in the playthrough while you are not in combat mode, or immediately after you leave combat mode. So if you decrease the difficulty during combat, you can restore it to the original setting before leaving combat mode and the minimum difficulty used won't change.
It seems to work on Avadon 1. I didn't test it on Avadon 2 and 3. It can be useful if you get stuck in a specific fight during a Hard/Torment playthrough.