To change the FOV for different resolutions, it says to edit the engine.u file in various ways. But how do I open the file properly?

Windows is acting like it doesn't know what program to use to open the file.

Notepad, presumably? Seems like the text shows a lot of unusual symbols when I open in notepad so I'm feeling skeptical.

This is where they discuss the file edits to affect the changes: https://www.wsgf.org/dr/star-wars-republic-commando

According to wsgf, that is a binary file and it should be opened with a hex editor such as HxD or similar.