what does "keep aspect ratio" do specifically? and is it for windowed or fullscreen?

Always thought it meant,leave it as is (don't change anything).

It's for fullscreen most likely, and it should add black bars on the left and right of the screen to maintain a 4:3 image (edit, or whatever the ratio is in the game's resolution), instead of stretching the image to fill the entire screen.
I think windowed mode automatically maintains the correct aspect ratio, but I'm not sure. (edit: actually, no, window resolution is also set in the config file)

also discussed here: http://www.gog.com/forum/general/keep_aspect_ratio_not_working_in_fullscreen

It corrects a game's aspect ratio to display in a 4:3 format. This applies to games that use, for example, a resolution of 320x200 pixels, stretching the image to look like it would have on an old monitor (or an approximation thereof).

EDIT: And it's for both fullscreen and windowed mode.

It does aspect correction. It only affects non-square pixel modes like VGA Mode 13h, which has a resolution of 320x200 pixels and is used by many DOS games (DOOM, etc). Recommended as such games were designed for 4:3 displays, and without aspect correction will look distorted and not as the developer intended.

See here: Dosbox.conf documentation