I got the original cd of Atomic Edition. I would like to play it in DOSBox, but can't find the best settings for it.

Can some post the .conf file the GOG version uses?

Super late response, but here's the file if you still need it:

http://pastebin.com/4gC2pxvQ

I can't say if these settings are optimal because there's a bit of flickering for some effects. Could just be a limitation of DOSBox though.

I go with opengl or ddraw for output
Then play around with scaler from none to normal3x.
ddraw looks good but comes out blocky compared to opengl and scaler none also helps with reducing
blocky graphics. Just play around until you're happy. opengl is bilinear so some blurring.
Also try eduke to play it.

# This is the configurationfile for DOSBox 0.72.
# Lines starting with a # are commentlines.
# They are used to (briefly) document the effect of each option.

[sdl]
# fullscreen -- Start dosbox directly in fullscreen.
# fulldouble -- Use double buffering in fullscreen.
# fullresolution -- What resolution to use for fullscreen: original or fixed size (e.g. 1024x768).
# windowresolution -- Scale the window to this size IF the output device supports hardware scaling.
# output -- What to use for output: surface,overlay,opengl,openglnb,ddraw.
# autolock -- Mouse will automatically lock, if you click on the screen.
# sensitiviy -- Mouse sensitivity.
# waitonerror -- Wait before closing the console if dosbox has an error.
# priority -- Priority levels for dosbox: lowest,lower,normal,higher,highest,pause (when not focussed).
# Second entry behind the comma is for when dosbox is not focused/minimized.
# mapperfile -- File used to load/save the key/event mappings from.
# usescancodes -- Avoid usage of symkeys, might not work on all operating systems.

fullscreen=true
fulldouble=false
fullresolution=1280x960
windowresolution=original
output=opengl
autolock=true
sensitivity=100
waitonerror=true
priority=higher,normal
mapperfile=mapper.txt
usescancodes=true

[dosbox]
# language -- Select another language file.
# memsize -- Amount of memory DOSBox has in megabytes.
# machine -- The type of machine tries to emulate:hercules,cga,tandy,pcjr,vga.
# captures -- Directory where things like wave,midi,screenshot get captured.

language=
machine=vesa_nolfb
captures=capture
memsize=16

[render]
# frameskip -- How many frames DOSBox skips before drawing one.
# aspect -- Do aspect correction, if your output method doesn't support scaling this can slow things down!.
# scaler -- Scaler used to enlarge/enhance low resolution modes.
# Supported are none,normal2x,normal3x,advmame2x,advmame3x,hq2x,hq3x,
# 2xsai,super2xsai,supereagle,advinterp2x,advinterp3x,
# tv2x,tv3x,rgb2x,rgb3x,scan2x,scan3x.
# If forced is appended (like scaler=hq2x forced), the scaler will be used
# even if the result might not be desired.

frameskip=0
aspect=true
scaler=none

[cpu]
# core -- CPU Core used in emulation: normal,simple,dynamic,auto.
# auto switches from normal to dynamic if appropriate.
# cycles -- Amount of instructions DOSBox tries to emulate each millisecond.
# Setting this value too high results in sound dropouts and lags.
# You can also let DOSBox guess the correct value by setting it to max.
# The default setting (auto) switches to max if appropriate.
# cycleup -- Amount of cycles to increase/decrease with keycombo.
# cycledown Setting it lower than 100 will be a percentage.

core=auto
cycles=max
cycleup=500
cycledown=20

[mixer]
# nosound -- Enable silent mode, sound is still emulated though.
# rate -- Mixer sample rate, setting any devices higher than this will
# probably lower their sound quality.
# blocksize -- Mixer block size, larger blocks might help sound stuttering
# but sound will also be more lagged.
# prebuffer -- How many milliseconds of data to keep on top of the blocksize.

nosound=false
rate=48000
blocksize=2048
prebuffer=10

[midi]
# mpu401 -- Type of MPU-401 to emulate: none, uart or intelligent.
# device -- Device that will receive the MIDI data from MPU-401.
# This can be default,alsa,oss,win32,coreaudio,none.
# config -- Special configuration options for the device. In Windows put
# the id of the device you want to use. See README for details.

mpu401=intelligent
device=default
config=

[sblaster]
# sbtype -- Type of sblaster to emulate:none,sb1,sb2,sbpro1,sbpro2,sb16.
# sbbase,irq,dma,hdma -- The IO/IRQ/DMA/High DMA address of the soundblaster.
# mixer -- Allow the soundblaster mixer to modify the DOSBox mixer.
# oplmode -- Type of OPL emulation: auto,cms,opl2,dualopl2,opl3.
# On auto the mode is determined by sblaster type.
# All OPL modes are 'Adlib', except for CMS.
# oplrate -- Sample rate of OPL music emulation.

sbtype=sb16
sbbase=220
irq=5
dma=1
hdma=5
mixer=true
oplmode=auto
oplrate=48000

[gus]
# gus -- Enable the Gravis Ultrasound emulation.
# gusbase,irq1,irq2,dma1,dma2 -- The IO/IRQ/DMA addresses of the
# Gravis Ultrasound. (Same IRQ's and DMA's are OK.)
# gusrate -- Sample rate of Ultrasound emulation.
# ultradir -- Path to Ultrasound directory. In this directory
# there should be a MIDI directory that contains
# the patch files for GUS playback. Patch sets used
# with Timidity should work fine.

gus=false

[speaker]
# pcspeaker -- Enable PC-Speaker emulation.
# pcrate -- Sample rate of the PC-Speaker sound generation.
# tandy -- Enable Tandy Sound System emulation (off,on,auto).
# For auto Tandysound emulation is present only if machine is set to tandy.
# tandyrate -- Sample rate of the Tandy 3-Voice generation.
# disney -- Enable Disney Sound Source emulation. Covox Voice Master and Speech Thing compatible.

pcspeaker=true
pcrate=22050
tandy=auto
tandyrate=22050
disney=true

[joystick]
# joysticktype -- Type of joystick to emulate: auto (default), none,
# 2axis (supports two joysticks,
# 4axis (supports one joystick, first joystick used),
# 4axis_2 (supports one joystick, second joystick used),
# fcs (Thrustmaster), ch (CH Flightstick).
# none disables joystick emulation.
# auto chooses emulation depending on real joystick(s).
# timed -- enable timed intervals for axis. (false is old style behaviour).
# autofire -- continuously fires as long as you keep the button pressed.
# swap34 -- swap the 3rd and the 4th axis. can be useful for certain joysticks.
# buttonwrap -- enable button wrapping at the number of emulated buttons.

joysticktype=auto
timed=true
autofire=false
swap34=false
buttonwrap=true

[serial]
# serial1-4 -- set type of device connected to com port.
# Can be disabled, dummy, modem, nullmodem, directserial.
# Additional parameters must be in the same line in the form of
# parameter:value. Parameter for all types is irq.
# for directserial: realport (required), rxdelay (optional).
# for modem: listenport (optional).
# for nullmodem: server, rxdelay, txdelay, telnet, usedtr,
# transparent, port, inhsocket (all optional).
# Example: serial1=modem listenport:5000

serial1=dummy
serial2=dummy
serial3=disabled
serial4=disabled

[dos]
# xms -- Enable XMS support.
# ems -- Enable EMS support.
# umb -- Enable UMB support.
# keyboardlayout -- Language code of the keyboard layout (or none).

xms=true
ems=true
umb=true
keyboardlayout=none

[IPX]
Enable=1
Connection=1
ipx=true

[autoexec]
# Lines in this section will be run at startup.

@ECHO OFF
mount C "."
c:
cls
duke3d.exe
exit

EDIT:

Just tried openglnb for output. This is now my favorite for duke 3d and DOOM . Had always used opengl or ddraw.

For flickering bug try dr.schliemann's fix here:

https://www.gog.com/forum/duke_nukem_series/duke_3d_graphic_flickering

It worked for me