A number of systems have asteroids; however, no artifact is shown on the system screen. Should we be exploring these "empty" asteroids anyway? Will there ever be anything there worth finding if no artifact is shown?

Asteroids with artifacts are mentioned immediately when you enter a system and are listed in the target list as Asteroid.
The mass of normal asteroids (be they stone, crystal or ice) are not in the target list and can only be used to be shot up and finding a bit of material (e.g. metal, crystals) in some of them once you have bought the Ore Harvester.