RDouglass1: Is Scanning of Asteroids Illegal??
No. The only thing you need to worry about when scanning asteroids are nearby pirates.
The kind of thing you are describing can happen in one of 3 scenarios.
1) You've claimed a ship which had attacked one of the commonwealth faction's assets and has been marked as a target for that faction's patrol fleets (this usually happens with pirate ships). This will automatically make you an enemy of that factions when said patrol fleets end up in the same sector as your newly targeted for destruction ship.
Getting out of this one is tricky. You either reload a save and forget about claiming the ship, or you claim it but sell it for at least some profit before any patrol fleets get in that sector and start treating you as a hostile.
2) You've claimed a ship which has messed up your default friend/foe settings. To correct this you need to go to Advanced -> Global Commands and reset the friend/foe status for each faction. Optionally you can then also transmit these settings to all your reachable properties.
3) You're repeatedly crashed or attacked a commonwealth ship and ignored its warnings (but I don't think that's the case). This can also happen if one of your AI controlled ships decides to go on a rampage and attack friendlies for some reason (though that is rare) or if one of your freighters accidentally crashes into a friendly ship and manages to destroy it by doing so. Save often, these things will happen once in a while :).
These are a couple of quirks you'll have to get used to in the X3 universe.