I have tried to make a fighter with ambidexterity and two weapon fighting, but I still keep getting a -4 proficiency for both hands when equiping weapons. What is going on!

Oh man, old DnD rules. Lets see if I get this correct.

If you use two regular sized weapons no feats, -6 main, -10 off hand
If you use a light weapon in off hand reduce by 2, so -4 main, -8 off hand
Ambidexterity ONLY reduces off hand by 4. So, -6 / -6, or -4, -4 from the above two.
Two weapon reduces both by 2. So, -4 / -8 if both regular, -2, -6 if off hand is light.
Combine both:

Two regular weapons:
-6, -10 normal
+0. +4 ambi
+2, +2 two weap spec
---------
-4, -4

If off hand is light (dagger or short sword):
-6, -10 normal
+2, +2 off hand is light
+0,+4 ambi
+2,+2 two weap spec
---------
-2, -2

Change your off hand to a light weapon. Short sword or dagger.
If later in the game, you would have plenty of boosts to counter the -4, -4. But, you still only get one single off hand attack per round anyways. So... I don't want to do anymore math. You figure out if it is worth it, and when, regarding buffs and type of off hand weapon.

Make sure to create your fighter with at least 18 strength so he'll still be effective when fighting monsters with damage reduction.