So i've played like 3 hours maybe and been through the town and into the mine but was wondering is the game world much bigger and I just haven't been to the other areas yet? Or is it all in these few areas?

Hard to say since we don't know exactly what you visited. For example, you didn't mention the beach, coastline and forest to the south, which should be accessible from the start. There are alson areas (like the second beach, the quarry, the magical forest... that can be accessed only after you upgraded your tools or repaired the bridges
Did you check the map? It should give you a fair idea of what you haven't discovered yet.

The main area, the outdoors, is not large, there's not much to explore, but I'm fairly sure it was designed this way on purpose, since days are short. Depending on your objectives (houses and stores are not always open, unlike some games) you'll need to use your horse frequently to travel faster, or the coffee cups with magical properties hah. In the late game even faster ways of traveling will be available.

But I have to admit it would be great if CA expanded the desert with much more places to visit, even if it goes against the Harvest Moon world style, that served as inspiration for the valley.

The underground areas is where the exploration shines, and since most levels are randomly generated, it doesn't make mining a repetitive and boring activity like milking cows. It doesn't feel like a job.
The desert mine is basically infinite, don't bother trying to reach the end.

Unlike Harvest Moon the time doesn't stops when you're in one of the mines, so you obviously should plan well, look if you don't have any pending quests that you want to complete first, if you care about socializing with the villagers, check the birthday list in town square to see if you won't miss a birthday, etc.
Some players like going to the mines in raining days, because they've little to do in the farm so they can go there quickly and spend their whole day getting precious ores and improving combat skills.
Always bring something to recover your health, the punishment for dying in the mines is brutal!

In an old build of the game, the mine walls could be destroyed to create tunnels and explore, also you could create a raft and move through the underground water. Unfortunately it was cut.