As mentioned, if you're looking at something personal and zero maintenance headaches, you can head to a blog site like wordpress or blogspot and sign up. Then you'll have a nice site for yourself.
If you're looking for something professional, you need to look for server host (unless you plan to host it from your home/connection). Then you'll have to install whatever software you want to run the website with. Then you'll need to go to a domain registrar like godaddy and pluck some dough for a domain name to use. Link up the IP address of your machine (or from the server host) and the domain using the DNS IP provided and you'll be up and running.
Psyringe: Depends on what you want to do. If you just want to start a blog, you only need to register at a site like wordpress.org or blogspot.com. If you want to register a domain name, and buy some webspace, then you're looking for a web hoster and domain registrar - I think godaddy is the most popular one in the States?
langurmonkey: So I just go to godaddy.com or register.com and purchase? Then it is mine?
godaddy will give you a domain name, but nothing else. You essentially have say www.foobar.com which points to nowhere. You have to subscribe to a server hosting service if you have the dough, or open your own internet connection traffic and point the domain name to your IP essentially. Then you'll be running the website on your own.