+1

I think you'll also find that it depends on the person in part too. Speaking from personal experience, (with my mother tongue being English), I found French to be near impossible to learn. Italian was a lot easier to pick up as was Latin. Although I've pretty much forgotten both now as I never use them anymore.
Mandarin, Cantonese and various other Chinese dialects were fairly easy to learn to understand what was being said, but the tones were a real challenge to reproduce when trying to speak. Not surprisingly, I soon gave up on those languages when I ran out of time to really dedicate to them.
Japanese, on the other hand, was very easy to pick up. The basic grammar is some of the easiest out there. Unlike say English, where it feels like there are more irregular grammar patterns than regular ones, Japanese only has a handful of irregular patterns. Everything else, is pretty much a simple case of following one of two rules for conjugating, etc.
ilves mentioned the differing levels of politeness as a barrier to picking up Japanese, but the "good" news here is that foreigners are simply not expected to master most levels of politeness. As long as you can differentiate between when to use a dictionary form and a ~masu form, you'll be all set.
The higher levels of politeness, termed "keigo" are something that even Japanese themselves will struggle with, and they tend not to learn the proper usage of these terms until they graduate (generally, keigo is learnt during the first month or so of employment and is taught by the employer, not at schools).

I had to drop French 2 twice in college to prevent an F on my record... can't imagine how hard these others might be for my obviously language-inept mind.
I ended up graduating with sign language.

日本語

I would say I am proficient in Mandarin and Cantonese (although English is my first language) but I didn't really find the tones that much of a problem. However the biggest problem for me is reading and writing. I suppose Chinese words are somewhat similar in style to Japanese, how would you go about reading and writing well? Because I suck at it.
Anyway, I saw somewhere (it was a Michael Palin travel show... aaargh) that Hungarian was considered to be one of the hardest languages to learn.
I actually took French and studied it for a couple of years. It wasn't hard but I was (am) a slacker and gave up in favor of Chinese. :D

There is no "easy" way. It's pretty much all about repetition. Which is probably way most Japanese these days are so-called "waapuro baka" (i.e., someone who can no longer write Japanese by hand because they use word processors* too much.
* Or mobile phones, PCs, and any other form of technology that negates the need to write by hand.

I found Finnish bloody hard.
To be honest I think English is probably the hardest. It's easy enough to get a basic understanding of but everything beyond basic conversational english is full of insane contradictory rules that only apply under certain conditions.
I before E except after C
Oh and except for those other few times
Nah binary is easy, you just need a calculator.

Having studied Chinese and Korean each for three years at my university, it can say that both languages are extremely challenging.
Chinese grammar is actually very simple and makes a lot of sense. The actual spoken language doesn't take a lot of time to pick up, even despite the tones. Some people learn tones easier than others, its just how it is. The written language, on the other hand, is incredibly difficult. Reading formal Chinese (and especially classical Chinese) is like reading a different language. Add the thousands upon thousands of characters that you have to memorize (a tiny bit easier if you're studying simplified characters), and you have a pretty damn tough language.
Korean, on the other hand is very easy to read and write, but the speaking is incredibly difficult. The pronunciation is very difficult for many people to get, and the grammar is pretty much backwards of most european grammar. Also, you have to truly understand Korean societal culture in order to speak well, you will speak differently with different people depending on their age or position. Also, Koreans love using long sentences when they speak, and because of the fluidity and speed (especially in Seoul) it can take a long time to get to the level to have a good conversation with a Korean.
I have found them both to be very challenging, but I do love studying them nonetheless :-)

That's because english is a language that recognises words based on common use and will accept the most common spelling and pronunciation on a whim. Germanic in origin followed by heavy influence by french, liking to use words with latin and greek roots for technical terms, and occasionally pulling a word from nowhere, so, yeah....
Now I took a (mandatory) french course in junior high and that was a lot more structured, most verbs conjugated exactly the same based on how they ended (er, ir, re) although there were a few exceptions. But pronunciation was fairly consistent, just memorising how to conjugate verbs was hard, I still don't actually know....
In short: "Je ne parle pas français." (I think that's right, maybe.)

French conjugation is painful and mastering it is extremely laborious. Once in a while I still need a book to make sure that I don't make mistake.

我同意。虽然我身为华人，但我本身也觉得华文比英文难学。基础是能打好，但要精通就挺难咯。
Translated: I concur. Despite me being Chinese, personally I feel it's a lot harder to study than the English Language. The basic foundations are easy enough to handle, but to be a master at it (i.e. writing) requires a lot of skill.
However, there is a trick to writing Chinese characters. Each character can usually be divided into smaller sub-characters. Once you memorize the few basic characters (and by few, I mean only about several hundred), it becomes an easier task to memorize which goes where to form what.

Luckily the english latin alphabet only has 26 characters, although this does lead to the problem of letters having far too many pronunciations.

Uhm, Chinese and Arabic maybe? Luckily the Interwebs are English-based for the most part, and English is a rather simple language to deal with at an intermediate level. Well, except for the Britons' pronunciation, of course.

Did you learn English first or Chinese first?