I'd actually say that whatever language you learn first (after your native one) will be the hardest, then whatever's next is far easier. I know the question is more like what language would be rated hardest, but in a way it's more about what's easiest - let me explain...
It's long been understood that learning one additional language makes learning all consequent languages a lot easier as your mind is now "formatted" for inputting new languages. The easiest language to learn is the artificial language Esperanto. A study was done in which one group spent 2 years learning French as their first foreign language. A second group spent 1 year learning Esperanto as their first foreign language and then 1 year learning French. At the end of that 2 year period, the 2nd group were actually better at French than the group that'd spent the whole 2 years learning it.
More specifically to the question, I'd actually say it depends on if you include learning a new alphabet as part of the difficulty. No matter hard something like Polish is, the characters used in writing are largely familiar to anyone else from Europe, Oceania, North and South America and so on. If you want to learn Japanese, the language itself is fairly well structured, if very unintuitive in its sentence structure for English-speaking people, but you have to then learn two "alphabets" of about 28 letters each plus you'll need to know an additional 2000 kanji (Chinese characters) in order to be able to read the average newspaper.
