I have tried to pickpocket the merchant in the Hive and now I cannot buy the Battered Tankard for the Old Mebbeth quest.
Can anyone help me?
I tried to use the Pacify Spell but doesn't work.

Did you try if killing him works ?

Another option is skipping this quest, Old Mebbeth and her quests are totally optional.

Aren't Old Mebbeth's quests required to play as mage? Otherwise how do you learn spells?

I don't think you can turn NPCs un-hostile once they've gone hostile. You can create the tankard with the CluaConsole:

C:CreateItem("TANKARD")

As PeterC2 says, doing Mebbeth's quests are necessary if you want to learn magic from her. There are other Mage trainers later in the game but not until you reach the Lower Ward.