Donki (famous discount store) or Okashi no Machioka (おかしのまちおか) sells snacks at cheap prices.
I'm not sure what you are looking for by saying "exotic candy" but if you are looking for something like Neru Neru Nerune (ねるねるねるね), you can find in supermarkets. Those are usually for kids so you have to look lower parts of the shelves!
If you are talking about "Amezaiku" the one that is created with scissors and hand, it seems impossible to find by the low price. http://www.ame-shin.com/en/
But if you are interested in Sushi-candy or some kinds of similar kit, you can find them at the most of the supermarket with very reasonable price!