Basically, the Onsen is not that expensive, it's more like affordable.
It depends on where you are going to visit, but how about trying to start off Oedo Onsen Monogatari (in Tokyo)? It's not an authentic type of Onsen, but it's more like amusement type, however, the good point is that you can try many kinds of Onsen at one time!
Staying in onsen ryokan costs roughly ¥10,000 ~. It can go higher if it's nice, new, big room with private outdoor onsen bath, etc. You can find low price ryokan starts from ¥10,000~ at most onsen districts such as Hakone, Atami, Kusatsu, Beppu, etc.