Jindai Botanical Garden is big and has various flowers and plants. It's in the western side of Tokyo, a bit away from central but nice and quiet. Jindai Soba noodles are also famous there.
You will need to take a little day trip from Tokyo but Hitachinaka Park in Ibaraki and Ashikaga Flower Park in Tochigi Prefecture are both wonderful. If you are visiting in May, Hitachinaka Park is filled with blue nemophila flowers and Ashikaga Flower Park lets you see the beautiful wisteria flowers.