No. Why even think about quitting or trying to quit?
Just enjoy it.
Taking a break every now and then is good though, as is taking a break from anything.
But that is not the same as quitting.
Quitting implies you are addicted to it, or it is causing you problems, or you think there is something bad about it and try to give up but fail.
If that is the case then that's a different story.
Just treat punting as something you do for fun and that's it.
Don't let it take over your life, or finances, or emotions, and have lots of other things that keep you busy and interests and things you do for fun; work, family, friends, sport, music, movies, travel, gym, restaurant, casino, hiking, footy etc.
It's just punting. It should just be the icing on the cake, not the whole cake.