I think that if you don’t feel comfortable with riding this horse then you shouldn’t ride him. Or at least have a break from riding him.
Maybe having some lessons on another horse at a different time, or riding a friends horse would help you build your confidence back up. And when you feel up to it, have another go on this horse.
Talk to your teacher about it. They want you to do well and enjoy your riding! I’m sure they’ll be more than happy to talk to you and help you out.
It defiantly sounds like you’re loosing confidence in your riding, and I know from personal experience that this can really ruin the joy you get from riding.
I really think you need to talk to your instructor about this! Be honest with yourself.
Good luck, and remember to enjoy your riding :)