I've been monogamous my whole life and when I met him he was never in a poly relationship before. When we started talking a year ago I told him that I'd try it, but when I fell in love with him I feel like everything changed. He doesn't want a relationship with other people, he told me he wants a physical relationship with them. But I can't help but feel jealous and insecure about it. I just feel numb and stressed.
In general, if something in your life makes you feel "numb and stressed" - a job, a movie, a relationship, an event - you should take major steps to get yourself out of that situation.
The first step is to talk to your partner about how you feel. You agreed to try polyamory, but have discovered that it isn't working for you. He might be willing to close the relationship for the sake of your relationship. But he might not. It hurts to have to end a relationship because of a fundamental incompatibility, but it happens. That's the point of dating - to figure out whether you're compatible. It's why we don't get married at the first spark of interest.
You have learned a lot about what does and doesn't work for you. Now you have to act on that information.