Can I be poly if I only want to be with certain people. I'm not poly in general but I'm in love with multiple people, like I'm only poly for them if that makes sense. Does that count?
That is exactly what being polyamorous means. Being polyamorous means you are interested in a relationship with multiple specific people, not everyone on the entire planet!
Monogamous people don't think "hm, I can't see myself in a monogamous person with my hairdresser or my coworker - maybe I'm not really mono!" Straight men don't think "oh no, there are women out there who I don't want to date - do I count as a straight man?"
Of course you only want to date the people you want to date. Of course you can only see yourself being polyamorous with the people you want to date polyamorously. That's totally fine. You're in love with the people you're in love with - and it happens to be multiple people - so you're polyamorous. It doesn't matter how you feel about anyone else!
Sure, maybe there are people you'd be happy in a monogamous relationship with. But forget about the maybes. In this reality, in this universe, you're in love with multiple people and want to be in a polyamorous relationship with them. That's all the information you need!