(I am going to put a content warning at the top of this post because there will be discussion of incest and sexual abuse; this is NOT a judgment of the letter writer or the situation, but it does come up in my answer, so if you are sensitive to that, take care of yourself.)
Bi poly young woman, have been seeing a much older man for a long time. He has an (adult) daughter around my age who I find extremely attractive and he and I started joking around but he ended up admitting he wouldn't mind at all if I started seeing her too (obvs separately, without him involved - no incest here!!) I'm freaking out because I like the idea too - is this OK? Am I OK? Would I be some kind of twisted horrible pervert for dating both a man AND his daughter?
To answer your actual question - no, you're not a twisted horrible pervert if you do that; try to let go of the self-recrimination and guilt. If no one is getting hurt, then you're not doing anything wrong. And you can decide that this isn't something you want to do without deciding that doing it, suggesting it, or wanting it is morally wrong.
I would caution you to take a step back and make sure this situation is 100% healthy for everyone involved. A father who jokes about his daughter's sex life and encourages someone else to sleep with her might just be a guy who is your partner trying not to put limitations on who you can see; or he might be someone with a problematic investment in, well, his daughter's sex life. I wasn't there when it came up in conversation and I genuinely don't know whether the 'joking around' came from an organic, honest place or whether he is bringing it up as part of an unhealthy fantasy or perspective of his. Is he encouraging this because he seems to be excited about it, or just supporting you in your attraction? Only you can make that call! But you need to make it with clarity.
Remember that parent/child power dynamics can be really complex, and can complicate issues of consent. Be really sure that everyone involved is okay with this arrangement and that it's healthy for everyone. Even then, you might make a mistake - you are not psychic, and you may misinterpret someone's true feelings or intentions, especially if they themselves are unclear. This is NOT to say that you are morally responsible for protecting the daughter from getting into a situation that isn't right for her, just saying that this is tricky territory and you should be aware of your own limitations when it comes to recognizing coercion or ambivalence.
If you go through with this, you'll need to talk really clearly about maintaining healthy boundaries. For example, it's critical in healthy polyamory for there to be some openness about sexual health; you need to be able to talk to your partner about what's going on with your body, which includes some details about your other sexual encounters. But do all three parties feel that it's appropriate for a father to have that insight into his daughter's sex life? If not, how will you manage this? What extra precautions need to be taken to keep everyone psychologically and physically safe?
Talk also about how your "polycule" will function. Will all three of you spend time together? If not, how will you manage things like family get-togethers when the father might want his partner and his daughter there? If so, how does the daughter feel about being in a semi-sexual/romantic role ("girlfriend" and "metamour") while simultaneously being in the daughter role? Would all three of you cuddle up on the couch with you in the middle? Is she okay seeing her father as her metamour? How does she feel about physical affection in the presence of her dad? Between you and the dad? Between you and her? How does the dad feel about being physical/sexual/romantic with you in front of the daughter? Is everyone okay with those boundaries, or does someone feel stifled or controlled?
Are you three prepared to manage the questions and judgments from others? Will you be keeping the arrangement on the down-low? How will things be presented at family functions? Do you have a united front and a prepared explanation? What if a family member has a problem with it and threatens to withhold their children from visiting the dad, or refuses to extend wedding invitations? Will one person ask the others to cave for the sake of the family (just tell everyone she's your roommate, upsetting Grandma isn't worth it, it's just one wedding weekend), or will you three stand up together and demand acceptance? Are you all on the same page about that?
This is a complicated situation, and it's okay if you decide it isn't worth it. But it's also okay if you work it out and decide that it's doable and worth doing. But it will take extra effort, precautions, preparation and self-knowledge to make sure it's safe and healthy for everyone.