Should I tell him that I know, or will it backfire on me?

I think you need to confront him. If you can't trust your partner than who can you trust? What is the point in a trustless relationship. It will only eat you up and end up making you resentful