Amid scandalous rumors that he left his wife because she doesn't have a penis, Joe Simpson -- father of Jessica -- recently canceled his regularly-scheduled golf date with an unidentified virile young male and instead played with a schlumpy age-appropriate friend.

A change in golf partners notwithstanding, Simpson's newly-dyed blond coif is not helping those "gayest gay in the history of gay" whispers.

On Saturday, Simpson cast aside his possible boy-toy that he'd been quietly bringing to the golf course for months -- the same dude whose name he'd been keeping off the club's sign-in sheets. Club members (snerk, we said "members") also said Joe wouldn't allow anyone to join the twosome in their weekly game of putting balls in holes (yes, we're 12).

As a source told Radar Online, “Joe has always played golf at his swanky club MountainGate on Saturdays at 12:30 p.m. with a much younger, 20-something male, who isn't a member... [he] is extremely good looking and Joe is very, very secretive about his identity.”

And in a tabloid-friendly twist, Britney Spears' former back-up dancer Bryce Chandler Hill claims to be that guy.

“Bryce claims he’s been dating Joe for roughly a year and absolutely loves to boast about it because he loves being the center of attention," said an insider close to Hill. "He’s also been quick to tell anyone that will listen that he’s using Joe to climb the showbiz ladder."

That said, multiple sources claim Hill is not the young man Simpson brings (or used to bring) golfing with him to MountainGate. This mysterious and dour young man is described as “a rugged, wanna-be skinny actor [who] looks very out of place at the country club” and seems extremely uncomfortable about being there.

Could be because he's on the down-low with a married father of two. Or maybe his backswing just needs a lot of work.