The costumes, the geography, the weapons and jewelry, the richness of the background - they throw all that out the window if PJ doesn't do it. In other words, "The Hobbit" would not be in the same league, would not be part of the same series of movies.
Then there's the actors. Especially Gandalf.
On top of that, while I haven't heard anyone else say this, I think that Elijah is the most logical choice to play Bilbo. He will be older than when doing Frodo, and since Frodo is Bilbo's nephew, a strong family resemblence would be logical. Some makeup to slant his look a little bit toward Ian's and bobs-your-uncle.
(Actually, I've been kind of counting on that outcome.)