Kids with high math aptitude do need good teaching to reach their potential; most cannot/will not do it on their own. But there's more to it than that. We are killing their interest and motivation.
Many elementary kids with high math aptitude who suffer through the likes of Investigations and EM are left terminally frustrated with math. How many of us have watched the transformation of our kids from eager first or second graders who loved math, to irritated 5th and 6th graders who no longer can even enjoy it. And since they've checked out for a few years, many are going to find it very difficult to pick up the pieces in high school. They'll have gaps. They'll be behind. And they are not too likely to struggle back to the heights they would otherwise reach.
We are losing a generation or two of potential mathematicians. They will be doing other things.