JOEL - It is wrong only if your goal is exaltation in the Celestial Kingdom and if you want your relationship with your wife and children to continue beyond this life.
In God's perspective, not having a temple marriage is wrong for anyone who has already taken the first step of baptism, which is the beginning of the path to eternal life with Him. Baptism symbolizes the covenant by which people promise to come into the fold of God, to take upon themselves the name of Christ, to stand as a witness for God, to keep his commandments, and to serve him to the end.
The Lord, as his part of the covenant, is to pour out his spirit upon them, redeem them from their sins, raise them in the first resurrection, and give them eternal life (Mosiah 18:7-10; D&C 20:37). If we do not continue on that path that we started at baptism, by not getting married in the Temple, our journey towards eternal life will be cut short and God will not be obligated to bless us with exaltation in the Celestial Kingdom. Your future wife and children may be deprived of the blessings of the gospel and priesthood in the home. When you die, your relationship with your spouse and children will end, just as it will for any other family who are not members of God's kingdom.
It depends on whose perspective you are looking at on whether it is "wrong" or not. Ultimately we all have the freedom to make our own choices, based on what we are willing to live with and what our goals are for this life and beyond.