JOEL - At the final Judgment, people will be assigned a degree of glory(Celestial, Terrestrial, Telestial). Further progress is believed possible within each kingdom but not to a higher kingdom. Marriage and family life, however, continue only in the highest degree of the Celestial Kingdom, allowing "eternal increase" through having spirit children.
To be a forever family, all family members must be worthy of the highest degree of the celestial kingdom. Besides that, which degree or kingdom the children go to will determine whether the parents are able to have any contact with them.
The D & C explains that those who go to the two lower categories of the Celestial Kingdom "are not gods, but are angels of God forever and ever," ministering servants who "remain separately and singly, without exaltation, in their saved condition, to all eternity" (D&C 132:16-17).
So in this sense, children who have lost their place in the highest degree of the Celestial kingdom will not be with their parents forever even though they were born in the covenant. The parents, however, may be able to visit them in the lower levels of the Celestial kingdom and in the Terrestrial Kingdom, but not the Telestial. Or the children may be "ministering servents" to them in the Celestial kingdom. Only beings from the Terrestrial world will visit those in the telestial. So if the children go to the Telestial kingdom or outer darkness I doubt they would ever see their parents(D&C 76:86-87).

Here is what Joseph Fielding Smith in The Doctrines of Salvation:
"Outside the celestial kingdom there is no family unit. That organization is reserved for those willing to abide in every covenant and every obligation which we are called upon to receive while we sojurn here in this mortal life(p. 67)
...children left without one and maybe without both parents, to be taken perhaps through the mercy of the Almighty into some other faithful family, to be adopted in such a family to be theirs through all eternity...(p. 83) All children born in the covenant belong to their parents in eternity but that does not mean that they, because of that birthright, will inherit celestial glory. The faith and the faithfulness of fathers and mothers will not save disobedient children. Salvation is an individual matter, and if a person who has been born in the covenant rebels and denies the Lord, he will lose the blessings of exaltation.
But children born in the covenant, who drift away, are still the children of their parents; and the parents have a claim upon them; and if the children have not sinned away all their rights, the parents may be able to bring them through repentance, into the celestial kingdom, but not to receive the exaltation.
When a man and a women are married in the temple for time and all eternity and then separate, the children will go with the parent who is justified and who has kept the commandments. If neither of them has kept his covenants, the children may be taken away from both of them and given to somebody else, and that would be by virtue of being born in the covenant.
A child is not to be sealed the second time when born in the covenant, but by virtue of that birthright can be transferred. (p. 91-92)