JOEL - Marriage and family life continue only in the highest degree of the Celestial Kingdom, allowing "eternal increase" through having spirit children.
To be a forever family, all family members must be worthy of the highest degree of the Celestial kingdom. Besides that, which degree or kingdom the children go to will determine whether the parents are able to have any contact with them.
The D & C explains that those who go to the two lower categories of the Celestial Kingdom "are not gods, but are angels of God forever and ever," ministering servants who "remain separately and singly, without exaltation, in their saved condition, to all eternity" (D&C 132:16-17).
So in this sense, children who have lost their place in the highest degree of the Celestial kingdom will not be with their parents continuously forever even though they were born in the covenant. However, the parents may be able to visit them in the lower levels of the Celestial kingdom and in the Terrestrial Kingdom, but not the Telestial. Or the children may be "ministering servents" to them in the Celestial kingdom. Only beings from the Terrestrial world will visit those in the Telestial. So if the children go to the Telestial kingdom or outer darkness I doubt they would ever see their Celestial parents(D&C 76:86-87).

Here is what Joseph Fielding Smith said in The Doctrines of Salvation:
"Outside the Celestial kingdom there is no family unit. That organization is reserved for those willing to abide in every covenant and every obligation which we are called upon to receive while we sojurn here in this mortal life(p. 67)
...children left without one and maybe without both parents, to be taken perhaps through the mercy of the Almighty into some other faithful family, to be adopted in such a family to be theirs through all eternity...(p. 83) All children born in the covenant belong to their parents in eternity but that does not mean that they, because of that birthright, will inherit celestial glory. The faith and the faithfulness of fathers and mothers will not save disobedient children. Salvation is an individual matter, and if a person who has been born in the covenant rebels and denies the Lord, he will lose the blessings of exaltation.
But children born in the covenant, who drift away, are still the children of their parents; and the parents have a claim upon them; and if the children have not sinned away all their rights, the parents may be able to bring them through repentance, into the Celestial kingdom, but not to receive the exaltation.
When a man and a women are married in the temple for time and all eternity and then separate, the children will go with the parent who is justified and who has kept the commandments. If neither of them has kept his covenants, the children may be taken away from both of them and given to somebody else, and that would be by virtue of being born in the covenant." (p. 91-92)

There is no way we can comprehend what our existance will be like in the Celestial kingdom or how we will feel about our children or other family members who did not make it there. But I am sure we will not experience sadness or pain. We will experience perfect joy and will understand and accept everything that happens to us there and to our children whereever they are.
Everyone who is saved into any of the three kingdoms will be happy that they are there and will be perfectly comfortable existing in the assigned kingdom.
I do not think that the memories of our children will be taken away from us, but we will accept and completely understand whatever happens to them, knowing that they are happy where they are.
When you think about it, the only family members that will really be together in the highest degree of the Celestial Kingdom will be husband and wife couples. Your Celestial children will be off with their own spouses creating their own universes.
Parents of wayward children also need to remember that there is still the opportunity for them to repent and accept the gospel in the spirit world after they die if they did not receive a fair chance at it while here on earth. According to the statement above by Pres. Smith, parents may through repentence, be able to help their children restore their Celestial status.