As you age, you will start to see changes occur in your skin, including your chest, neck and face. One of the main causes of sagging skin is the thinning of the outer layer, or epidermis. As your skin thins, it loses layers of fat that kept it looking youthful and plump. With less fat, skin begins to sag and look more aged, wrinkled and fragile.