Graves were a feature added for Halloween in 1.23. There are a variety of them, and they spawn randomly [most commonly in deserts, wastelands, and snowy habitats].
But not only do they give the world an eerie feeling, but you can obtain items such as wicker lamps, torches, and saddles from them. Kaalus said on his blog post on October 21, 2013: "There will be a large variety of graves to be found in the world, if you look carefully. Some with surprises…"
Dig up the land around the grave to retrieve items. But not every grave has items. It may take a while to find one that does.
Usually, graves just spawn in groups of 1-3. But when you find a graveyard [groups of 3-10], it will be well worth it. Because chances are, a few of them will hold items.