If you have a retainer holding the lifers in then you must have a late model engine with a roller cam. NO i would not change the lifters and this goes for either a flat tappet or roller lifter set up. A new flat tappet cam, then yes new lifters.
The valves would only be replaced if the seat is in to bad of shape to resurface. Again this is one of those points where you ask yourself how far do I go. Stainless vales, port the heads and so on.