I think there are 2 things it's important to bear in mind for those worrying that it's too soon to lift restrictions.
Firstly, the idea was never to completely eliminate the virus, it was simply to give the hospitals some breathing room so that they didn't collapse completely with so many people needing treatment all at once. Flattening the curve, I believe they called it, and that's been achieved.
And secondly, people are now a lot more conscious of the risks, and of the precautions to take. People wear masks in a lot of places (although I get the impression that's nowhere near as widespread in the UK as it is here in Spain for example), wash their hands frequently, use hand gel, don't shake hands, keep a safe distance from other people, communal areas are disinfected much more frequently, etc. All of which should hopefully mean that even with the restrictions being eased, the virus shouldn't be able to spread anywhere near as easily and as quickly as it did the first time around.