The problem is that in many places there are no alternatives to driving. Taking away licenses from “those who shouldn’t have licenses” restricts their access to regular life so massively that you don’t do it unless there’s no more room to doubt the decision. The question moves from “do you meet the maximum safety standards” to “do you meet the minimum safety standards”.

The solution is to either make driving foolproof or to provide viable alternatives to those unfit (or unwilling) to drive.