Where Factoring Can’t Help