Indeed, you can classify objects directly from the whole image. However, we do caution you not to overuse this method. Your training set can become biased to a few images and thus the resulting classifier can become biased to those training images and not be robust across the whole population of objects.
One use case in which this method is recommended, or at least very useful, is when the penetrance of your phenotype is low, i.e. if you only have a few objects (that you know of!), perhaps in some positive control images, that display your phenotype. Then it is reasonable to use these positive controls for your training set in order to discover other, rare objects in your data that you may not have found otherwise.