Here's a quick video of Pam Ronald (UC Davis Plant Pathology) on the labeling of "genetically modified, GM" food.
Knowing simply that a crop has been genetically engineered gives you no information about its safety. In order to make a safety judgment, the consumer would need to know what specific gene was put into the plant - but this raises complications.
Her example: You could label all non-organic U.S.-grown papaya as containing a small genetically engineered fragment of the papaya ringspot virus (the source of disease resistance that saved the industry) - but then you'd also have to label organic papaya that are naturally infected with the virus (and therefore have orders of magnitude more foreign viral nucleic acid and protein than the "transgenic" ones).
She didn't have a solution in this short sound bite, but she pointed out that alerting consumers that their foods are full of viruses would freak them out unnecessarily. Plant viruses can't infect humans.