The major difference is that nuclear waste ends up in rich countries' backyards, whereas heavy metals from construction and disposal of solar panels contaminate communities thousands of miles away. Out of sight, out of mind.
My understanding is that the heavy metals are almost entirely from thin film cadmium telluride panels. Not from the much more common silicon based panels which are made of silicon, glass and aluminum for the frame. About as safe materials as one could hope for. The regular panels do sometimes contain a little lead, but this is small amounts for solder which could quite easily be replaced by lead free solder.
