One source states "Perlite is insoluble, stable, and inert with no known effects of exposure; can be treated as an innocuous dust. Perlite is partially soluble in hydroflouric acid".
However Perlite creates an exceedingly fine dust composed of volcanic glass particles, anyone that has worked with it can testify to what it feels like to breathe it. The hazards thus need to be investigated more thoroughly, we recommend caution.
Perlite at Wikipedia