Monastic attire helps monastics keep a dignified image and helps both monastics and laity remain mindful of their conduct. So it's absolutely not a good idea for monastics to wear lay clothes when they are not reciting sutras or performing rituals at the monasteries.