I haven't got it but it should be only when on, that is why there are different types of clothing for different buffs, I could be wrong though

Hi! All bonus attributes from clothing stack and it applies to all ninjas in your line-up. So do not need to worry about needing to use one specific clothing.

~Jib

The attribute is permanently added to your stats no matter which outfit you are wearing at the time. Clothing can be stacked, and so if you have lots of costume it will gives a huge boost to your stats. Only three permanent outfit are available at these moments. Ceremonial Kimono, Wild West & Summer Youth Sai.

Thank you Jib :)