Both are possible.
You have to decide what you want.
1. If you want to charge for the material qty to vendor :
-->Then you can scrap the material at vendor end and post a debit note from FI side.
--> You can create a return PO for the vendor for the qty of material which you want to charge (Price will be material price).
2. Return to warehouse :
--> Simply you have to do 542 movement type from vendor to your own unrestricted, then stock will come under your warehouse. Then you have to decide you will do for the damaged material.