Before going to answer this first we have to know the various types of materials that are present in an USB drives.
The major materials of the USB drives are copper,silver and to some extent there will be gold plating also.
It contains the alloy like Brass also.
Irrespective of this it contains the integrated circuits[ICs] which are made up of semiconducting materials mostly like silicon.
From the above we get that a USB drive does not contain any type of magnetic materials.The data in the USB drive are stored in the form of electric charges .
Whenever a USB drive is taken towards a magnet and placed gently close to it,the magnetic field can't affect the data stored inside the drive as it is not magnetic in nature.The rest magnet cant not affect the the charge particles present inside the drive.
But if the magnetic filed has great strength and there will be any relative motion between drive and magnet,then it will induce a force on the drive which has still less impact on the drive. A magnet can hamper the stored data if that will be magnetic in nature.