Individual drivers can be unloaded by module id. To determine the module ID assigned to a driver, use modinfo(1M). Find the driver's name in the output. The first column of that entry is the driver's module ID.
# modinfoId Loadaddr Size Info Rev Module Name ... 124 ff211000 1df4 101 1 xx (xx driver v1.0)
The number in the Info field is the major number chosen for the driver.