Created Unassigned: Magick.NET-Q16-x86.Native.dll staying locked till app pool recycle [1419]

One of the magick.net assembly is staying locked until we manually recycle app pool. I have tried all versions above 7.0.0.22 and it happens in every one of them. This is causing deployment errors.

VS 2015 Build warning
4>C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(4604,5): warning MSB3061: Unable to delete file "C:\Workspace\Dev\App\bin\Magick.NET-Q16-x86.Native.dll". Access to the path 'C:\Workspace\Dev\App\bin\Magick.NET-Q16-x86.Native.dll' is denied.

Octopus error
Remove-Item : Cannot remove item
11:39:05Error
C:\inetpub\wwwroot\App\bin\Magick.NET-Q16-x86.Native.dll:
11:39:05Error
Access to the path
11:39:05Error
'C:\inetpub\wwwroot\App\bin\Magick.NET-Q16-x86.Native.dll' is
11:39:05Error
denied.