Can I create a custom subject area from a custom data model in BI publisher?

You can only create a custom subject area from the objects that are available in Application Composer. Joins that you've build on your custom SQL may or may not be available during custom subject area creation, depending on how the objects you're using are related. So, it will really depend on what custom SQL you're using on your data model.

In addition, data models aren't likely to use data security so they don't have any restrictions. By contrast, custom subject areas apply data security according to the roles and access granted from the application.