In order to post a members-only entry in an asylum, you will need to select the "Friends" security level on the update page [http://www.insanejournal.com/update.bml]. Only the members of the asylum, while logged in, will be able to view the protected entry.
The "Private" and "Custom" security levels cannot be used while updating in an asylum.