Database installation is common task in setup programming.Here are the steps to install sql server database with setup using custom action. the summary is following
1. create a text file that contains a SQL statement to create a database, Tables, functions and stored procedures
2. Add installer class to execute sql statements
3. Create deployment project for Primary output for the Existing project.
4. Create custom installation dialog for database server.
5. Create custom action for installing database, build setup project and install it.
For more details, See here.