When the access patterns of a service are such that there are a lot more reads than writes, a common practice for scaling it horizontally is to have the service talk to a master database only for operations that result in state changes, while all read-only operations are performed against one or more read slaves. Those read replicas are only eventually consistent with the master.
This practice is an easy solution if you don't want to or cannot go the full CQRS way. All you need to do is maintain two separate data sources and point all your write operations to the master, and all your read operations to your slave.
However, how do you make sure that you're actually putting your writes where your master is? It's easy to accidentally get this wrong, and usually, you will only find out at runtime, when the read slave is denying your service to perform its writes.
Wouldn't it be nice if you could already verify at compile-time that your operations are hitting the correct database?
With Slick 3, the latest version of Typesafe's functional-relational mapper, this is actually very easy to do, provided you know a bit about Scala's type system and Slick's notion of database actions and effect types.
In this blog post, I will explain all you need to know in order to restrict the evaluation of Slick database actions based on their effect types, resulting in the elimination of yet another source of bugs.
The full example application on which the code snippets in this blog post are based is available on GitHub.
Database actions
Whether you want to query a table, insert or update a row, or update your database schema, in Slick 3, each of those things is expressed as a DBIOAction[+R, +S <: NoStream, -E <: Effect].
R is the result type of the action, and S indicates the result type for streaming results, where the super-type NoStream refers to non-streaming database actions.
For this article, those two type parameters are not of importance, however. Let's look at the third parameter instead.
The Effect type
E is some type of Effect and describes the database action's effect type. Slick 3 defines the following sub-types of Effect:
trait Effect
object Effect {
trait Read extends Effect
trait Write extends Effect
trait Schema extends Effect
trait Transactional extends Effect
trait All extends Read with Write with Schema with Transactional
}
The Effect type is a so-called phantom type. This means that we never create any instances of Effect at runtime. Rather, the sole purpose of this type is to give additional information to the compiler, so that it can prevent certain error conditions before running the application.
If you use Slick's lifted embedding syntax for creating database actions, those actions will always have the appropriate subtype of Effect. For example, if you have a table statuses, you might implement the following StatusRepository:
class StatusRepository {
def save(status: Status) = statuses.insertOrUpdate(status)
def forId(statusId: StatusId) = statuses.filter(_.id === statusId).result.headOption
}
Here, the type of action returned by save is automatically inferred to be DBIOAction[Int, NoStream, Write]. The action returned by forId, on the other hand, is DBIOAction[Option[Status], NoStream, Read].
Effect types of composed actions
If you compose several database actions using one of Slick's provided combinators, the correct intersection type will automatically be inferred. For instance, a common approach for updating the state of an aggregate is to have an application service load the aggregate from a repository, perform some business logic on the aggregate, and then save the updated version back to the repository. The application service is providing the transactional boundary for changing the state of the aggregate. This may look similar to the following, hopefully with a more complex business logic than in this oversimplified example:
def categorize(statusId: StatusId, newCategory: String) = {
val actions = for {
statusOpt <- statusRepository.forId(statusId)
result <- statusOpt match {
case Some(status) =>
val newStatus = status.copy(category = newCategory)
statusRepository.save(newStatus).map(_ => Right(newStatus))
case None => DBIO.successful(Left("unknown status"))
}
} yield result
actions.transactionally
}
Here, the inferred effect type of the composed action – actions.transactionally, will be:
DBIOAction[Either[String, Status], NoStream, Read with Write with Transactional]
Evaluating a database action
A DBIOAction merely describes what you want to do – in order to actually have it executed, you need to run it like this:
class StatusReadService(database: DatabaseDef) {
def statusesByAuthor(author: String, offset: Int, limit: Int): Future[Seq[Status]] = {
database.run {
statuses
.filter(_.author === author)
.sortBy(_.createdAt.desc)
.drop(offset)
.take(limit)
.result
}
}
}
If you haved used Slick 3 before, you will probably be familiar with this. Let's look at the signature of run as defined in DatabaseDef:
def run[R](a: DBIOAction[R, NoStream, Nothing]): Future[R]
Having an effect type of Nothing means that Slick accepts database actions with any effect type, not caring about whether it is a Write, a Read, or something else.
This is perfectly fine for a generic library. However, when you are working with a master and a slave database, it means that your code will look like this:
class DatabaseModule {
val masterDatabase: DatabaseDef = Database.forConfig("databases.master")
val slaveDatabase: DatabaseDef = Database.forConfig("databases.slave")
}
class StatusReadService(database: DatabaseDef) {
// ...
}
class StatusService(statusRepository: StatusRepository, database: DatabaseDef) {
// ...
}
On the type level, you are not able to differentiate between your master and slave databases, so performing an undesired effect against one of your databases by accident is a very real issue.
Luckily, with its Effect type, Slick 3 gives you all the compile-time information you need to implement a restriction of effect types for your own application.
Restricting effect types
In order to restrict the effect types that are allowed for a certain database, we are going to attach a role (e.g. master or slave) to each database and associate certain privileges to each role that will then be checked when trying to run a database action – all at compile time.
Roles
The first thing we need to introduce is another phantom type called Role:
trait Role
trait Master extends Role
trait Slave extends Role
Like Slick's Effect, our Role trait will never be instantiated in our application.
Privileges
Instead, Role will appear as a type parameter in yet another new phantom type:
@implicitNotFound("'${R}' database is not privileged to to perform effect '${E}'.")
trait HasPrivilege[R <: Role, E <: Effect]
The HasPrivilege phantom type is meant to provide implicit evidence that a certain role R is allowed to perform the Slick effect type E.
The @implicitNotFound annotation allows us to provide a custom error message for the case that no implicit evidence of HasPrivilege can be found, where it is required.
We are going to spell out the privileges for our two roles to the compiler like this:
type ReadWriteTransaction = Read with Write with Transactional
implicit val slaveCanRead: Slave HasPrivilege Read = null
implicit val masterCanRead: Master HasPrivilege Read = null
implicit val masterCanWrite: Master HasPrivilege Write = null
implicit val masterCanPerformTransactions: Master HasPrivilege ReadWriteTransaction = null
As you can see, while we do provide evidences in terms of implicit vals, we don't actually create any instances of HasPrivilege. Since the code that will make use of our evidence will never work with our implicit evidences at runtime, we can safely assign null here.
Also, please not that Slave HasPrivilege Read is just another notation for HasPrivilege[Slave, Read].
Unfortunately, we have to provide an implicit evidence for every combination of effect types we want to allow.
In this example, we want to allow combining reads with writes and transactions, but it's not allowed to combine reads and writes without also using a transaction.
Check your privileges!
Now, in order to actually restrict the database actions that can be run according to the role of the database, we need to introduce a wrapper around Slick's DatabaseDef – as we saw earlier, the latter does not care about the type of effect, and of course, it doesn't know anything about our roles.
Hence, we are introducing a class DB:
class DB[R <: Role](databaseConfiguration: DatabaseConfiguration[R]) {
private val underlyingDatabase = databaseConfiguration.createDatabase()
def run[A, E <: Effect](a: DBIOAction[A, NoStream, E])
(implicit p: R HasPrivilege E): Future[A] = underlyingDatabase.run(a)
}
Our new wrapper type has a type parameter R that specifies its role, and it creates the underlying Slick DatabaseDef from an instance of DatabaseConfiguration with the same role, which we will look at in a moment.
For now, the important thing is the run method on our DB class, which looks very similar to the Slick's run method we saw earlier.
The crucial difference is that our run method has a second type parameter E that specifies the type of effect of our action, and that our DBIOActions effect type parameter is that E instead of just Nothing.
Moreover, our run method has a second, implicit parameter list, with evidence of HasPrivilege[R, E], i.e. that our database role R is privileged to execute the effect E.
Instead of using DatabaseDef directly, we will now always make use of our role-annotated DB.
Providing role-annotated databases
To achieve that, we introduce a type DatabaseConfiguration which, like DB, is annotated with a role:
sealed trait DatabaseConfiguration[R <: Role] {
def createDatabase(): DatabaseDef
}
object DatabaseConfiguration {
object Master extends DatabaseConfiguration[Master] {
def createDatabase() = Database.forConfig("databases.master")
}
object Slave extends DatabaseConfiguration[Slave] {
def createDatabase() = Database.forConfig("databases.slave")
}
}
The DatabaseConfiguration is the one place where we interact with the untyped outside world, reading our database connection information for the respective key, databases.master or databases.slave. Hence, the only place where we can still get things wrong is in our application.conf configuration file, if we accidentally provide the wrong database host, for example.
Our database module providing the master and slave databases to our application will now look like this:
class DatabaseModule {
val masterDatabase: DB[Master] = new DB(DatabaseConfiguration.Master)
val slaveDatabase: DB[Slave] = new DB(DatabaseConfiguration.Slave)
}
Unlike the previous version, the masterDatabase and slaveDatabase fields are now properly typed, and our service implementations that made use of DatabaseDef before must now explicitly pick a database with the correct Role for their purposes:
class StatusService(statusRepository: StatusRepository, database: DB[Master]) {
// ...
}
class StatusReadService(database: DB[Slave]) {
// ...
}
Oops, I used the slave for a write...
To verify that all of this has the desired effect, let's make our StatusService use the slave database:
class StatusService(statusRepository: StatusRepository, database: DB[Slave]) {
// ...
}
When we try to compile this, we will receive this nice error message from the compiler:
'com.danielwestheide.slickeffecttypes.db.Slave' database is not privileged to
perform effect 'slick.dbio.Effect.Write'.
Plain SQL queries
For plain SQL queries, of course, Slick cannot infer any effect types automatically. Hence, if you need to fall back from lifted embedding to plain SQL queries, you have to annotate the resulting database actions explicitly:
def statusesByCategory(category: String, offset: Int, limit: Int): Future[Seq[Status]] = {
val action: SqlStreamingAction[Seq[Status], Status, Read] =
sql"""select id, created_at, author, text, category from statuses
where category = $category
order by created_at desc limit $limit offset $offset""".as[Status]
database.run(action)
}
If you don't do provide an explicit type here, the actions's effect type parameter will be inferred to be Effect, the super type of all effect types.
Custom effect types
Since Effect is not a sealed trait, you may introduce your own effect types and prevent certain databases from performing those effects.
For example, you may want to disallow certain expensive queries to be performed against the master database. To do that, you could introduce an effect type ExpensiveRead, and only allow slave databases to run actions with that type:
trait ExpensiveRead extends Read
implicit val slaveCanDoExpensiveReads: Slave HasPrivilege ExpensiveRead = null
Just as with plain SQL queries, you can annotate your database action explicitly to be of type ExpensiveRead:
def statusesByCategory(category: String, offset: Int, limit: Int): Future[Seq[Status]] = {
val action: SqlStreamingAction[Seq[Status], Status, ExpensiveRead] =
sql"""select id, created_at, author, text, category from statuses
where category = $category
order by created_at desc limit $limit offset $offset""".as[Status]
database.run(action)
}
If you accidentally use a DB[Master] in this read service, you will get a compile error.
Of course, once you have to start annotating your database action explicitly in order to benefit from these compile-time checks, you are prone to another source of errors. Failing to annotate your actions correctly may lead to the same kinds of runtime errors we wanted to prevent.
Conclusion
In this article I have shown how to use Slick's Effect type, together with a few other phantom types, in order to have your compiler help you verify that you run your Slick database actions against the correct database. While it's possible to use this technique with plain SQL queries and custom effect types, the greatest benefit will come in cases where you only use lifted embedding and the standard effect types.
Thanks a lot to @missingfaktor who collaborated with me on developing this technique on top of Slick 3 and gave a lot of valuable input.