Now you’re thinking with actors!

Luciano,

I’m relatively new to Akka, but I’ve been designing and building systems in Erlang for years. If I might speak a bit toward the architecture of actor based servers:

On Thu, May 5, 2011 at 6:02 PM, Luciano Fiandesio wrote:

Hi Patrik, I have been reading past posts on this mailing list about pooling and load balancing of actors and I’m uncertain about the best practices here.

Given the scenario where multiple remote typed actors of the same type have to process incoming messages in parallel (and execute blocking and potentially time consuming operations), I understand I have the following options:

• use an Untyped Dispatcher or Load balancer (http://doc.akka.io/ routing-java)
• create an actor per message, and set the maximum number of actors running at the same time (not sure how to do that using the Java API)

Which approach you take a largely a matter of need, but, in general, actors are to be considered as having remarkably little overhead: you may make millions at a time without trouble. Whereas thread-per-connection servers are limited in their ability to scale, actor-per-connection servers are not. If you can encapsulate a bit of workflow as a concurrent activity create an actor that performs that work. In some instances your workers will be long-lived—because the work is infinite—and in others short—because the work is finite.

I imagine that you are a morally right and ethically straight man, that you do not hang about with the dregs of humanity: let me tell you about 4chan. It’s a collection of named message boards, each board having threads where users can make posts in a flat hierarchy, sharing images and racial insults. The threads are temporary—they’re maximum lifetime fluctuates, but they last no more than a day. Further, boards have a bounded capacity, they act like LRU caches: threads that see no active posting in a busy board are eventually terminated. The hierarchy looks something like this:

   4chan
   |-- r9k
   |   |-- against_dog
   |   `-- against_god
   |-- a
   |   `-- anime_millhouse
   |-- b
   |   |-- queen_boxxy
   |   `-- the_game
   `-- s
       `-- moar

At the top we find ‘4chan’. This is the application supervisor and runs infinitely. The entities in the set {r9k,a,b,s} are the boards and are the children of 4chan. At boot time, 4chan creates its children in a 0neForOne arrangement, restarting them as required. The boards, too, are infinite supervisors: their children are the temporary threads. The board threads are created dynamically in response to user inputs—when they die (either due to reaching the end of their lifetime or because of a bug) the board thread does not restart the actor. Each board thread manages its own state and lifecycle—likely by exposing a Jetty resource for REST operations. User input validation, captcha verification, tri-force rendering and so on are all managed solely by a single board thread actor instance: only at creation and forced destruction does a board ever interact with its thread.

Being that we are modeling 4chan, the boards must create only a limited number of threads. Internal to each board actor, maintain an LRU cache. For each NewThread event—achieved by having the board act as a Jetty resource or by tethering a slave Jetty actor to the board actor—simply create a new thread actor, place it in the LRU cache and feed PoisonPill to any thread which falls out of the LRU. Board thread timeouts may be achieved by forming the thread as an FSM with an internal self-destruct state reached eventually by careful timeInterval delays. If the number of board threads may be unbounded, abandon the internal LRU cache and trust only to thread self-destruction.

If we assume that each 4chan board thread must make an entry in an FBI database for each poster’s IP address—and that this takes a non-trivial amount of time—we can easily maintain performance by doing the following:

• Create a new ‘snitch’ actor for each post.
• Sending a message with the IP payload from the board to the snitch.
• The snitch performs the long write to the FBI database.
• Job done, the snitch kills itself.

You’ll note, we might potentially make an unbounded number of snitches. If that’s a problem, keep a simple counter: increment on snitch creation, decrement on destruction. It is best to assume, however, that you don’t need such management: the actor style—at least as espoused in the Erlang community—is to do the simplest thing which will work, only complicating the solution once you have shown that a loaded system has scaling issues. Actors are a cheap resource: make as many as you want. Pools of actors are a hold-over from thread-based system design than from actor-based systems.

Until, of course, you need one.

• create a number of typed actors and link them to a supervisor actor that uses work stealing dispatcher (again, not sure how to do that).

Are the options exhaustive? Which is the recommended approach, considering that the actors will execute blocking operations?

I would suggest that you create one actor per concurrent block of operations, watched by a OneForOne supervisor. If an individual child dies, save off the actor’s initial state—you can do that in Akka, right?—and restart the concurrent block of operations in a new actor.