Cloud-Compute-Cannon and Haxe
The Bionano research group at Autodesk needed an open source tool for performing scalable computes (e.g. molecular simulations) in the cloud. The problem it solves is “I want to run 1000 simulations and get the results quickly, and I don’t have access to a high performance cluster (or I don’t know how to use the cluster)”. We built cloud-compute-cannon (CCC) for this. This blog describes some of the technology behind it.
Architecture and technical decisions
Cloud-compute-cannon is a client/server application. The server is installed on the users cloud (or their local machine) and creates workers to perform compute jobs. The server needs to:
- Install locally and on cloud providers (e.g. AWS or GCE via their SDKs)
- Interact with Docker daemons via the remote API
- Interact with remote machines via SSH
- Talk to a database (that stores state)
- Interact with a thin CLI client
- Potentially other, as yet unknown, requirements
Instead of worker machines pulling jobs from a queue, CCC works by pushing jobs to workers. The disadvantage with this approach is that if the servers goes down, the system stops working. The advantage is that the workers are extremely dumb: they are just bare CoreOS machines with nothing extra installed. Node.js can easily monitor many many workers in case they go down. By making the workers dumb, there is a single place for upgrading the system (simplicity was a goal here).
Node.js was a pretty easy choice for the server because of the large third party library support. Redis was chosen for the database because its simplicity and crucially, it had scripting support. Scripting means no concurrency issues as redis lua scripts block other requests.
The added advantage of Node.js was using the same codebase for the client and server.
What is Haxe?
Currently, Haxe compiles to the following targets:
|Name||Kind||Static typing||Sys||Use Cases||Since|
|Flash||byte code||Yes||No||Games, Mobile||alpha (2005)|
|Neko||byte code||No||Yes||Web, CLI||alpha (2005)|
|ActionScript 3||source||Yes||No||Games, Mobile, API||1.12 (2007)|
|C++||source||Yes||Yes||Games, CLI, Mobile, Desktop||2.04 (2009)|
|Java||source||Yes||Yes||CLI, Mobile, Desktop||2.10 (2012)|
|C#||source||Yes||Yes||Mobile, Desktop||2.10 (2012)|
|Python||source||No||Yes||CLI, Web, Desktop||3.2 (2015)|
|Lua||source||No||Yes||Games, CLI, Web, Desktop||3.3 (2016)|
Although Haxe has many very useful language features, the features most important to this project are:
1) Compile time typing
Functions, collections, and variables are typed, and checked at compile time. For example, the compiler will prevent you from calling a function with an Int argument when it’s expecting an array. For a large-ish codebase of a cloud application this is crucial, as it reduces the number of bugs that make it into functional testing (and functional testing scalable cloud applications is very time consuming).
In Haxe, you cannot compile with the last line. The compiler has caught a mistake the developer wrote.
Haxe macros are incredibly powerful, and the full capabilities are beyond the scope of this blog post. For this project they automatically generate bindings between the CLI client and the server.
On the server, there are ‘services’ where the methods correspond to RPC (remote procedure calls), where the @rpc decorator marks it as a remote API endpoint:
On the client
The above code generates (at compile time) a function that creates a JSON-RPC call to the remote server if the CLI command is called. E.g. calling the CLI client with the jobs sub-command:
creates the following JSON and sends it to the server (via http):
The server maps the call to the correct function, calls the function, then sends the result back in the HTTP response. The CLI interacts with the server API, but it could easily be another server.
The important point is that new services have the corresponding CLI client commands automatically generated. There is no need to touch the client code ever again, and there is no chance that a typo will accidentally break client-server communication.
3) Code completion
I just really like code completion. Code completion is handled by the Haxe compiler, so all text editors and IDEs can easily add sophisticated code completion.
These are classes and data structures that do not exist at run-time, only at compile time. In Cloud-compute-cannon, they are used to maintain specific compile-checked enum types between the server and the database scripts.
The redis scripts are in Lua, which is a new Haxe target! However the Lua target is not yet mature, so the database scripts are written manually, and embedded as raw strings in Haxe classes. This has the advantage of some compile time checking of constants and abstract types.
The following is a Haxe abstract enum. It only exists at compile time, at runtime the values are just strings.
However, the values can be embedded in a Lua script:
And this Lua script is embedded in a Haxe class (this example is contrived for simplicity):
Then if we change the JobStatus abstract, all the values are compile time checked. This means that it is impossible for a simple typo in a string value to result in a value being sent between the server and the database.
When the Lua target stabilizes, the scripts may get written in Haxe and compiled. This would allow better code re-use, and allow compile-time checking of the actual types and objects being created, stored, and send from within the database.
This demonstrates some of the power that the Haxe toolkit provides:
- Flexibility: write code for multiple platforms without losing the power of a compiler.
- Future proofing: as new platforms become available, the Haxe toolkit can target them.
- More robust code: compiler prevent simple errors, and make maintenance (e.g. refactoring) of large code bases much easier.