Ben North (GitHub / blog), March 2018 (repository root)
Among the components of Amazon Web Services are the following two parts of their ‘serverless’ approach:
- Lambda — a ‘Lambda function’ is a self-contained piece of code which AWS runs on your behalf in response to triggers you specify;
- Step Functions — a ‘Step Function’ is a mechanism for controlling the interlinked operation of multiple steps, including invocation of Lambda functions.
While Lambda functions can be written in many languages, to write a Step Function you describe the logic as a state machine in JSON. This seems cumbersome when compared to our normal way of describing how to control interlinked computations, which is to write some Python (or C#, or Java, or…).
Based on this observation, the tools presented here are a ‘plausibility argument of concept’ for the idea that you could write your top-level logic as a Python program and have it compiled into a Step Function state machine. (I haven’t developed this far enough to call it a ‘proof of concept’.)
One of the desired properties of the system is that the source program should be more or less ‘normal Python’. It should be possible to use it in two ways:
- Run it as a Python program with the usual Python interpreter;
- Compile it into a Step Function and run in the AWS cloud.
The ability to run your logic as a normal Python program allows local development and testing.
Compile Python code to Step Function state machine
The ‘Python to Step Function compiler’ tool,
Wrap original Python code as Lambda function
pysfnc.py, reads in a file of Python code and emits JSON corresponding to the control flow of a specified ‘entry point’ function in that code. The resulting JSON is used for the creation of an AWS Step Function. Various supplied Python functions allow the programmer to express intent in terms of retry characteristics, parallel invocations, error handling, etc. Nonetheless the code is valid normal Python and executes with (mostly) equivalent semantics to those the resulting Step Function will have.
The ‘Python to Step Function wrapper compiler’ tool,
pysfnwc.py, constructs a zip-file containing the original Python code together with a small wrapper. The zip-file is suitable for uploading as an AWS Lambda function. This gives the top-level Step Function access to what were callees in the original Python code.
In the below I have omitted details like creation of IAM users, creation of roles, etc. See Amazon’s documentation on these points.
Python 3.6, with
pip install click
pip install attrs
pip install pytest # To be able to run unit tests