TDD AWS – the Moto Library

Moto is a library that allows mocking AWS services when using the Python Boto library. It can be used with other languages as stated in the documentation – standalone server mode. I don’t have direct experience with that feature, though, as Python is my language of choice for coding PoCs or data projects in general.

The TDD way in the Cloud

I don’t need to preach about the benefits of using TDD to design and test your components. In any case, if you are still not convinced, check out books like:

• Test Driven Development: By Example (Addison-Wesley Signature Series (Beck))
• Architecture Patterns with Python: Enabling Test-Driven Development, Domain-Driven Design, and Event-Driven Microservices.

Developing for the cloud comes with its own set of challenges. For instance, testing. Not easy when you need to use services that are not locally available.

The appearance of libraries like Moto has made testing much more manageable. Like any library, it has its peculiarities, but the learning curve or resistance is not exceptionally high, especially if you have previous experience with Pytest or other testing frameworks.

Prerequisites

I assume you have previous knowledge of AWS, Boto, Python, TDD and Pytest. In this case, I’m providing some complete listings to learn by example, not the entire exercise, though, so you can fill any gaps and enhance your learning experience.

Installing Moto

It’s straightforward –

$ pip install moto[all] - if you want to install all the mocks
$ pip install moto[dynamodb]

To mock AWS services and test them properly, you will need a few more dependencies –

$ pip install boto3 pytest

Mocking DynamoDB – Setting up

Let’s go beyond the canonical examples and find out how to mock DynamoDB, the excellent AWS NoSQL database.

As shown in the following code listing, we’ll create the books table with the PK id. The function mocked_table returns a table with some data. Later on, this table will be mocked by Moto.

test_helper.py

import boto3
import json

def mocked_table(data):
    dynamodb = boto3.client("dynamodb")
    table = dynamodb.create_table(
        TableName='notes',
        KeySchema=[
            {
                'AttributeName': 'id',
                'KeyType': 'HASH'
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'id',
                'AttributeType': 'HASH'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 1,
            'WriteCapacityUnits': 1
        }
    )
    
    table = boto3.resource('dynamodb').Table("books")
    with table.batch_writer() as user_data:
        for item in data:
            table.put_item(Item=item)

    return table

Moto recommends using test credentials to avoid any leaking to other environments. You can provide them in the configuration file conftest.py, using Python fixtures –

conftest.py

import os

import boto3
import pytest

from moto import mock_dynamodb2

os.environ['AWS_DEFAULT_REGION'] = 'eu-west-1'

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'

Now, we are ready to start designing and developing.

Mocking DynamoDB with Moto and TDD

We will code the function get_book that retrieves a particular Item from the table. Following the TDD cycle, we have to code the test first, make it fail etc … not showing here the entire cycle, but just a few steps to get the idea.

The test unit would look like this –

get_books_test.py

import pytest
import json
import boto3

from moto import mock_dynamodb2

book_OK = {
    "pathParameters":{
        "id": "B9B3022F98Fjvjs83AB8a80C185D",
    }
}
book_ERROR = {
    "pathParameters":{
        "id": "B9B3022F98Fjvjs83AB8a80C18",
    }
}

@mock_dynamodb2
def test_get_book():
    from get_book import get_book # dont't change the order
    from test_helper import mocked_table # must be imported first
    
    dynamodb = boto3.client("dynamodb") # before getting the client 
    
    data = [{'id' : 'B9B3022F98Fjvjs83AB8a80C185D','user' : 'User1'}]   
    mocked_table(data)

    result = get_book(book_OK)

    item = result['Item']
    assert item['id'] == 'B9B3022F98Fjvjs83AB8a80C185D'
    assert item['user'] == 'User1'

As you can see, nothing very different from a Pytest test unit, except for the Moto annotations and AWS’S specific code.

Let’s increase the test coverage, ensuring that the functionality “Item not found” is working as expected.

@mock_dynamodb2
def test_get_book_not_found():    
from get_book import get_book # dont't change the order
    from test_helper import mocked_table # must be imported first
    
    dynamodb = boto3.client("dynamodb") # before getting the client 
    
    data = [{'id' : 'B9B3022F98Fjvjs83AB8a80C185D','user' : 'User1'}]   
    mocked_table(data)
    
    result = get_book(book_ERROR)
    
    assert 'Item' not in result # book not found

OK, now that we have the test unit by design, we need to write the function get_book. We can start with something basic that satisfies the import and the method signature. By the way, I showed the test unit fully coded, but you can do this gradually and code the essential function initially.

get_book.py

import json
import boto3

dynamodb = boto3.resource('dynamodb')
tableName = 'books'

def get_book(id):
    return {'id' : 'B9B3022F98Fjvjs83AB8a80C185D','user' : 'User1'}

To execute the tests –

$ pytest

The test will fail. The book that we are returning is not in the expected format. So let’s add the DynamoDb calls – that will be mocked by Moto.

get_book.py

import json
import boto3

dynamodb = boto3.resource('dynamodb')
tableName = 'books'

def get_book(id):
    table = dynamodb.Table(tableName)
    result = table.get_item(
         Key={
              'id': id,
         }
    )
    return result

Now the test unit will pass 🙂

Some important things to point out:

• The test case is built using the Pytest framework, then Moto for mocking the calls.

• @mock_dynamodb – marks this method to indicate to the Moto framework that dynamodb will be decorated.
• The methods we want to mock must be imported before getting the client instance so that the Moto framework can decorate them properly. This is very important; if you don’t do that, the test won’t work.
• The assert methods come from the Pytest framework – check the docs for more examples.

Conclusion

Testing is not easy and can be tedious; for some people, even a nuisance. Using TDD – or BDD – changes your mindset entirely because you are designing your system, not only testing. But this is something that shouldn’t be news to you. TDD and BDD have been around for a while.

Not for the cloud, though.

Testing in the cloud is not accessible; it’s all about integration and cost. Libraries like Moto helps to alleviate that, and I have to say that it does pretty well.