Client Side Encryption#

Overview#

DynamoDB supports encryption at rest (Server-Side Encryption) and uses SSL to encrypt the transit data (Encrypt on the fly) by default.

Some advanced users also want to encrypt the data before it is sent to DynamoDB (Client-Side Encryption). pynamodb_mate provides an elegant way to encrypt your data before it is sent over the network. You can use different encryption keys for each DynamoDB item attribute.

How it works#

pynamodb_mate uses the pycryptodome crypto library under the hood. Internally, it always serializes your data into binary, encrypts it, and then sends it to DynamoDB. For fields that you still want to be able to query on, you use determinative=True. The same input data will always result in the same encrypted data, and it uses AES ECB. It is proven to be NOT secure for man-in-the-middle attacks, but you can still use it with DynamoDB because the DynamoDB API uses SSL to encrypt the data in transit. For determinative=False, the same input will result in different encrypted data, and it uses AES CTR.

Define attribute to use Client Side Encryption (AES)#

[1]:

import pynamodb_mate.api as pm
from rich import print as rprint # this is for this demo only

ENCRYPTION_KEY = "my-password"

class ArchiveModel(pm.Model):
    class Meta:
        table_name = f"pynamodb-mate-example-client-side-encryption"
        region = "us-east-1"
        billing_mode = pm.constants.PAY_PER_REQUEST_BILLING_MODE

    aid = pm.UnicodeAttribute(hash_key=True)

    secret_message = pm.attributes.EncryptedUnicodeAttribute(
        # the field level encryption key
        encryption_key=ENCRYPTION_KEY,
        # if True, same input -> same output (less secure),
        # so you can still use this field for query
        # ``filter_conditions=(ArchiveModel.secret_message == "my message")``.
        # if False, same input -> different output (more secure),
        # but you lose the capability of query on this field
        determinative=True,
    )

    secret_binary = pm.attributes.EncryptedBinaryAttribute(
        encryption_key=ENCRYPTION_KEY,
        determinative=False,
    )

    secret_integer = pm.attributes.EncryptedNumberAttribute(
        encryption_key=ENCRYPTION_KEY,
        determinative=True,
    )

    secret_float = pm.attributes.EncryptedNumberAttribute(
        encryption_key=ENCRYPTION_KEY,
        determinative=False,
    )

    secret_data = pm.attributes.EncryptedJsonDictAttribute(
        encryption_key=ENCRYPTION_KEY,
        determinative=False,
    )

# create DynamoDB table if not exists, quick skip if already exists
ArchiveModel.create_table(wait=True)

You just need to pass the unencrypted data to the DynamoDB ORM model, and pynamodb_mate will automatically encrypt it before sending it to the DynamoDB API.

[2]:

msg = "attack at 2PM tomorrow!"
binary = "a secret image".encode("utf-8")
data = {"Alice": 1, "Bob": 2, "Cathy": 3}
model = ArchiveModel(
    aid="aid-001",
    secret_message=msg,
    secret_binary=binary,
    secret_integer=1234,
    secret_float=3.14,
    secret_data=data,
)
model.save()
print(f"preview the DynamoDB item: {model.item_detail_console_url}")
print("you will see that the raw data in DynamoDB is encrypted")

preview the DynamoDB item: https://us-east-1.console.aws.amazon.com/dynamodbv2/home?region=us-east-1#edit-item?table=pynamodb-mate-example-client-side-encryption&itemMode=2&pk=aid-001&sk&ref=%23item-explorer%3Ftable%3Dpynamodb-mate-example-client-side-encryption&route=ROUTE_ITEM_EXPLORER
you will see that the raw data in DynamoDB is encrypted

You can print the raw data in DynamoDB without using pynamodb_mate. You will see the encrypted data in binary format.

[3]:

from boto_session_manager import BotoSesManager

bsm = BotoSesManager()
res = bsm.dynamodb_client.get_item(TableName=ArchiveModel.Meta.table_name, Key={"aid": {"S": "aid-001"}})
rprint(res["Item"])

{
    'aid': {'S': 'aid-001'},
    'secret_integer': {'B': b'K\x02\x9c\xf4\x8c8@\xcfn\xce\xb7NV\x8d\rH'},
    'secret_binary': {'B': b'{"nonce": "h7PYfVMR0%", "token": "<5UTbuxi_674Yx|q{I"}'},
    'secret_data': {'B': b'{"nonce": "*6nD)O5pyI", "token": "_MV*k4Y{(WmL01-?L~jawW#Is3ixE(kTpDUl#_!$G4}"}'},
    'secret_message': {
        'B': b'\x89\x17\x17]\xad@\xab\xda\xd9\x13\xd9\xe9\xb5\xc2,\x02F+gxO\x8b\xebI\x88[J<\x90\xee\xdc\xff'
    },
    'secret_float': {'B': b'{"nonce": "FTIU)7>ayz", "token": "o|_U!"}'}
}

If you retrieve the data using the ORM model, you will see the unencrypted data.

[4]:

model = ArchiveModel.get("aid-001")
rprint(model.to_dict())

{
    'aid': 'aid-001',
    'secret_binary': b'a secret image',
    'secret_data': {'Alice': 1, 'Bob': 2, 'Cathy': 3},
    'secret_float': 3.14,
    'secret_integer': 1234,
    'secret_message': 'attack at 2PM tomorrow!'
}

For determinative fields (where the same unencrypted data always leads to the same encrypted data), you can still use them for queries.

[5]:

for item in ArchiveModel.scan(
    ArchiveModel.secret_message == "attack at 2PM tomorrow!"
):
    rprint(item.to_dict())

{
    'aid': 'aid-001',
    'secret_binary': b'a secret image',
    'secret_data': {'Alice': 1, 'Bob': 2, 'Cathy': 3},
    'secret_float': 3.14,
    'secret_integer': 1234,
    'secret_message': 'attack at 2PM tomorrow!'
}

Of course, you can perform updates as well.

[6]:

# Update
model.update([
    ArchiveModel.secret_message.set("Hold the fire now!")
])
model.refresh()

rprint(model.to_dict())

{
    'aid': 'aid-001',
    'secret_binary': b'a secret image',
    'secret_data': {'Alice': 1, 'Bob': 2, 'Cathy': 3},
    'secret_float': 3.14,
    'secret_integer': 1234,
    'secret_message': 'Hold the fire now!'
}

Custom Encrypted Attribute.#

pynamodb_mate has four built-in Encrypted attribute:

pynamodb_mate.api.attributes.EncryptedNumberAttribute
pynamodb_mate.api.attributes.EncryptedUnicodeAttribute
pynamodb_mate.api.attributes.EncryptedBinaryAttribute
pynamodb_mate.api.attributes.EncryptedJsonDictAttribute

You can create your own encrypted attributes for arbitrary input. You just need to define how you want to serialize/deserialize the data to/from binary.

[7]:

# A Custom Encrypted Attribute that store Json list
import json

class EncryptedJsonListAttribute(pm.attributes.SymmetricEncryptedAttribute):
    """
    Encrypted JSON data Attribute.
    """
    def user_serializer(self, value: list) -> bytes:
        return json.dumps(value).encode("utf-8")

    def user_deserializer(self, value: bytes) -> list:
        return json.loads(value.decode("utf-8"))

[ ]: