AWS DynamoDb. The great ugly database

For some time now I am working quite heavily on AWS in my current project. And of course we try and give our systems the shape of micro services (who wouldn't?). When using micro services architecture it is very often a good practice to not use a single database to which every single service is connecting whenever it needs data. Instead it is recommended to give every service which uses data its own database to store its model. AWS give us a very interesting NoSql database named DynamoDb which purpose is to save and fetch data by the key (or multiple keys if you set them).
Our client architect told us "if something can be put to DynamoDb then we put it there". That sounds like DynamoDb is one great database right? Well, yeah. But my first experience with it was actually pretty negative. I did change my opinion over time and eventually we started to get along. And I want to walk you through the journey I had to take in order to discover its beauty. And since I am a software developer my journey started with a simple class.

Ok, so let's have a look at the most simple client class:

var dynamoDbObject = new Dictionary<string, AttributeValue>()
{
     ["Id"] = new AttributeValue
     {
         S = "Id"
     },
     ["Name"] = new AttributeValue
     {
         S = "TestName"
     },
     ["FamilyName"] = new AttributeValue
     {
         S = "TestFamilyName"
     },
     ["Email"] = new AttributeValue
     {
         S = "TestEmail"
     },
     ["Age"] = new AttributeValue
     {
         N = "20"
     },
     ["IsActive"] = new AttributeValue
     {
         BOOL = true
     },
     ["Orders"] = new AttributeValue
     {
         L = new List<AttributeValue> 
         {
             new AttributeValue
             {
                 M = new Dictionary<string, AttributeValue>
                 {
                     ["OrderId"] = new AttributeValue 
                     { 
                         S = Guid.NewGuid().ToString() 
                     },
                     ["ProductName"] = new AttributeValue 
                     { 
                         S = "TestProduct" 
                     }
                 }
            }
        }
    }
};

Now that is a lot of code don't you think? It's obvious that when working with DynamoDb there is going to be some wrapping going on. So let's create some extension method to make us not repeat the above lines over and over:

public static Dictionary<string, AttributeValue> ToDynamoDbObject(this Client client)
{
    return new Dictionary<string, AttributeValue>()
    {
        ["Id"] = new AttributeValue
        {
            S = client.Id
        },
        ["Name"] = new AttributeValue
        {
            S = client.Name
        },
        ["FamilyName"] = new AttributeValue
        {
            S = client.FamilyName
        },
        ["Email"] = new AttributeValue
        {
            S = client.Email
        },
        ["Age"] = new AttributeValue
        {
            N = client.Age.ToString()
        },
        ["IsActive"] = new AttributeValue
        {
            BOOL = client.IsActive
        },
        ["Orders"] = new AttributeValue
        {
            L = client.Orders.Select(x => new AttributeValue
            {
                M = new Dictionary<string, AttributeValue>
                {
                    ["OrderId"] = new AttributeValue
                    {
                        S = x.OrderId.ToString()
                    },
                    ["ProductName"] = new AttributeValue
                    {
                        S = x.ProductName
                    }
                 }
             }).ToList()
         }
     };
 }

Ok, so we are moving forward. But there is something strange about this code. What on earth are those S, M, N, L properties? Well they represent basic data type like S=String, N=Number (which is a string because of some reason), L=List, M=Map (object) etc. It is not that difficult to guess but still how hard it is to name those properties something like "StringValue" or whatever. Plus what is SS for example? Or BS (how appropriate)? I mean, without documentation those things are not obvious.

So we have out extension method and with this we can finally add our document to the database:

static void Main(string[] args)
{
    var client = new Client
    {
        Name = "TestName",
        FamilyName = "TestFamilyName",
        Age = 20,
        IsActive = true,
        Orders = new List<Order>
        {
            new Order
            {
                OrderId = Guid.NewGuid(),
                ProductName = "Awesome PC"
            },
            new Order
            {
                OrderId = Guid.NewGuid(),
                ProductName = "Awesome Laptop"
            }
        }
    };

   var dynamoClient = new AmazonDynamoDBClient();
    dynamoClient.PutItem("MyTestTable", client.ToDynamoDbObject());
}

Everything looks fine and dandy. But the "fun" part has just begun.
This solution is not perfect. And if you will not test the crap out of such solution you might be surprised when your production will start throwing exceptions. Let's modify our example a little bit:

static void Main(string[] args)
{
    var client = new Client
    {
        Id = Guid.NewGuid().ToString(),
        Name = "TestName",
        FamilyName = "TestFamilyName",
        Email = "",
        Age = 20,
        IsActive = true,
        Orders = new List<Order>
        {
            new Order
            {
                OrderId = Guid.NewGuid(),
                ProductName = "Awesome PC"
             },
             new Order
             {
                 OrderId = Guid.NewGuid(),
                 ProductName = "Awesome Laptop"
             }
        }
    };

   var dynamoClient = new AmazonDynamoDBClient();
   dynamoClient.PutItem("TestTable", client.ToDynamoDbObject());
}

This example will throw "'One or more parameter values were invalid: An AttributeValue may not contain an empty string'" exception. That's right, you can not place empty string in string property! In this situation you must make your property null. But that doesn't make sense right? Empty string and null value are two different things. But not for DynamoDb. But to make things event more interesting, it appears that null values is actually a true/false value type!

public static Dictionary<string, AttributeValue> ToDynamoDbObject(this Client client)
{
    return new Dictionary<string, AttributeValue>()
    {
        ["Id"] = client.Id.ToDynamoString(),
        ["Name"] = client.Name.ToDynamoString(),
        ["FamilyName"] = client.FamilyName.ToDynamoString(),
        ["Email"] = client.Email.ToDynamoString(),
        ["Age"] = client.Age.ToString().ToDynamoString(),
        ["IsActive"] = new AttributeValue
        {
            BOOL = client.IsActive
        },
        ["Orders"] = new AttributeValue
        {
            L = client.Orders.Select(x => new AttributeValue
            {
                M = new Dictionary<string, AttributeValue>
                {
                    ["OrderId"] = new AttributeValue
                    {
                        S = x.OrderId.ToString()
                    },
                    ["ProductName"] = new AttributeValue
                    {
                        S = x.ProductName
                    }
                }
           }).ToList()
        }
    };
}


private static AttributeValue ToDynamoString(this string text)
{
    return string.IsNullOrEmpty(text) ? new AttributeValue { NULL = true } : new AttributeValue { S = text };
}

Things are getting quite crazy don't they? But we are not over yet.

static void Main(string[] args)
{
    var client = new Client
    {
        Id = Guid.NewGuid().ToString(),
        Name = "TestName",
        FamilyName = "TestFamilyName",
        Email = "test@awesome.com",
        Age = 20,
        IsActive = true,
        Orders = new List<Order> {}
    };

    var dynamoClient = new AmazonDynamoDBClient();
    dynamoClient.PutItem("TestTable", client.ToDynamoDbObject());
}

This code will throw 'Supplied AttributeValue is empty, must contain exactly one of the supported datatypes'. As you probably noticed we made our Order list empty. I know what you are thinking right now. "They wouldn't dare!! Please say it ain't so!". But yep, you can't have empty list like that. The situation is not as bad as with string because you can actually have an empty list. All you have to do is set property IsLSet = true!

What black sorcery is this?! What does IsLSet even stands for? And to add insult to injury, you must leave L property null otherwise you will get another exception. That makes our final extension look like that:

public static Dictionary<string, AttributeValue> ToDynamoDbObject(this Client client)
{
    return new Dictionary<string, AttributeValue>()
    {
        ["Id"] = client.Id.ToDynamoString(),
        ["Name"] = client.Name.ToDynamoString(),
        ["FamilyName"] = client.FamilyName.ToDynamoString(),
        ["Email"] = client.Email.ToDynamoString(),
        ["Age"] = client.Age.ToString().ToDynamoString(),
        ["IsActive"] = new AttributeValue
        {
            BOOL = client.IsActive
        },
        ["Orders"] = client.Orders == null || !client.Orders.Any()
            ? new AttributeValue { IsLSet = true } 
            : new AttributeValue
            {
                L = client.Orders.Select(x => new AttributeValue
                {
                    M = new Dictionary<string, AttributeValue>
                    {
                        ["OrderId"] = new AttributeValue
                        {
                            S = x.OrderId.ToString()
                        },
                        ["ProductName"] = new AttributeValue
                        {
                            S = x.ProductName
                        }
                    }
               }).ToList()
         }
    };
}

private static AttributeValue ToDynamoString(this string text)
{
    return string.IsNullOrEmpty(text) ? new AttributeValue { NULL = true } : new AttributeValue { S = text };
}

Phew! That was interesting. But let's put that coding hell aside. DynamoDb give us some benefits from architecture and business point of view:

1. It is in cloud (duh!). If you love it, you leave it. If you hate it, you kill it.
2. It's actually quite fast.
3. It can be scaled asynchronusly. You can set read and write capacity separately. This is quite neat. It can be helpful when you want to go full CQRS and have separate database for reading and separate for writing.
4. It is dirt cheap. Seriously. On average load micro service I usually create 5 read and 5 write table. Like the one below:

Just 3 bucks a month?! Event with my poor Polish wallet I can afford that.

DynamoDb also has auto scaling but currently it is quite slow. But event if you have once in a while a moment of bigger load, DynamoDb will actually not throttle your requests! This is because AWS allows a moment of unusual load. So as long as those moments will not last longer than 30 sec or something you should be fine.

To be hones if it were up to me I would use DynamoDb really hard. Its cheap, quick, easy to set up, scalable , autoscalable etc. The only downside is the SDK. But if you will wrap it in your own classes then you can get use to its crappyness. Overall if your architecture is on AWS and you are planing to go with micros services I highly recommend you give DynamoDb a try.