Intro.

While security researchers demonstrate their ranks in various bounty programms and threat hunters share yet another thrilling story of tracking and pinning down an APTs, avarage security engineers stay in the shadows. Most of our work is under NDA and IP rights belong to the companies we work for hence can hardly be presented to the world. Since one can’t demonstrate his actual “kingdom” why not to build a mock-up one with all the bells and whistles. The benefits are obvious:

• Save time during job interviews and skills assessments. Just give your interviewers access to your setup and code.

• Gather a collection on blueprints for use in future projects

• Have fun, experiment, try to break your own guardrails, learn new things.

I’ve also noticed that the majority of similar projects like CloudGoat or GCPGoat focus on breaking stuff rather than building the systems properly from grounds up. Through this series of posts, I will take you through the various stages of the project, from planning and design to implementation and compliance verification. We will go over the details on how to setp the infrastructure and establsih security controls to protect it and, including access control, network segmentation, intrusion detection and prevention, and continuous monitoring. Moreover there will be bits on what evidence of operationl controls for future audits and certifications, an important yet most overlooked topic in the majority of the tutorials.

What Are We Building

We are going to deploy Techschool’s Simple Bank backend service along with the supporting infrastructure and secury tooling. While the original course covers a lot of devops related topics such as containerising apps, uploading images to the ECR and deploying them to the AWS EKS, it’s main focus is still on software development, hence most of the provided recipies are aimed at simply getting your code up and running. We will spice it up with some AWS Well Architected Framework practices and DevSecOps.

I do recommend though to go through the whole Techschool’s course anyway. This will add a lot to your understanding of modern application stack!

Before We Begin

Let’s draw some diagrams first. Sadly, despite all the shift-somewhere manifestos and DevSecOps buzz, in 2023, the PCI Council, and hence the majority of security auditors, consider a drawing to be the primary source of truth for the organization, not your infrastracture code (see PCI DSS 4.0 Req 1.2.3). Even sadder, in 2023, the problem of automating the net/infra diagram drawing still presists.

When I first confronted it, I thought that it should be pretty easy to take a Terraform state and convert it into diagrams, right? Surprisingly, there is a whole bunch of questions that has no good answer.

• How do you draw the relations that are not even described in the state file?
• What if you use Pulumi instead?
• What if your state drifts or not all parts of your infra exist as code?

Maybe we could query the cloud API instead and get all the resources? Except as far as I know, it’s still next to impossible to get a full and adequate list of resources and their interconnections from the AWS API. Bizzare fact - the best solution here is aws-nuke a tool designed to completely wipe your AWS accounts. Quel bordel!

There are some commercial tools that claim they are capable of fully automated (to some extent) diagram creation form both source and API. Worth noting are cloudviz, cloudcraft and hava. Convincing the management to allocate budget for these tools can be challenging, as they are often perceived as ‘beautifiers’ rather than essential investments. Show them the following math to proove your point. It generally takes about a week of time and 2 engineers to go thtough all the infra/code and create the diagram. This is done primarily for a single time demonstration of compliance to the auditors. Since for your engineers the code is the primary source of truth the diagram will be abandoned the next day the auditor leaves. Hence, it’s a recurring yearly expence. Now, considering a devops labor cost of $70 per hour, with 8 hours a day for 7 days involving 2 engineers, the total cost amounts to $7,840 (!!!). This is a crazy number, even when compared to the costliest enterprise plans of the mentioned commercial tools.

For our poor-man’s infosec setup we’ll use Diagrams as Code.

Please note that this tool is also far from perfect when drawing big and complicated infrastructures and you might get back to good-old manual draw.io, Lucidchart or Visio.

Install the tooling with pip3 install diagrams, clone the supporting repo, change the code to your liking, run python <diagram-name>.py whenever you need a diagram.

Since the auditors consider “Thee Diagramm” a document much as any other policy, they toften ask you to demonstrate that “the document was approved by the relevant authority”. Storing your digrams and relevant code in version control system such as Git, digitaly signing the commits and following a general pull request authorisation sequence for changes will generate enough evidence, to satisfy the most rigorous audit.

This rather simple piece of code…

from diagrams import Diagram
from diagrams.aws.management import OrganizationsAccount, OrganizationsOrganizationalUnit, Organizations

graph_attr = {
    "splines": "spline",
}

with Diagram("AWS Accounts Layout", outformat=["jpg"], filename="../img/acc_arch", direction='LR'):
    org_root = Organizations ("MockUpBank Org")
    ou_infra = OrganizationsOrganizationalUnit("Infra OU")
    ou_security = OrganizationsOrganizationalUnit("Security OU")
    ou_prod = OrganizationsOrganizationalUnit ("Prod OU")
    ou_sandbox = OrganizationsOrganizationalUnit ("Sandbox OU")
    master_acc = OrganizationsAccount ("root")
    deploy_acc = OrganizationsAccount ("Deployment account")
    sec_acc = OrganizationsAccount ("Security account")
    log_acc = OrganizationsAccount ("Log Archive")
    dev_acc = OrganizationsAccount ("Dev Acc")
    prod_acc = OrganizationsAccount ("Production")
    org_root - master_acc
    org_root - ou_infra 
    org_root - ou_security 
    org_root - ou_prod
    org_root - ou_sandbox

    ou_infra - deploy_acc
    ou_security - sec_acc
    ou_security - log_acc
    ou_sandbox - dev_acc
    ou_prod - prod_acc

…will generate us this pretty picture:

In our AWS Organazation we are going to have the following elements:

• root account - core account with billing and full congtrol over the AWS Organization.
• Deployment OU - OU that hosts another special account that hosts TF state, gitops tools, together with cross account access roles.
• Infrastructure OU - OU that hosts accounts reserved for self-hosted infa/devops-ish stuff like Keycloak, Jenkins, Prometheus and etc.
• Sandbox OU - OU that hosts accounts for developers. This project just adds one generic account.
• Security OU - OU that hosts security related accounts including logging accounts and security tools.
• Production OU - This is where our production application and customer facing workloads go.

Some more code and we get this:

I’ll explain what’s happening here in the next post.