Mellow Root

Some pitfalls with security group rules in Terraform

In Terraform there are two ways to create security groups with rules. Either having the rules as sub-blocks or as separate resources.

Using sub-blocks:

resource "aws_security_group" "web_sg" {
  name        = "web_sg"
  description = "Security group for web servers"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Using separate resources:

resource "aws_security_group" "web_sg" {
  name        = "web_sg"
  description = "Security group for web servers"
}

resource "aws_security_group_rule" "http" {
  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.web_sg.id
}

So, when do you want to use what? I always prefer the latter. Some cases are only possible with the latter as you'll see below.

Adding the same rule to multiple security groups

When breaking out the rules as separate resources we can use loops to add them to multiple security groups. But it's not as straightforward as one might think. Look at the following code:

resource "aws_security_group" "web_sg_1" {
  name        = "web_sg_1"
  description = "Security group for web servers"
}

resource "aws_security_group" "web_sg_2" {
  name        = "web_sg_2"
  description = "Security group for web servers"
}

resource "aws_security_group_rule" "http" {
  for_each = [
    aws_security_group.web_sg_1.id,
    aws_security_group.web_sg_2.id,
  ]

  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = each.key
}

This code fails! Terraform complains the security groups IDs will be known only after apply. Instead, we'll have to chain for_each:

locals {
  sg_names = [
    "web_sg_1",
    "web_sg_2",
  ]
}

resource "aws_security_group" "web_sg" {
  for_each = toset(local.sg_names)

  name        = each.value
  description = "Security group for web servers"
}

resource "aws_security_group_rule" "http" {
  for_each = aws_security_group.web_sg

  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = each.value.id
}

Here we use a loop to create two security groups, and then another to add the same rule to both. This scenario isn't too common, but I've encountered it!

Cycle dependencies between security groups

Let's say you have a service and a database. You want to allow outgoing traffic from the service only on port 5432 and only to the database. Likewise, you want the database to only accept connections on port 5432 and only if it's from the service.

Security group rules can allow CIDRs, but they can also allow security group IDs, which is much better. But if we try this code:

resource "aws_security_group" "service" {
  name        = "service"
  description = "Security group for service"

  egress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [
      aws_security_group.database.id,
    ]
  }
}

resource "aws_security_group" "database" {
  name        = "database"
  description = "Security group for database"

  ingress {
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [
      aws_security_group.service.id,
    ]
  }
}

This will fail with Error: Cycle: aws_security_group.service, aws_security_group.database. To get around this we need to break out the rules into separate resources:

resource "aws_security_group" "service" {
  name        = "service"
  description = "Security group for service"
}

resource "aws_security_group" "database" {
  name        = "database"
  description = "Security group for database"
}

resource "aws_security_group_rule" "egress_postgres" {
  type                     = "egress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.database.id
  security_group_id        = aws_security_group.service.id
}

resource "aws_security_group_rule" "ingress_postgres" {
  type                     = "ingress"
  from_port                = 5432
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.service.id
  security_group_id        = aws_security_group.database.id
}

This way, the security groups are created before any rules are applied, which means we avoid the cycle dependencies!

Multiple projects/states attaching rules to the same security group

You might end up with a security group created in one project and want to attach security group rules from a different project. If these two projects use the same state, you cannot use the sub-blocks for ingress and egress, at all. Let's look at an example.

In Project A we create a security group that allows port 80, and in Project B we add another rule for 443:

# project-a/main.tf

resource "aws_security_group" "web_sg" {
  name        = "web_sg"
  description = "Security group for web servers"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}


# project-b/main.tf

data "aws_security_group" "web_sg" {
  name = "web_sg"
}

resource "aws_security_group_rule" "https" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = data.aws_security_group.web_sg.id
}

Imagine these two files are owned by two different states.

This works, for a while. The next time Project A is deployed your aws_security_group_rule will be removed. If Project A also used aws_security_group_rule there wouldn't be any problems though.

(I know this example is ridiculous, but there are real use cases where this is very useful, but that's out of scope for this post)

Follow my blog:

or RSS

#aws #terraform