Revoke network access if required
NOTE
Before commencing any work on stopping network access contact the affected application teams to ensure they are aware of what is happening.
If the work is required outside of normal working hours then the teams are unlikely to respond and you can continue with the items listed below at your own discretion based on the nature of the incident.
You may wish to contact the Modernisation Platform Product Manager before taking the more wide-reaching mitigations detailed below e.g. NACL/TGW changes that affect a whole business unit/VPC.
Further work is required to define the business processes behind taking the various decisions to revoke network access so that this document can be improved.
Summary
This document lists the steps required to revoke network access, and isolate compromised infrastructure from other applications and environments.
A network attack is an exceptional situation so quick amendments are needed. It is anticipated that this can be done quickly via the AWS console.
The steps listed deny access on a platform level. If additional changes are required at an application level then these must be undertaken by the relevant application team.
All changes are to be made manually rather than through code. Once the incident is resolved the Terraform code can be run to reset everything to it’s previous state.
Many of the changes below involve updating VPC NACLs. The code that defines the rules is here. As a general rule the rule numbering format is as follows:
1000 = intra-vpc traffic
2000 = inter-vpc traffic (dynamic rules)
3000 = deny east-west
4000 = private address ranges
5000 = internet access
6000 = public address ranges (dynamic)
7000 = access from internet
CAUTION
All networking changes via Terraform (e.g. GitHub workflows) should be stopped while this work is ongoing. Doing so could affect the mitigations being made via the steps below.
The following workflows should be disabled:
- core-vpc-production-deployment (and dev/test/preprod where needed)
- core-network-services-deployment
Some of the changes listed in this document will impact all applications connected to a particular NACL/TGW attachment. If you were to amend, for example
hmpps-production-private-nacl
, then all applications hosted in thehmpps-production
VPC will be impacted.In most cases, running the relevant Terraform codebase for the environment will revert the manual changes made to mitigate the incident. There are some exceptions to this which are flagged below.
Stopping inbound/outbound access for a single application
This is the best option to deploy where the incident is contained to a single application which you want to quarantine. The mitigation blocks access at security group level for the infrastructure hosting the application e.g. load balancers/instances etc. by removing all the rules as the default behaviour for a SG with no rules is to deny
all traffic.
The size, architecture and complexity of the application will dictate the most appropriate method.
You will need to login to the application account with the AdministratorAccess
SSO role.
Method 1: Delete all rules in existing security groups
Best for dealing with small applications with few instances, security groups and rules
- Go to VPC -> Security groups
- Select a security group
- Select Actions -> Edit inbound rules and Delete for each rule listed and finally Save rules
- Select Actions -> Edit outbound rules and Delete for each rule listed and finally Save rules
- Apply the above steps for every security group
Method 2: Attach a new empty security group to network interfaces
Best for dealing with larger applications with multiple instances, security groups and rules
Note
This approach works well for ec2 instances but for services like Application Load Balancers, Lambda and RDS you cannot edit the ENI security groups like this. Instead you must access each service individually to change the associated security group.
- Go to VPC -> Security groups -> Create security group
- Enter a Security group name and description e.g.
SG to block all traffic
- Select Delete to remove the default outbound rule to
0.0.0.0/0
- Select Create security group
- Go to Ec2 -> Network & Security -> Network Interfaces
- Select the relevant network interface you wish to edit
- Select Actions -> Change security groups
- Select Remove to remove all the existing SGs associated
- Then type in the name of your new “block all” SG and select Add security Group -> Save
- Repeat steps above for each Network interface you need to edit
Stopping inbound internet access for an application on MP
Ths mitigation removes the route in the public subnet allowing inbound access from the internet. This could be appropriate to use when there is an external threat which is affecting multiple applications hosted in the same business unit.
This work is carried out from the core-vpc-<environment>
account e.g. core-vpc-production
with the AdministratorAccess
SSO role.
- Go to VPC -> Route tables
- Select the public route table (e.g.
hmpps-preproduction-general-public
) - Select Actions -> Edit Routes
- Locate the route to the Internet Gateway and select Remove -> Save Changes
This change will affect all applications in the VPC that you have selected. No applications in the VPC will be accessible from the internet.
Stopping outbound internet access for an application on MP
This mitigation will block all outbound internet access for applications hosted in a single VPC. Similar to above, this may be appropriate if the incident is affecting multiple applications in the same business unit.
Due to the way Network Firewall applies rules in AWS (see here ), this section only covers changes to the VPC NACL.
This work is carried out from the core-vpc-<environment>
account e.g. core-vpc-production
with the AdministratorAccess
SSO role.
The process needs to be done for the data
, public
and private
NACLs. Examples for production are shown here:
hmpps-production-data-nacl
hmpps-production-public-nacl
hmpps-production-private-nacl
- Go to VPC -> Security -> Network ACLs
- Click on the checkbox for the ACL you want to change then select outbound rules
- Add a new
deny
for 0.0.0.0/0, rule number 4999, which will prevent access to the items listed after this point. All internet rules are 5000 and above based on our current code - Save changes
Notes on reverting via Terraform
This change cannot be reverted by re-applying Terraform because we use the
aws_network_acl_rule
resource, which manages rules individually and is non-authoritative. Manually added rules are not detected as drift and so you will need to manually revert your changes.
Stopping inbound access from the MOJ to an application via the Modernisation Platform Transit Gateway
This mitigation will stop all inbound traffic destined for the private address range space for applications hosted in a single VPC. This could be appropriate to use when an incident is affecting an application/s hosted outside of the Modernisation Platform but has network connectivity into MP for certain services. Taking this action will impact all applications hosted in the business unit VPC.
Due to the way Network Firewall applies rules in AWS (see here ), this section only covers changes to the VPC NACLs.
This work is carried out from the core-vpc-<environment>
account e.g. core-vpc-production
The process needs to be done for the data
, public
and private
NACLs. Examples for production are shown here:
hmpps-production-data-nacl
hmpps-production-public-nacl
hmpps-production-private-nacl
- Go to VPC -> Security -> Network ACLs
- Click on the checkbox for the one you want to change then select inbound rules
- Amend all the rules that are >= 4000 and < 5000 so they are in a
deny
state (i.e. all private address ranges,10.0.0.0/8
,172.16.0.0/12
and192.168.0.0/16
) - Save the changes
NOTE the rules between 4000 and 5000 is based on the code being correctly set up to put the rules in this range. Amend the instructions if this changes.
Stopping outbound access to the MOJ for an application via the Modernisation Platform Transit Gateway
This mitigation will stop all outbound traffic destined for the private address range space for applications hosted in a single VPC. This could be appropriate to use when an incident is affecting an application/s hosted inside the Modernisation Platform but has network connectivity out into other platforms/networks for certain services. Taking this action will impact all applications hosted in the business unit VPC.
Due to the way Network Firewall applies rules in AWS (see here ), this section only covers changes to the VPC NACLs.
This work is carried out from the core-vpc-<environment>
account, e.g. core-vpc-production
The process needs to be done for the data
, public
and private
NACLs. Examples for production are shown here:
hmpps-production-data-nacl
hmpps-production-public-nacl
hmpps-production-private-nacl
- Go to VPC -> Security -> Network ACLs
- Click on the checkbox for the one you want to change then select outbound rules
- Amend all the rules that are >= 4000 and < 5000 so they are in a
deny
state (i.e. all private address ranges,10.0.0.0/8
,172.16.0.0/12
and192.168.0.0/16
) - Save the changes
NOTE the rules between 4000 and 5000 is based on the code being correctly set up to put the rules in this range. Amend the instructions if this changes.
Blocking access for something moving internally on MP
This mitigation will stop all traffic to and from a VPC hosted in the Modernisation Platform. This could be appropriate when an incident is affecting an application/s hosted inside the Modernisation Platform that has network connectivity into another business unit VPC hosted in Modernisation Platform.
It can also be noted that this is a more nuclear version of the previous two steps, perhaps useful for when the exact nature/location of the incident is unclear and reassurance is needed that a MP-hosted VPC is fully isolated from any other MP-hosted VPC or other external MoJ platform/network.
At a platform level, we can delete the VPC Transit Gateway attachment to prevent internal movement. We can do so through the following steps:
- Connect to
core-network-services
. - Got to VPC -> Transit gateway -> Transit gateway attachments
- Pick the required attachment (e.g.
platforms-preproduction-attachment
) - On the Actions pick Delete transit gateway attachment
- Type delete in the box and click on the Delete button
- The changes are saved
Notes on reverting via Terraform
To revert this change you need to:
- First run an apply in the
core-vpc-*
code according to which environment was updated.- Then run an apply in the
core-network-services
code. This step may cause an error if ran within a couple of hours of making manual changes as the deleted TGW attachment stays present in a ‘deleted’ state for a while which currently our code cannot tolerate.