S3/Cloudwatch Logs

This runbook summarises Modernisation Platform log source, the pipelines that feed it, where the data lives, and how long it is retained, for audit, security, and operational visibility.

Log source	Description	Storage / pipeline	Logging account and source definition	Location (S3 / CloudWatch)	Retention (days)
Organisation-wide CloudTrail	API activity across every MP account (auditing/security).	CloudTrail -> S3 modernisation-platform-logs-cloudtrail	core-logging (logs_cloudtrail_s3.tf)	CloudWatch and S3 with versioning + replication	STANDARD_IA at 90 d, Glacier at 365 d, delete at 730 d
AWS Config (all accounts)	Resource snapshot + configuration history for compliance/drift.	Config -> S3 modernisation-platform-logs-config	core-logging (logs_config_s3.tf)	S3 with versioning + replication	STANDARD_IA at 90 d, Glacier at 365 d, delete at 730 d
Route53 public DNS	DNS query logs for public zones.	Route53 -> CloudWatch /aws/route53/core-public-dns-query-logging -> Firehose -> S3 modernisation-platform-logs-r53-public-dns-logs	core-network-services (logging.tf) / core-logging (logs_r53_public_dns_firehose.tf)	CloudWatch and S3 with versioning + replication	CloudWatch 365 d; S3: 90 d -> STANDARD_IA, 365 d -> Glacier, delete 730 d
Route53 resolver / private DNS	Internal resolver queries (east-west visibility).	Resolver query logging -> CloudWatch r53-resolver-logs* -> S3 Cortex buckets	core-logging (logs_r53_resolver_config.tf)	CloudWatch + short-retention S3	CloudWatch 365 d; S3 14 d
VPC / TGW flow logs	Flow metadata for shared / inspection VPCs and TGW.	Flow logs -> CloudWatch groups per VPC/TGW; S3 core-logging-vpc-flow-logs	core-network-services (firewall.tfx) / core-vpc (vpc.tf) / core-logging (logs_s3.tf)	CloudWatch + short-retention S3	CloudWatch 365 d; S3 14 d
Network firewall logs	Captures denied traffic from the inspection firewall.	Network Firewall logging -> CloudWatch fw-*-logs-*	core-network-services (firewall.tf)	CloudWatch	365 days
WAF request logs	HTTP request samples for WAF protections.	Member WAFs -> Firehose -> S3 modernisation-platform-waf-logs	Member accounts / core-logging (logs_waf_firehose.tf; logs_waf_cloudwatch.tf; logs_waf_s3.tf))	S3 with versioning + replication	STANDARD_IA at 90 d, Glacier at 365 d, delete at 730 d
VPN attachment	VPN tunnel logs and metrics.	CloudWatch ${vpn}-vpn-attachment-logs	core-network-services (vpn.tf)	CloudWatch log groups per attachment	365 days
MP workflow data	GitHub Actions polling output for Grafana monitoring.	Lambda -> CloudWatch modernisation-platform-workflow-data (/aws/lambda/github-workflow-data-poller)	core-logging (github_workflow_lambda.tf)	CloudWatch	90 days
Cortex ingestion logs	Short-lived holding areas for files sent to Cortex XSIAM (flow / resolver / generic logs).	Direct S3 ingestion (core-logging-vpc-flow-logs, r53-resolver-logs, generic-logs)	core-logging (logs_s3.tf; logs_s3_sqs.tf)	S3 (no versioning)	14 days
EC2 Image Builder logs	Build logs and artefacts documenting AMI and component install runs.	EC2 Image Builder pipelines -> ephemeral build instances -> S3 ec2-image-builder-logs-*	core-shared-services (bucket.tf)	S3 (no versioning, with replication)	STANDARD_IA at 90 d, Glacier at 365 d, delete at 730 d
Generic execution logs	Automation code (CUR crawler, GitHub poller, ETL jobs) capturing run output and errors.	Automation jobs -> CloudWatch log groups	Varies by workload	CloudWatch	Varies

This page was last reviewed on 19 January 2026. It needs to be reviewed again on 19 July 2026 by the page owner #modernisation-platform .