Tools

Tools: How to Deploy on Air-Gapped AWS EKS for Regulated Financial Services (2026)

2026-05-27 0 views admin
Tools: How to Deploy on Air-Gapped AWS EKS for Regulated Financial Services (2026)

What Does "Air-Gapped" Mean in AWS Context?

Why This Matters for Regulated Workloads

How Do You Design a Zero-Egress VPC for AWS EKS?

Subnet Design

VPC Endpoints to Provision

IAM Roles

How Do You Run an EKS Cluster with No Public Internet Path?

Cluster Creation

Node Groups

Bootstrapping In-Cluster Controllers

How Do You Build a CI/CD Pipeline Without Internet Access?

Self-Hosted Runner in VPC

First Deployment Validation

How Should Data Stores Be Configured in an Air-Gapped VPC?

RDS PostgreSQL with pgvector

ElastiCache Redis and DynamoDB

How Do You Manage Secrets and Security Without Exposing Credentials?

AWS WAF on the ALB

Kubernetes Secrets Without Kubernetes Secrets

TLS and HTTPS Enforcement

How Do You Run Amazon Bedrock and Transcribe in an Air-Gapped Cluster? Financial services data breaches cost an average of $6.08 million per incident — 22% above the global average across all industries (IBM Cost of a Data Breach 2024, 2024). For regulated institutions, the answer isn't just better firewalls. It's network architecture that eliminates the attack surface at the infrastructure level. Air-gapped AWS EKS deployments — where private subnets have zero internet egress and all traffic routes through VPC endpoints — are becoming the standard for regulated financial services workloads. This guide walks through the full architecture, from VPC design to CI/CD pipeline, based on a real deployment we executed for a fintech platform at Prodinit. An air-gapped VPC means your private subnets have no route to the internet — no NAT Gateway in private subnets, no internet gateway attachment to private route tables. All communication between your workloads and AWS services (S3, ECR, Secrets Manager, CloudWatch, Bedrock) must route through VPC endpoints. AWS supports two endpoint types (AWS VPC Endpoints documentation): For a typical EKS deployment, you'll need interface endpoints for: ECR API, ECR Docker, S3 (or gateway), Secrets Manager, Systems Manager, CloudWatch Logs, STS, ELB, Bedrock (if using AI services), and Transcribe (if using speech-to-text). Network isolation is a hard requirement under FFIEC guidelines and SEC cybersecurity rules for financial institutions. An air-gapped VPC enforces this at the infrastructure layer — there's no misconfigured security group that can accidentally allow outbound internet access, because the route simply doesn't exist. On the Client deployment, we discovered mid-project that several Helm charts we'd planned to use for in-cluster controllers (ALB Ingress Controller, cluster-autoscaler) attempt to pull their own images from public registries at install time. We had to mirror every controller image into private ECR before the cluster could bootstrap. This is a class of problem that only surfaces when you actually try to deploy — not in planning. Regulated financial services environments under FFIEC and SEC cybersecurity guidelines require network isolation enforced at the infrastructure level — not as a policy overlay but as a structural property of the network. A multi-AZ VPC with private subnets carrying no internet route, combined with VPC interface endpoints for every AWS service, eliminates the outbound internet path entirely rather than restricting it. Start with a multi-AZ VPC with distinct public and private subnet tiers. Private subnet route tables should contain exactly two entries: the local VPC CIDR, and gateway endpoint routes for S3/DynamoDB. Nothing else. Create interface endpoints in your private subnets for each required AWS service: Create a dedicated security group for endpoints that allows HTTPS (443) inbound from your VPC CIDR. Don't open it wider than needed. Set up three distinct IAM role categories before touching EKS: Use IAM Roles for Service Accounts (IRSA) for in-cluster components. This ties Kubernetes service accounts to IAM roles without storing credentials anywhere. As of 2024, 80% of organizations run Kubernetes in production (CNCF Annual Survey 2024). The hard part isn't Kubernetes — it's running it without any public internet path. When creating the EKS cluster, set the API server endpoint access to private only: With --endpoint-public-access false, kubectl only works from inside the VPC. This is intentional. Access the cluster via a bastion host or AWS Systems Manager Session Manager. Place node groups in private subnets with autoscaling enabled: This is where most air-gapped deployments stall. cluster-autoscaler and the AWS Load Balancer Controller both try to pull images from public registries during helm install. You must mirror them to ECR first: Override the image in Helm values to point to your private ECR before installing. Standard GitHub Actions hosted runners and most CI/CD platforms assume outbound internet access for image pulls and API calls — assumptions that silently break the moment you remove internet egress. The working architecture requires a self-hosted runner deployed inside the VPC, all image pushes to private ECR via VPC endpoint, and all cluster deployments executed through AWS Systems Manager Session Manager with zero inbound ports. Standard CI/CD tooling assumes internet access. GitHub Actions' hosted runners can't reach a private EKS API endpoint. CodePipeline agents can't pull from Docker Hub. You need a fundamentally different pipeline architecture. The pattern that works: If using GitHub Actions, deploy a self-hosted runner inside the VPC. It can reach the private EKS API endpoint and ECR via VPC endpoints: Don't call the pipeline "working" until you've traced the full path end to end: image push → ECR → EKS pod pull → running pod → ALB health check → live traffic. Each hop can fail independently in an air-gapped setup. Every data store in a regulated EKS deployment — RDS PostgreSQL, DynamoDB, and ElastiCache Redis — must be provisioned in private subnets with publicly_accessible: false set explicitly at the resource level, not just through security group rules. Security groups can be modified; the publicly_accessible flag removes the public DNS endpoint entirely, closing the exposure regardless of any future policy drift. All data stores must be provisioned in private subnets with no public endpoint exposure. For AI-augmented fintech applications, pgvector enables vector similarity search inside Postgres — useful for semantic search over transaction data, document embeddings, or fraud pattern matching. Install the extension after provisioning: Both run entirely in private subnets. ElastiCache Redis requires a subnet group scoped to private subnets. DynamoDB uses a gateway endpoint (free) — no interface endpoint needed. Kubernetes Secret objects are base64-encoded, not encrypted — any cluster administrator with RBAC read access can decode them with a single command. In regulated environments, AWS External Secrets Operator resolves this by pulling credentials from AWS Secrets Manager at pod startup and syncing them into ephemeral Kubernetes Secrets. Credentials never appear in manifest files, Git history, or container image layers. Attach a Web ACL to your Application Load Balancer with at minimum: The ALB sits in public subnets (it receives external traffic), but the security group only allows 443 inbound. Backend EKS nodes only allow traffic from the ALB security group. Storing secrets as Kubernetes Secret objects is fine for development, but they're base64-encoded, not encrypted, and cluster admins can read them. In a regulated environment, use External Secrets Operator to pull from AWS Secrets Manager instead: Pods mount the resulting Kubernetes Secret normally — but the actual credential lives in Secrets Manager, has rotation enabled, and never touches a manifest file. Provision an ACM certificate for your domain and configure the ALB to redirect HTTP to HTTPS. Set HTTPS-only enforcement at the ALB listener level — don't rely on application code to enforce it. Amazon Bedrock and Amazon Transcribe both support VPC interface endpoints, meaning all LLM inference and speech-to-text requests from private EKS workloads never leave the AWS network. For regulated industries, this keeps AI processing within the same network boundary as the rest of the application — data residency compliance is maintained without routing inference traffic through the public internet. For platforms using Amazon Bedrock (LLM inference) or Amazon Transcribe (speech-to-text), both services support VPC interface endpoints — meaning model inference requests never leave the AWS network. IAM policies for Bedrock should be scoped to specific model ARNs — don't grant bedrock:* broadly. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Code Block

Copy

VPC: 10.0.0.0/16 ├── Public Subnets (one per AZ) │ ├── 10.0.1.0/24 (us-east-1a) │ ├── 10.0.2.0/24 (us-east-1b) │ └── NAT Gateways (for public-subnet resources only) └── Private Subnets (one per AZ) ├── 10.0.10.0/24 (us-east-1a) └── 10.0.11.0/24 (us-east-1b) — No internet route in route table — VPC endpoints attached VPC: 10.0.0.0/16 ├── Public Subnets (one per AZ) │ ├── 10.0.1.0/24 (us-east-1a) │ ├── 10.0.2.0/24 (us-east-1b) │ └── NAT Gateways (for public-subnet resources only) └── Private Subnets (one per AZ) ├── 10.0.10.0/24 (us-east-1a) └── 10.0.11.0/24 (us-east-1b) — No internet route in route table — VPC endpoints attached VPC: 10.0.0.0/16 ├── Public Subnets (one per AZ) │ ├── 10.0.1.0/24 (us-east-1a) │ ├── 10.0.2.0/24 (us-east-1b) │ └── NAT Gateways (for public-subnet resources only) └── Private Subnets (one per AZ) ├── 10.0.10.0/24 (us-east-1a) └── 10.0.11.0/24 (us-east-1b) — No internet route in route table — VPC endpoints attached # ECR — required for image pulls aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.api \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.dkr \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints # ECR — required for image pulls aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.api \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.dkr \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints # ECR — required for image pulls aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.api \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints aws ec2 create-vpc-endpoint --vpc-id vpc-xxx \ --service-name com.amazonaws.us-east-1.ecr.dkr \ --vpc-endpoint-type Interface \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints eksctl create cluster \ --name fintech-prod \ --region us-east-1 \ --vpc-private-subnets subnet-xxx,subnet-yyy \ --node-private-networking \ --endpoint-private-access true \ --endpoint-public-access false eksctl create cluster \ --name fintech-prod \ --region us-east-1 \ --vpc-private-subnets subnet-xxx,subnet-yyy \ --node-private-networking \ --endpoint-private-access true \ --endpoint-public-access false eksctl create cluster \ --name fintech-prod \ --region us-east-1 \ --vpc-private-subnets subnet-xxx,subnet-yyy \ --node-private-networking \ --endpoint-private-access true \ --endpoint-public-access false # nodegroup.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: fintech-prod region: us-east-1 managedNodeGroups: - name: workers instanceTypes: ["m6i.xlarge", "m6i.2xlarge"] minSize: 2 maxSize: 10 desiredCapacity: 3 privateNetworking: true iam: withAddonPolicies: autoScaler: true cloudWatch: true # nodegroup.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: fintech-prod region: us-east-1 managedNodeGroups: - name: workers instanceTypes: ["m6i.xlarge", "m6i.2xlarge"] minSize: 2 maxSize: 10 desiredCapacity: 3 privateNetworking: true iam: withAddonPolicies: autoScaler: true cloudWatch: true # nodegroup.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: fintech-prod region: us-east-1 managedNodeGroups: - name: workers instanceTypes: ["m6i.xlarge", "m6i.2xlarge"] minSize: 2 maxSize: 10 desiredCapacity: 3 privateNetworking: true iam: withAddonPolicies: autoScaler: true cloudWatch: true # Pull, retag, and push to private ECR docker pull registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 docker tag registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 \ 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 # Pull, retag, and push to private ECR docker pull registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 docker tag registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 \ 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 # Pull, retag, and push to private ECR docker pull registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 docker tag registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 \ 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/cluster-autoscaler:v1.29.0 Developer pushes code ↓ GitHub Actions (or CodePipeline) ↓ Build image in CI environment (with internet access) ↓ Push image to Private ECR via VPC endpoint ↓ Trigger deployment (CodePipeline or self-hosted runner in VPC) ↓ kubectl/Helm apply via Systems Manager Session Manager ↓ EKS pulls image from private ECR (no internet needed) ↓ ALB routes traffic Developer pushes code ↓ GitHub Actions (or CodePipeline) ↓ Build image in CI environment (with internet access) ↓ Push image to Private ECR via VPC endpoint ↓ Trigger deployment (CodePipeline or self-hosted runner in VPC) ↓ kubectl/Helm apply via Systems Manager Session Manager ↓ EKS pulls image from private ECR (no internet needed) ↓ ALB routes traffic Developer pushes code ↓ GitHub Actions (or CodePipeline) ↓ Build image in CI environment (with internet access) ↓ Push image to Private ECR via VPC endpoint ↓ Trigger deployment (CodePipeline or self-hosted runner in VPC) ↓ kubectl/Helm apply via Systems Manager Session Manager ↓ EKS pulls image from private ECR (no internet needed) ↓ ALB routes traffic # .github/workflows/deploy.yml jobs: deploy: runs-on: self-hosted # runner inside VPC steps: - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789:role/cicd-deploy-role aws-region: us-east-1 - name: Login to ECR run: | aws ecr get-login-password | docker login \ --username AWS \ --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com - name: Deploy to EKS run: | aws eks update-kubeconfig --name fintech-prod helm upgrade --install my-app ./helm/my-app \ --set image.tag=${{ github.sha }} # .github/workflows/deploy.yml jobs: deploy: runs-on: self-hosted # runner inside VPC steps: - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789:role/cicd-deploy-role aws-region: us-east-1 - name: Login to ECR run: | aws ecr get-login-password | docker login \ --username AWS \ --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com - name: Deploy to EKS run: | aws eks update-kubeconfig --name fintech-prod helm upgrade --install my-app ./helm/my-app \ --set image.tag=${{ github.sha }} # .github/workflows/deploy.yml jobs: deploy: runs-on: self-hosted # runner inside VPC steps: - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789:role/cicd-deploy-role aws-region: us-east-1 - name: Login to ECR run: | aws ecr get-login-password | docker login \ --username AWS \ --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com - name: Deploy to EKS run: | aws eks update-kubeconfig --name fintech-prod helm upgrade --install my-app ./helm/my-app \ --set image.tag=${{ github.sha }} # Terraform resource "aws_db_instance" "postgres" { identifier = "fintech-postgres" engine = "postgres" engine_version = "16.1" instance_class = "db.r6g.large" multi_az = true db_subnet_group_name = aws_db_subnet_group.private.name vpc_security_group_ids = [aws_security_group.rds.id] publicly_accessible = false storage_encrypted = true # Enable pgvector via parameter group parameter_group_name = aws_db_parameter_group.postgres_pgvector.name } # Terraform resource "aws_db_instance" "postgres" { identifier = "fintech-postgres" engine = "postgres" engine_version = "16.1" instance_class = "db.r6g.large" multi_az = true db_subnet_group_name = aws_db_subnet_group.private.name vpc_security_group_ids = [aws_security_group.rds.id] publicly_accessible = false storage_encrypted = true # Enable pgvector via parameter group parameter_group_name = aws_db_parameter_group.postgres_pgvector.name } # Terraform resource "aws_db_instance" "postgres" { identifier = "fintech-postgres" engine = "postgres" engine_version = "16.1" instance_class = "db.r6g.large" multi_az = true db_subnet_group_name = aws_db_subnet_group.private.name vpc_security_group_ids = [aws_security_group.rds.id] publicly_accessible = false storage_encrypted = true # Enable pgvector via parameter group parameter_group_name = aws_db_parameter_group.postgres_pgvector.name } CREATE EXTENSION IF NOT EXISTS vector; CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops); CREATE EXTENSION IF NOT EXISTS vector; CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops); CREATE EXTENSION IF NOT EXISTS vector; CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops); # ExternalSecret — pulls from Secrets Manager into a Kubernetes Secret apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials creationPolicy: Owner data: - secretKey: password remoteRef: key: fintech/prod/db property: password # ExternalSecret — pulls from Secrets Manager into a Kubernetes Secret apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials creationPolicy: Owner data: - secretKey: password remoteRef: key: fintech/prod/db property: password # ExternalSecret — pulls from Secrets Manager into a Kubernetes Secret apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: ClusterSecretStore target: name: db-credentials creationPolicy: Owner data: - secretKey: password remoteRef: key: fintech/prod/db property: password # Bedrock VPC endpoint aws ec2 create-vpc-endpoint \ --service-name com.amazonaws.us-east-1.bedrock-runtime \ --vpc-endpoint-type Interface \ --vpc-id vpc-xxx \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints # Bedrock VPC endpoint aws ec2 create-vpc-endpoint \ --service-name com.amazonaws.us-east-1.bedrock-runtime \ --vpc-endpoint-type Interface \ --vpc-id vpc-xxx \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints # Bedrock VPC endpoint aws ec2 create-vpc-endpoint \ --service-name com.amazonaws.us-east-1.bedrock-runtime \ --vpc-endpoint-type Interface \ --vpc-id vpc-xxx \ --subnet-ids subnet-xxx subnet-yyy \ --security-group-ids sg-endpoints - Air-gapped EKS requires VPC interface and gateway endpoints for every AWS service your workloads touch — there's no fallback to the public internet - Your CI/CD pipeline must be redesigned from scratch: images push to private ECR via VPC endpoint, and deployment runs through Systems Manager or a bastion inside the VPC - Kubernetes External Secrets Operator + AWS Secrets Manager is the cleanest pattern for pod-level secret injection without exposing credentials in manifests - Every data store (RDS, DynamoDB, ElastiCache) must live in private subnets, accessed via security group rules — no public endpoints - Gateway endpoints — for S3 and DynamoDB only; free, added as route table entries - Interface endpoints — for all other AWS services via AWS PrivateLink; billed per hour per AZ - Developer access role — scoped to read operations, no production deploy permissions - CI/CD role — ECR push, EKS kubectl apply, Secrets Manager read - Node instance role — ECR pull, CloudWatch logging, S3 read for application buckets - Core Rule Set (CRS) — protects against OWASP Top 10 - Known Bad Inputs — blocks common injection payloads

🏷️ Tags

toolsutilitiessecurity toolsdeploygappedregulatedfinancialservices

More from Tools

Tools: When you bring your data home, who is going to keep an eye on it? (2026)

Tools: When you bring your data home, who is going to keep an eye on it? (2026)

2026-05-27 0
Tools: keys per tenant: ditching our custom LLM billing layer Virtual

Tools: keys per tenant: ditching our custom LLM billing layer Virtual

2026-05-27 0
Tools: Latest: Remetric: find waste in self-hosted Prometheus, Grafana, and Loki

Tools: Latest: Remetric: find waste in self-hosted Prometheus, Grafana, and Loki

2026-05-27 0
Tools: Latest: A Caddy Cert Expired Because systemd-resolved Was Selectively Lying

Tools: Latest: A Caddy Cert Expired Because systemd-resolved Was Selectively Lying

2026-05-27 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views