AWS Secrets Manager: How to Set Up Secrets and Fetch Them in a Python Lambda

1. What Is AWS Secrets Manager? ## 2. Creating a Secret in AWS Secrets Manager ## Step 3: Configure Encryption ## Step 4: Name the Secret ## Step 5: Review and Create ## 3. IAM Permissions for Lambda ## 4. Fetching Secrets in a Python Lambda ## Step 1: Python Dependencies ## Step 2: Python Code to Fetch Secrets ## Step 3: Using Secrets in Lambda Handler ## 5. Performance Consideration (Important) ## 6. Environment-Based Secret Management ## 7. Security and Best Practices Managing sensitive information such as database passwords, API keys, and tokens is a critical part of building secure cloud applications. Hardcoding secrets in source code or configuration files is a common anti-pattern that leads to security vulnerabilities. AWS Secrets Manager provides a secure, scalable, and auditable way to store and retrieve secrets dynamically at runtime. In this blog, we will cover: AWS Secrets Manager is a managed service that helps you: Step 1: Open AWS Secrets Manager Log in to the AWS Console Navigate to Secrets Manager Click Store a new secret Step 2: Choose Secret Type Select Other type of secret if you want to store custom values such as API keys. Example (Key/Value pairs): DB_USERNAME = admin DB_PASSWORD = StrongPassword@123 DB_HOST = mydb.cluster-xyz.us-east-1.rds.amazonaws.com Secrets Manager stores these values as encrypted JSON. Give the secret a clear, environment-aware name: myapp/dev/database myapp/staging/database myapp/prod/database This naming strategy avoids accidental cross-environment access. Click Store. Your secret is now securely stored. Your Lambda function must have permission to read secrets. Attach this policy to the Lambda execution role: Important: Always scope the Resource to specific secrets instead of using "*". AWS Lambda already includes: No additional libraries are required. Each call to GetSecretValue is a network call. Recommended Optimization: Cache Secrets Because Lambda execution environments are reused, you can cache secrets at module level: This reduces latency and API calls significantly. Use environment variables to control which secret is loaded: SECRET_NAME = myapp/dev/database Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/dev/database*" } ] } Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/dev/database*" } ] } CODE_BLOCK: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myapp/dev/database*" } ] } COMMAND_BLOCK: import json import boto3 from botocore.exceptions import ClientError def get_secret(secret_name, region_name="us-east-1"): client = boto3.client( service_name="secretsmanager", region_name=region_name ) try: response = client.get_secret_value(SecretId=secret_name) except ClientError as e: raise RuntimeError(f"Unable to retrieve secret: {e}") # Secrets are usually stored as JSON strings if "SecretString" in response: return json.loads(response["SecretString"]) else: # Binary secrets (rare case) return response["SecretBinary"] Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: import json import boto3 from botocore.exceptions import ClientError def get_secret(secret_name, region_name="us-east-1"): client = boto3.client( service_name="secretsmanager", region_name=region_name ) try: response = client.get_secret_value(SecretId=secret_name) except ClientError as e: raise RuntimeError(f"Unable to retrieve secret: {e}") # Secrets are usually stored as JSON strings if "SecretString" in response: return json.loads(response["SecretString"]) else: # Binary secrets (rare case) return response["SecretBinary"] COMMAND_BLOCK: import json import boto3 from botocore.exceptions import ClientError def get_secret(secret_name, region_name="us-east-1"): client = boto3.client( service_name="secretsmanager", region_name=region_name ) try: response = client.get_secret_value(SecretId=secret_name) except ClientError as e: raise RuntimeError(f"Unable to retrieve secret: {e}") # Secrets are usually stored as JSON strings if "SecretString" in response: return json.loads(response["SecretString"]) else: # Binary secrets (rare case) return response["SecretBinary"] COMMAND_BLOCK: def lambda_handler(event, context): secret_name = "myapp/dev/database" secrets = get_secret(secret_name) db_user = secrets["DB_USERNAME"] db_password = secrets["DB_PASSWORD"] db_host = secrets["DB_HOST"] # Example usage print(f"Connecting to DB at {db_host} with user {db_user}") return { "statusCode": 200, "body": "Secrets fetched successfully" } Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: def lambda_handler(event, context): secret_name = "myapp/dev/database" secrets = get_secret(secret_name) db_user = secrets["DB_USERNAME"] db_password = secrets["DB_PASSWORD"] db_host = secrets["DB_HOST"] # Example usage print(f"Connecting to DB at {db_host} with user {db_user}") return { "statusCode": 200, "body": "Secrets fetched successfully" } COMMAND_BLOCK: def lambda_handler(event, context): secret_name = "myapp/dev/database" secrets = get_secret(secret_name) db_user = secrets["DB_USERNAME"] db_password = secrets["DB_PASSWORD"] db_host = secrets["DB_HOST"] # Example usage print(f"Connecting to DB at {db_host} with user {db_user}") return { "statusCode": 200, "body": "Secrets fetched successfully" } CODE_BLOCK: _cached_secrets = None def get_cached_secret(secret_name): global _cached_secrets if _cached_secrets is None: _cached_secrets = get_secret(secret_name) return _cached_secrets Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: _cached_secrets = None def get_cached_secret(secret_name): global _cached_secrets if _cached_secrets is None: _cached_secrets = get_secret(secret_name) return _cached_secrets CODE_BLOCK: _cached_secrets = None def get_cached_secret(secret_name): global _cached_secrets if _cached_secrets is None: _cached_secrets = get_secret(secret_name) return _cached_secrets CODE_BLOCK: import os secret_name = os.environ["SECRET_NAME"] secrets = get_cached_secret(secret_name) Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: import os secret_name = os.environ["SECRET_NAME"] secrets = get_cached_secret(secret_name) CODE_BLOCK: import os secret_name = os.environ["SECRET_NAME"] secrets = get_cached_secret(secret_name) - What AWS Secrets Manager is - How to create and store secrets - IAM permissions required - How to fetch secrets in a Python AWS Lambda - Best practices - Securely store secrets (credentials, API keys, tokens) - Encrypt secrets using AWS KMS - Control access via IAM - Rotate secrets automatically (for supported services) - Retrieve secrets programmatically at runtime - Database credentials (RDS, Aurora) - Third-party API keys - JWT signing secrets - OAuth client secrets - Choose the default AWS-managed KMS key - Select a customer-managed KMS key for stricter compliance - Same codebase across DEV / STAGE / PROD - Environment-specific secrets - Safer deployments - Never hardcode secrets - Use least-privilege IAM policies - Use separate secrets per environment - Enable automatic rotation where supported - Cache secrets inside Lambda for performance - Log carefully—never log secret values - Prefer Secrets Manager over SSM Parameter Store for highly sensitive data