Hybrid Development Workflow from On-Prem to AWS

Most teams develop directly in the cloud.

We don't.

Not because AWS isn't good.

Because development and production optimize for different things.

Our day-to-day development happens on an on-premises Kubernetes cluster built with kubeadm. Once a feature reaches a stable state, we provision a short-lived development environment on Amazon EKS to validate cloud-specific behavior before customer deployments.

This isn't about avoiding AWS—it's about using the right infrastructure at the right stage.

Why On-Prem?

Development is highly iterative.

• Recreating deployments
• Deploying and rolling back services
• Running AI inference workloads
• Validating storage and networking changes
• Testing CI/CD pipelines

Doing all of this directly in AWS means continuously paying for resources that provide little value during rapid development:

• EC2 instances
• EKS control plane
• Load Balancers
• NAT Gateways
• EBS volumes
• Snapshots
• Data transfer

By using our own Kubernetes cluster, we avoid recurring cloud costs, iterate faster, and have complete control over the infrastructure.

The goal of our on-prem environment is simple:

Maximize developer productivity.

Why We Validate on AWS

Once a feature reaches a stable state, we provision a temporary Amazon EKS environment.

It's to validate the platform against the same cloud infrastructure our users will eventually run.

We verify:

• Terraform provisioning
• Amazon EKS
• IAM Roles for Service Accounts (IRSA)
• Amazon EBS CSI Driver
• AWS Load Balancer Controller
• VPC networking
• Security Groups
• DNS
• Ingress
• Kong Gateways
• Cloud-native integrations

These are platform-level validations that cannot be fully reproduced in an on-prem development environment.

What Actually Changes?

One of Kubernetes' biggest strengths is portability.

The workloads don't change.

The infrastructure does.

On-Prem                         AWS
────────────────────────────────────────────────────
kubeadm                         Amazon EKS
Longhorn                        Amazon EBS CSI Driver
MetalLB                         AWS Load Balancer Controller
Access Keys & Secret Keys       IAM Roles (IRSA)
Self-managed networking         VPC, Security Groups, Route Tables

Our Deployments, Services, ConfigMaps, Secrets, Helm charts, and container images remain the same.

The differences are almost entirely in the platform layer.

Storage

PersistentVolumeClaims remain identical.

Only the StorageClass changes:

• On-prem: Longhorn
• AWS: Amazon EBS CSI Driver

The applications don't know the difference.

Authentication

Identity is another major difference.

On-prem workloads that require AWS access authenticate using Access Keys and Secret Keys, typically stored as Kubernetes Secrets.

On EKS, we replace this with IAM Roles for Service Accounts (IRSA).

Instead of distributing long-lived credentials, pods assume IAM roles through their Kubernetes ServiceAccount and receive temporary credentials from AWS STS.

This provides:

• No long-lived credentials
• Automatic credential rotation
• Least-privilege access
• Simpler permission management

More importantly, developers don't need to manage AWS credentials inside development environments.

Application Connectivity

This is where the biggest architectural difference exists.

On-Prem Development

The objective is developer velocity.

Developers frequently run services locally while consuming dependencies running inside the shared Kubernetes cluster.

To support this workflow, we maintain on-prem Kubernetes overlays.

These overlays expose selected services using NodePorts and inject node IP-based endpoints into application configuration.

Communication typically looks like this:

Developer Laptop
        │
 VPN (Tailscale/WireGuard)
        │
NodeIP:NodePort
        │
Kubernetes Service
        │
Application

This allows developers to work locally without deploying every service into Kubernetes while still consuming shared platform services.

The networking model is optimized for development—not production.

AWS Development

AWS has a different objective.

It's about standardizing deployments so they behave exactly like production.

Instead of communicating through NodeIP:NodePort, applications use Kubernetes-native service discovery.

Internal communication becomes:

service-a
     │
     ▼
user-service.platform.svc.cluster.local

External traffic is routed through Kong Gateway.

The frontend also stops using IP addresses and instead communicates using environment-specific domains.

For example:

app.dev.company.com
api.dev.company.com

instead of

10.0.1.15:30080

By moving to Kubernetes DNS and gateway-based routing:

• Applications no longer depend on node IP addresses.
• Pods can move between nodes without breaking connectivity.
• Service discovery matches production.
• Networking becomes environment-independent.
• Configuration becomes cleaner and easier to manage.

The application code remains exactly the same.

Only how services discover each other changes.

What Does This Validation Catch?

Running on EKS isn't just checking whether the application starts.

It's validating production assumptions.

Examples include:

• Missing IAM permissions
• Incorrect IRSA configuration
• AWS Load Balancer Controller issues
• Amazon EBS CSI provisioning failures
• Security Group misconfigurations
• VPC routing problems
• Kubernetes DNS resolution failures
• Kong Gateway routing issues
• TLS and ingress configuration
• S3 integration
• Environment-specific overlay mistakes
• Service discovery issues caused by hardcoded IP addresses
• Applications accidentally depending on NodePorts instead of Kubernetes Services

These are problems that typically won't appear in an on-prem development cluster but can cause failures in production if left undiscovered.

By validating against AWS before release, we identify these issues early while the cost of fixing them is still low.

The Biggest Benefit

Designing around standard Kubernetes APIs means we don't maintain separate deployments for on-prem and AWS.

The same:

• Deployments
• Services
• ConfigMaps
• Secrets
• Helm charts
• Container images

are used in both environments.

Only the infrastructure implementation changes.

This gives us the best of both worlds:

• Lower development costs using on-prem infrastructure
• Faster engineering iterations
• Short-lived AWS environments for cloud validation
• Better security using IAM Roles instead of long-lived AWS credentials
• Production-like networking before release
• Minimal environment-specific code
• Confidence that deployments behave consistently across environments

We optimize each environment for a different purpose.

On-prem optimizes for engineering velocity.

AWS optimizes for production confidence.

The applications stay the same.

Only the infrastructure, networking, identity, and platform integrations evolve.