Introduction
Deploying Python code to AWS Lambda seems straightforward—until it’s not. A common mistake many developers make is building their Lambda deployment package inside a Debian-based container, such as python:3.9-bookworm, only to watch it fail silently or with cryptic errors once deployed. At first glance, the environment seems close enough: both are Linux, right? But under the hood, key ABI differences in how Python wheels are compiled can lead to mismatches, and subsequently, broken dependencies.
The primary culprit is the difference in system compatibility tags that pip uses to select and install prebuilt binary wheels. Debian-based images use newer manylinux_2_28 wheels, while AWS Lambda, based on Amazon Linux 2, only supports manylinux_2_17. If your package relies on any native extensions, the wrong build environment guarantees your function will crash on startup.
In this blog post, we’ll break down why this happens, how to detect and debug these issues, and most importantly, how to correctly build Python Lambda packages either by using Amazon Linux 2 containers or adjusting your pip install commands with cross-compilation flags.
Understanding Python Wheels and manylinux Tags
To understand the problem, you must understand Python wheel compatibility. When pip installs packages, it tries to use prebuilt .whl files if available. These wheels are tagged to indicate their compatibility with specific OS and ABI versions—like manylinux_2_28, manylinux_2_17, or manylinux2014. The tag is crucial. If the wheel contains native binaries (e.g., compiled C extensions), running a manylinux_2_28 wheel on a manylinux_2_17 system will likely fail.
Take Pillow as an example. Running pip install Pillow on Debian-based Python 3.9 might fetch:
Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl
While the same command on Amazon Linux 2 yields:
Pillow-10.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
The difference isn’t cosmetic. These wheels are compiled against different libc versions and kernel interfaces. When a Lambda function tries to run a binary built for the wrong environment, it either fails to load or crashes at runtime with vague errors like “undefined symbol” or “segmentation fault.”
Why Debian Fails for Lambda Deployments
When using python:3.9-bookworm to install dependencies, your pip grabs manylinux_2_28 wheels because Debian Bookworm supports them. But AWS Lambda is strict — it runs on Amazon Linux 2, and only supports manylinux_2_17 wheels. That means even if your application installs and runs fine in your local container, it won't work on Lambda. Worse, there’s often no warning during deployment—the failure only surfaces in logs or cold start crashes.
This discrepancy becomes a real issue when working with packages that include native code, such as:
Pillownumpyscipycryptography
You can’t just ignore this issue or rely on luck. A clean local build is no guarantee of a functional Lambda deployment unless it matches the target environment’s expectations.
Solution 1: Use Amazon Linux 2-Based Containers
The most foolproof solution is to install your dependencies inside a container that mimics the Lambda environment—i.e., Amazon Linux 2 with the same Python version. This ensures wheels or compiled binaries match the exact ABI and glibc version that Lambda expects.
You can use images like:
FROM amazonlinux:2
RUN yum install -y python39 python39-pip zip
WORKDIR /app
COPY requirements.txt .
RUN python3.9 -m pip install --upgrade pip
RUN python3.9 -m pip install -r requirements.txt -t /app/python
CMD zip -r9 /lambda-package.zip /app/python
This approach gives you total confidence that what you build will work inside Lambda. You can even use the amazonlinux:2 image to compile wheels or use layers for reuse across multiple functions.
Solution 2: Cross-Compile with pip's --platform Flag
If you’re limited to using Debian (e.g., in a CI pipeline or company-standard environment), there’s a workaround. Use pip’s --platform flag to explicitly request manylinux_2_17 wheels. Combine it with --only-binary to avoid building from source.
Example Dockerfile snippet:
FROM python:3.9-bookworm
COPY requirements.txt .
RUN pip install \
--platform manylinux_2_17_x86_64 \
--only-binary=:all: \
-r requirements.txt \
-t /opt/python
This instructs pip to ignore the native platform and download wheels as if it were running on manylinux_2_17. It’s a powerful option—but also risky. Not all packages offer wheels for manylinux_2_17, and pip may fail if none are available. You also lose the safety net of local testing, since your container isn’t the same as the target runtime.
Still, for teams that can't fully adopt Amazon Linux 2 containers, this can be an effective compromise.
Bonus Tip: Use AWS Lambda Layers for Large Dependencies
If you're working with heavy dependencies like numpy, pandas, or scikit-learn, it’s worth looking into Lambda Layers. These reusable components let you package dependencies separately and share them across functions. Combined with Amazon Linux 2 builds, they improve cold start time and reduce function package sizes.
Build your layer in an Amazon Linux 2 container, zip the /python folder, and upload it via AWS CLI or CDK:
zip -r layer.zip python
aws lambda publish-layer-version --layer-name my-layer --zip-file fileb://layer.zip --compatible-runtimes python3.9
Using layers also enforces consistency across functions and makes updates easier to manage.
Conclusion
Building Python packages for AWS Lambda is not just about writing code — it’s about respecting the underlying platform. Using Debian-based containers to install dependencies seems convenient, but unless you align your environment with Amazon Linux 2 or explicitly request compatible wheels, your functions are ticking time bombs waiting to break in production.
For mission - critical workloads, always build on Amazon Linux 2 or use pip’s --platform workaround with caution. By understanding how pip, manylinux, and Lambda interact, you can eliminate subtle bugs, reduce cold start failures, and ship functions that just work.
Stick to the runtime. Respect the ABI. And when in doubt, test on the same system your code will run on.
Glossary
- ABI (Application Binary Interface): A low-level interface between an application and the operating system, defining how data structures and functions are accessed in machine code.
- manylinux: A set of standards for building portable Linux wheels that can run on various Linux distributions.
- Lambda Layer: A ZIP archive that contains libraries, a custom runtime, or other function dependencies. Layers are a distribution mechanism for libraries, custom runtimes, and other function dependencies.
- Cross-compilation: The process of building software on one platform (e.g., Debian) to run on another (e.g., Amazon Linux 2).
- Cold start: The latency that occurs when a serverless function is invoked for the first time or after a period of inactivity, requiring the cloud provider to allocate resources and initialize the runtime.
- Pillow: A Python Imaging Library that adds image processing capabilities to your Python interpreter.
- numpy: A fundamental package for scientific computing with Python, providing support for large, multi-dimensional arrays and matrices.
- scipy: A Python library used for scientific and technical computing, built on NumPy.
- cryptography: A package designed to expose cryptographic recipes and primitives to Python developers.
- pip: The package installer for Python, used to install and manage software packages written in Python.