TIL / File-hash change detection keeps a big Lambda fleet's CI fast
File-hash change detection keeps a big Lambda fleet's CI fast
The problem
A CI/CD pipeline that redeploys a whole fleet of Lambda functions on every push works fine at small scale. Past a hundred-plus functions it gets slow, and most of that time is spent repackaging and redeploying code that didn’t change at all in this push.
The fix
Hash each function’s source directory, compare it against the hash recorded from the last successful deploy (stored alongside the stack, e.g. in SSM Parameter Store or a small DynamoDB table), and only redeploy the functions whose hash moved.
import hashlib
from pathlib import Path
def hash_dir(path: Path) -> str:
h = hashlib.sha256()
for f in sorted(path.rglob("*.py")):
h.update(f.read_bytes())
return h.hexdigest()
changed = [fn for fn in functions if hash_dir(fn.source_dir) != last_deployed_hash[fn.name]]
Gotcha
Hash the function’s actual dependency lockfile too, not just its own source - a shared library bump with no change to the function’s own code still needs a redeploy, and skipping that is a sneaky way to end up running stale code in production while CI reports green.