Serverless & Event-Driven Operations

Event Sources & Triggers

18 min Lesson 4 of 28

Event Sources & Triggers

A Lambda function is inert until something invokes it. AWS groups invocation into three concurrency models — synchronous (API Gateway, ALB, Cognito triggers), asynchronous (S3 events, SNS, EventBridge), and poll-based streaming (SQS, Kinesis, DynamoDB Streams). Each model differs in retry semantics, error handling, and back-pressure behaviour. Choosing the wrong model is the root cause of most production event-driven bugs.

API Gateway — Synchronous HTTP Entry Point

API Gateway (HTTP API v2 or REST API v1) is the dominant synchronous trigger. The caller blocks waiting for a response; Lambda's concurrency limit directly becomes your service's throughput ceiling. HTTP API v2 latency overhead is roughly 1–2 ms; REST API adds ~5–10 ms for mapping templates and authorisers.

Practical rules at scale: set a Reserved Concurrency on each function so a traffic spike does not starve shared account limits (default 1,000 per region, soft limit). Use provisionedConcurrencyConfig to eliminate cold starts on P99 paths. Enable payload compression (minimumCompressionSize: 1024) and response caching for GET endpoints. Wire a Lambda Authoriser (JWT or IAM) rather than doing auth inside the function — it is cached per TTL and keeps your business logic clean.

# Serverless Framework — HTTP API v2 trigger with authoriser and provisioned concurrency
functions:
  orderApi:
    handler: src/orders.handler
    reservedConcurrency: 500
    provisionedConcurrencyConfig:
      provisionedConcurrentExecutions: 10
    events:
      - httpApi:
          path: /orders/{orderId}
          method: GET
          authorizer:
            name: jwtAuthorizer
            identitySource: $request.header.Authorization

  jwtAuthorizer:
    handler: src/auth.handler

API Gateway has a hard 29-second integration timeout. If your Lambda runs longer than that the client receives a 504 even though the function continues to execute. For long-running work, return 202 immediately and push to SQS or Step Functions.

SQS — Async Decoupling with Back-Pressure

SQS is the most operationally robust Lambda trigger. Lambda's internal poller scales from 5 concurrent batches up to 1,000 (Standard queues) or 300 × partition count (FIFO). Key parameters that matter in production:

BatchSize (1–10,000): larger batches mean higher throughput but longer per-item latency. For fan-out workers processing DB writes, 100–500 is typical.
MaximumBatchingWindowInSeconds (0–300): Lambda waits this long to fill a batch before invoking. Reduces invocations at the cost of latency; set 0 for real-time pipelines.
FunctionResponseTypes: [ReportBatchItemFailures]: the single most impactful SQS config. Without it, a single failed item in a batch causes all items to return to the queue, triggering duplicates. With it, your function returns a batchItemFailures array and only the failed items are re-queued.
Dead Letter Queue (DLQ): attach one to the SQS queue (not the function) so messages that exhaust maxReceiveCount land in a known location for analysis.

# CloudFormation / SAM — SQS trigger with partial batch failure reporting
OrderProcessorFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: src/processor.handler
    Events:
      SQSTrigger:
        Type: SQS
        Properties:
          Queue: !GetAtt OrderQueue.Arn
          BatchSize: 100
          MaximumBatchingWindowInSeconds: 5
          FunctionResponseTypes:
            - ReportBatchItemFailures

OrderQueue:
  Type: AWS::SQS::Queue
  Properties:
    VisibilityTimeout: 360        # 6x function timeout
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt OrderDLQ.Arn
      maxReceiveCount: 3

OrderDLQ:
  Type: AWS::SQS::Queue

Set the SQS VisibilityTimeout to at least 6× your Lambda timeout. If your function times out at 60 s but the queue's visibility timeout is 30 s, the message becomes visible again while the function is still executing — you get a duplicate invocation mid-flight.

EventBridge — Event Bus & Scheduled Rules

EventBridge (formerly CloudWatch Events) is AWS's event router. It sits at the centre of almost every event-driven architecture because it decouples producers from consumers entirely — the producer emits a structured JSON event to a bus; rules filter and fan out to targets (Lambda, SQS, SNS, Step Functions, API destinations, even cross-account buses). The default event bus receives AWS service events; custom buses are best practice for application events.

EventBridge is asynchronous by default: Lambda receives the event and EventBridge does not wait for a response. Retries are configurable per target (0–185) with exponential back-off up to 24 hours. Wire a DLQ on the EventBridge target rule, not just on the function, to capture events that failed before Lambda ever ran.

Schema Registry lets you codegen typed bindings from your event schemas — critical for large teams to avoid silent schema drift. Enable it on your custom bus and version your schemas with SchemaVersion metadata in each event.

Three Lambda invocation models — synchronous (blocks caller), asynchronous (fire-and-forget with retries), and poll-based (Lambda manages polling and back-pressure).

S3 Events — Object-Level Notifications

S3 event notifications (s3:ObjectCreated:*, s3:ObjectRemoved:*, etc.) are delivered asynchronously. Each notification is a single JSON payload describing the bucket, key, size, and ETag. Practical points that bite teams in production:

At-least-once delivery: S3 events are not exactly-once. Under very high object creation rates (thousands/s) you can receive duplicate notifications for the same object. Design your processor to be idempotent — e.g. write a processed marker to DynamoDB with a conditional expression before doing expensive work.
No delivery guarantee for very small objects: writes under a few milliseconds apart on the same key can collapse into one notification. Use EventBridge notification delivery from S3 (the newer path) if you need higher fidelity — it routes through the event bus with retry and DLQ support.
Event filtering: add prefix/suffix filters on the notification config to avoid triggering your function on its own output (classic infinite-loop bug: function reads raw/, writes processed/, but notification covers all keys).

# Terraform — S3 notification to Lambda with prefix filter and EventBridge alternative
resource "aws_s3_bucket_notification" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.processor.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "raw/"          # only trigger on raw/ uploads
    filter_suffix       = ".json"
  }
}

# Grant S3 permission to invoke Lambda
resource "aws_lambda_permission" "s3_invoke" {
  statement_id  = "AllowS3Invoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.processor.function_name
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.uploads.arn
}

# --- Preferred for production: route via EventBridge for retry + DLQ ---
resource "aws_s3_bucket_notification" "uploads_eb" {
  bucket      = aws_s3_bucket.uploads.id
  eventbridge = true                       # all events go to default bus
}

EventBridge Rules — Filtering and Fan-Out

EventBridge rules match events using content-based filtering — you can filter on any field in the event JSON, including nested fields and arrays. A well-structured event schema with a typed detail-type and source field dramatically reduces rule complexity across a large service mesh.

# AWS CLI — put a rule that matches OrderPlaced events over $1000
aws events put-rule \
  --name "HighValueOrders" \
  --event-bus-name "orders-bus" \
  --event-pattern '{
    "source": ["com.acme.orders"],
    "detail-type": ["OrderPlaced"],
    "detail": {
      "amount": [{"numeric": [">", 1000]}],
      "currency": ["USD"]
    }
  }' \
  --state ENABLED

# Add Lambda target with retry and DLQ
aws events put-targets \
  --rule "HighValueOrders" \
  --event-bus-name "orders-bus" \
  --targets '[{
    "Id": "fraud-check",
    "Arn": "arn:aws:lambda:us-east-1:123456789012:function:FraudCheck",
    "RetryPolicy": {"MaximumRetryAttempts": 3, "MaximumEventAgeInSeconds": 3600},
    "DeadLetterConfig": {"Arn": "arn:aws:sqs:us-east-1:123456789012:HighValueDLQ"}
  }]'

EventBridge throughput default is 10,000 events/second per bus per region (soft limit, increasable). At this scale, fan-out to hundreds of Lambda targets is routine. Cross-account and cross-region event buses let you build org-wide event fabrics with a single control plane — the pattern used by Amazon's own internal microservices.

Choosing the Right Trigger

Senior engineers frame trigger selection around four axes: latency tolerance, ordering requirement, back-pressure, and failure visibility. Synchronous (API GW) when the caller needs an immediate answer. SQS when you need durable queuing, back-pressure, and partial-batch failure handling. EventBridge when you need content-based routing across services or accounts. S3 events (via EventBridge) when the trigger is object lifecycle. Kinesis Data Streams when you need ordered, replayable, high-throughput streaming — the one trigger not detailed here but covered in Tutorial 5.

One anti-pattern seen repeatedly in production: chaining synchronous Lambda invocations (lambda.invoke from within a function). This couples latency, burns double concurrency, and hides failures. Replace every synchronous chain with SQS or EventBridge — the caller emits, the callee polls, and the two scale and fail independently.