Event-Driven Microservices Break in Ways That Are Hard to See (Sponsored)
When a message gets lost between services, that's where end-to-end observability makes the difference.
Datadog put together a free eBook showing how real engineering teams use full-stack observability to cut MTTR, reduce tooling costs, and connect engineering metrics to business outcomes.
Your CI Is the Slowest Part of Your Workflow β Depot CI Fixes That (Sponsored)
If your team uses GitHub Actions, you already know the pain: jobs queue for minutes, every run starts cold, and debugging a failed step means re-running the entire pipeline. As AI tools speed up how fast we write code, CI has become the biggest bottleneck in shipping it.
Depot CI is a new CI engine built to run your existing GitHub Actions workflows on a faster foundation. Jobs start in seconds instead of minutes. Steps run in parallel within a single job, so you don't have to split workflows across many jobs just to save time. And if something breaks, you can SSH into a running job, replay from a specific failed step, or search logs across runs. No more "commit and pray."
Migration takes one command: depot ci migrate. It finds your existing workflows from GitHub Actions, GitLab, Jenkins, Bitbucket and many others.
You can also define custom runner images with your dependencies pre-installed, so every job starts ready instead of downloading packages from scratch. The entire system is API-first, which means both engineers and AI agents can trigger and monitor runs programmatically.
In Microservices Architecture, services need to communicate with each other.
The simplest approach is synchronous HTTP calls, but it creates tight coupling between services.
If one service is slow or unavailable, it affects all dependent services.
Message brokers solve this problem by enabling asynchronous, message-based communication between your services.
Services publish events to the broker, and other services consume them independently.
When you are working in Azure, you can use Azure Service Bus to implement a message broker.
It supports both point-to-point and publish-subscribe messaging patterns.
In this post, we will explore:
Azure Service Bus: Core Concepts
How To Create Azure Service Bus in Azure Portal
Setting Up Azure Service Bus with .NET Aspire
How to Publish and Consume Messages with Azure Service Bus
The Event-Driven Pipeline: Payments, Fraud Detection, and Notifications
Running the Application Locally with Azure Service Bus Emulator
Deploying Microservices to Azure with Aspire
Azure Service Bus vs Other Azure Messaging Services
Azure Service Bus is a fully managed enterprise message broker in the Azure Cloud.
It supports two messaging patterns: Queues and Topics with Subscriptions.
Queues implement point-to-point communication.
One sender sends a message, and one receiver processes it.
Each message is consumed by exactly one consumer.
Use queues when a message should be processed by a single service.
For example, sending a stock update command to the Stocks service.
Topics with Subscriptions implement publish-subscribe communication.
One sender publishes a message to a topic, and multiple subscribers can each receive a copy of that message.
Each subscription acts as its own virtual queue.
Use topics when multiple services need to react to the same event.
For example, when a payment is processed, both the Fraud Detection service and the Notifications service may need to be notified.
Our solution has five microservices that communicate through Azure Service Bus:
Product-Service - manages products and handles purchases
Stock-Service - manages product stock levels
Payment-Service - processes payments and tracks their status
FraudDetection-Service - analyzes payments for fraud risk
Notification-Service - sends email notifications
Here is the message flow:
When a user purchases a product:
Product-Service publishes UpdateStockEvent to the update-stock queue and PurchaseCompletedEvent to the payment-created queue
Stock-Service consumes from update-stock and updates the stock count
Payment-Service consumes from payment-created, creates a payment record, and publishes PaymentRegisteredEvent to the payment-registered topic
FraudDetection-Service subscribes to the payment-registered topic, analyzes the payment, and publishes FraudDecisionEvent to the fraud-decision queue
Payment-Service consumes from fraud-decision, updates the payment status, and publishes PaymentProcessedEvent to the payment-processed topic
Notification-Service subscribes to the payment-processed topic and sends a notification
In our project, we use both:
Queues for update-stock, payment-created, and fraud-decision (one sender, one receiver)
Topics for payment-registered and payment-processed (one sender, multiple receivers)
When a consumer receives a message, Service Bus does not delete it immediately.
It locks the message and hides it from other consumers for a configurable lock duration.
The consumer has that time to process the message and complete the call.
If it completes successfully, Service Bus deletes the message.
If the handler throws or the lock expires, Service Bus releases the message and makes it available for retry.
Service Bus tracks how many times a message has been delivered using the delivery count.
When the delivery count exceeds MaxDeliveryCount (default: 10), Service Bus automatically moves the message to the dead-letter queue.
At-least-once delivery is the core guarantee Service Bus provides.
A message will be delivered at least once, but in rare cases it may be delivered more than once β for example, if a network failure prevents the completion acknowledgement from reaching the Service Bus.
Your consumers must be idempotent: processing the same message twice must produce the same result.
Service Bus can automatically discard duplicate messages within a configurable time window.
Enable it on a queue or topic at creation time.
Service Bus tracks messages by their MessageId.
If a message with the same MessageId arrives within the duplicate detection window (default: 10 minutes), Service Bus silently drops it.
Use duplicate detection when your producer may retry sending β for example, after a timeout or a transient network failure.
Sessions enable ordered, grouped processing of related messages.
Set a SessionId on each message to group them together.
Service Bus guarantees that all messages with the same SessionId are processed in order by a single consumer at a time.
No two consumers can process messages from the same session simultaneously.
Use sessions when order matters β for example, processing all state transitions for the same payment in sequence.
Sessions require RequiresSession = true on the queue or subscription.
First, create a new Azure Service Bus namespace in the Azure Portal.
Go to the Azure Portal and search for "Service Bus":
Create a new Service Bus namespace with the following settings:
Resource group: Choose your resource group or create a new one
Namespace name: Choose a globally unique name
Location: Select the region closest to your services
Pricing tier: Select Standard
The pricing tier is important.
The Basic tier only supports queues.
If you need topics and subscriptions (which we do), you must use at least the Standard tier.
For most development and production workloads, Standard is sufficient.
Choose Premium only if you need dedicated resources, virtual network integration, or messages larger than 256 KB.
The RunAsEmulator() method tells Aspire to run a local Azure Service Bus emulator in Docker.
This means you do not need an Azure subscription to develop locally.
Aspire automatically injects the Service Bus connection string into each service via WithReference(serviceBus).
For queues, we use AddServiceBusQueue.
For topics, we use AddServiceBusTopic followed by AddServiceBusSubscription to create subscriptions on that topic.
Each service project needs the Aspire Azure Service Bus client package:
This opt-in switch activates the Azure SDK's built-in ActivitySource, which emits OpenTelemetry spans for every send and receive operation.
Without it, the Aspire dashboard Traces tab will not show Service Bus activity, and you will not be able to see distributed traces across your services.
You also need to register the Azure Service Bus activity source in the Aspire ServiceDefaults project.
Open Extensions.cs and add .AddSource("Azure.Messaging.ServiceBus") to the OpenTelemetry tracing configuration:
csharp
1.WithTracing(tracing => tracing
2.AddAspNetCoreInstrumentation()3// ... other sources4.AddSource("Azure.Messaging.ServiceBus"));
This tells OpenTelemetry to collect and export spans produced by the Azure SDK.
After both changes, the Aspire dashboard will show end-to-end distributed traces across all five services.
All events are defined as C# records in a shared Shared.Contracts project.
This project has no dependencies and is referenced by all services.
UpdateStockEvent carries the product ID and the quantity change (negative for purchases).
PurchaseCompletedEvent carries the full purchase details needed to create a payment.
Other events in the project follow the same pattern: PaymentRegisteredEvent, FraudDecisionEvent, and PaymentProcessedEvent.
Creating a ServiceBusSender for every message is expensive.
We cache senders by queue/topic name so that each destination gets one reusable sender.
ConcurrentDictionary.GetOrAdd ensures thread safety when multiple threads publish to the same destination.
We set a unique MessageId on each message.
Service Bus uses this for duplicate detection β if a message with the same MessageId arrives within the duplicate detection window, Service Bus silently drops it.
The publisher is registered as a singleton because ServiceBusSender is thread-safe and designed for reuse:
Here is how Product-Service publishes events when a user purchases a product:
csharp
1app.MapPost("/products/{id:int}/purchase",async(2int id,3PurchaseProductRequest request,4ProductService productService,5IStocksApiClient stocksApiClient,6IServiceBusPublisher serviceBusPublisher)=>7{8var product =await productService.GetByIdAsync(id);9if(product isnull)10{11return Results.NotFound($"Product with ID {id} not found.");12}1314var hasEnoughStock =await stocksApiClient.HasEnoughStockAsync(id, request.Quantity);15if(!hasEnoughStock)16{17return Results.Conflict("Not enough stock for this purchase.");18}1920var totalPrice = product.Price * request.Quantity;2122// Publish stock update event (async via Service Bus)23await serviceBusPublisher.PublishAsync("update-stock",24newUpdateStockEvent(id,-request.Quantity));2526// Publish purchase completed event for payment processing27await serviceBusPublisher.PublishAsync("payment-created",28newPurchaseCompletedEvent(29TransactionId: Guid.NewGuid().ToString(),30ProductId: product.Id,31ProductName: product.Name,32Quantity: request.Quantity,33TotalPrice: totalPrice,34UserId:"user-1",35Timestamp: DateTime.UtcNow));3637return Results.Ok(newPurchaseProductResponse(38ProductId: product.Id,39Quantity: request.Quantity,40TotalPrice: totalPrice));41});
Notice that the stock check is still synchronous via HTTP.
We need an immediate answer to know if the purchase can proceed.
But the stock update is now asynchronous.
We publish an UpdateStockEvent to the update-stock queue and return immediately.
The Stocks service will process it in the background.
Next, we create a generic consumer BackgroundService that wraps ServiceBusProcessor from the Azure SDK.
It deserializes incoming messages and dispatches them to the corresponding IServiceBusMessageHandler<T>:
csharp
1publicclassServiceBusQueueConsumerService<TMessage>(2ServiceBusClient client,3IServiceScopeFactory scopeFactory,4ILogger<ServiceBusQueueConsumerService<TMessage>> logger,5string queueName): BackgroundService
6{7protectedoverrideasyncTaskExecuteAsync(CancellationToken stoppingToken)8{9var processor = client.CreateProcessor(queueName,newServiceBusProcessorOptions10{11 AutoCompleteMessages =true,12 MaxConcurrentCalls =1013});1415 processor.ProcessMessageAsync +=async args =>16{17try18{19var message = JsonSerializer.Deserialize<TMessage>(args.Message.Body.ToString());20if(message isnotnull)21{22usingvar scope = scopeFactory.CreateScope();23var handler = scope.ServiceProvider.GetRequiredService<IServiceBusMessageHandler<TMessage>>();24await handler.HandleAsync(message, stoppingToken);25}26}27catch(Exception ex)28{29 logger.LogError(ex,"Error processing message from queue {QueueName}", queueName);30throw;31}32};3334 processor.ProcessErrorAsync += args =>35{36 logger.LogError(args.Exception,"Error in Service Bus processor for queue {QueueName}", queueName);37return Task.CompletedTask;38};3940await processor.StartProcessingAsync(stoppingToken);4142// Keep the service alive while the processor is running.43while(!stoppingToken.IsCancellationRequested){}4445await processor.StopProcessingAsync(stoppingToken);46await processor.DisposeAsync();47}48}
ServiceBusProcessor** is the Azure SDK's built-in message pump. It pulls messages from the queue and calls your handler.
We use the following options:
AutoCompleteMessages = true - tells SDK to automatically mark messages as completed after your handler finishes without throwing. If your handler throws an exception, the message becomes available for retry.
MaxConcurrentCalls = 10 - allows processing up to 10 messages in parallel for better throughput.
The while loop at the end keeps the BackgroundService alive while the processor runs.
We register the handler and the consumer service in the DI container:
The handler is registered as scoped, so it gets a fresh DbContext and other scoped dependencies per message.
The consumer service is registered via a factory lambda that passes the queue name.
For consuming messages from a topic, we use a different consumer class, which is identical to this one, except for how it creates a processor:
Creates a Payment entity in the database with status Processing
Publishes a PaymentRegisteredEvent to the payment-registered topic
We publish to a topic here and the Fraud Detection service subscribes to it.
In the future, you could add more subscribers (analytics, audit logging) without changing the Payments service.
This handler subscribes to the payment-registered topic via ServiceBusTopicConsumerService (registered in DI).
The topic name and subscription name are specified during DI registration.
The fraud detection engine evaluates risk factors like high-risk countries, suspicious amounts, late-night transactions, and high-risk card BINs.
Each factor contributes a weighted score, and the total determines the decision:
When the fraud decision arrives, Payment-Service updates the payment status:
csharp
1publicclassFraudDecisionMessageHandler(2IPaymentRepository paymentRepository,3IServiceBusPublisher serviceBusPublisher,4ILogger<FraudDecisionMessageHandler> logger)5: IServiceBusMessageHandler<FraudDecisionEvent>6{7publicasyncTaskHandleAsync(FraudDecisionEvent message,CancellationToken cancellationToken)8{9var paymentStatus = message.Decision switch10{11"Allow"=> PaymentStatus.Confirmed,12"Review"=> PaymentStatus.Reviewing,13"Block"=> PaymentStatus.Rejected,14 _ => PaymentStatus.Reviewing
15};1617var payment =await paymentRepository.GetByTransactionIdAsync(message.TransactionId);18if(payment isnull)19{20 logger.LogWarning("Payment {TransactionId} not found for fraud decision", message.TransactionId);21return;22}2324 payment.Status = paymentStatus;25await paymentRepository.UpdateAsync(payment);2627var paymentProcessedEvent =newPaymentProcessedEvent(28 payment.TransactionId,29 payment.Status.ToString(),30 DateTime.UtcNow);3132await serviceBusPublisher.PublishAsync("payment-processed", paymentProcessedEvent, cancellationToken);3334 logger.LogInformation(35"Payment {TransactionId} processed with status {Status}",36 payment.TransactionId, payment.Status);37}38}
The handler maps the fraud decision to a payment status using a switch expression.
After updating the database, it publishes a PaymentProcessedEvent to the payment-processed topic.
The final step in the pipeline:
Notification-Service subscribes to the payment-processed topic and sends an email notification.
The Azure Service Bus emulator is a Docker container that runs a local instance of Service Bus.
This means you do not need an Azure subscription to develop and test locally.
To run the application, set the AppHost project as the startup project and press F5.
Aspire will:
Start the PostgreSQL container
Start the Redis container
Start the Azure Service Bus emulator container
Create all queues, topics, and subscriptions defined in AppHost.cs
Start all five API services with injected connection strings
Open the Aspire dashboard in your browser
The first run may take a few minutes while Docker pulls the container images. Subsequent runs will be much faster.
In the Aspire dashboard, you can view:
Resources - All running services and their health status
Console logs - Real-time logs from each service
Structured logs - Filterable and searchable logs
Traces - Distributed traces across services
Here is what our Aspire dashboard looks like:
Here, you can see all the queues and topics created by the AppHost.
To test the full pipeline, call the purchase endpoint on Product-Service:
bash
1POST /products/{id}/purchase
2{3"quantity":14}
Then watch the logs in the Aspire dashboard.
You will see messages flowing in distributed traces through all five services in sequence:
For production, you need to switch from the emulator to a real Azure Service Bus.
Aspire supports deploying to Azure Container Apps using the Azure Developer CLI (azd).
First, install the Azure Developer CLI:
bash
1winget install microsoft.azd
Then run the following commands from the project's solution folder:
bash
1azd auth login
2azd init
azd auth login authenticates with Azure and downloads the Azure Developer CLI configuration.
You only need to run this once.
azd init initializes the Azure deployment configuration.
It scans your solution and generates Bicep infrastructure-as-code templates for all resources.
When deploying to Azure, we need a different configuration for AzureServiceBus with the RunAsEmulator() call removed:
csharp
1if(builder.Environment.IsProduction())2{3// Existing Azure Service Bus: queues and topics already exist4var serviceBus = builder.AddConnectionString("service-bus");56 stocksApi.WithReference(serviceBus);7 productsApi.WithReference(serviceBus);8 paymentsApi.WithReference(serviceBus);9 fraudDetectionApi.WithReference(serviceBus);10 notificationsApi.WithReference(serviceBus);11}
If we use an existing Service Bus, we can just use the AddConnectionString call and pass the connection string to the WithReference method to all services.
For this to work, we need to create queues and topics on the existing Azure Service Bus via Azure Portal or the Azure CLI:
First, add the queues:
payment-created
update-stock
fraud-decision
Second, create the topics:
payment-registered
payment-processed
Third, create subscriptions for the topics:
payment-registered -> fraud-detection
payment-processed -> notifications
Finally, run the azd up command to provision Azure resources and deploy your application.
bash
1azd up
Aspire will create:
Azure Container Apps for each of your five services
Azure Database for PostgreSQL flexible server
Azure Cache for Redis
Azure Container Registry to store Docker images
All connection strings and service discovery are wired automatically, just as they would be locally.
Let's test the full pipeline by calling the purchase endpoint on the purchase API endpoint:
If you have created an Azure Service Bus in the Azure portal, you need to use the AddConnectionString to connect all services to it.
You can also AddConnectionString to connect to an existing Service Bus when running your services locally (if public access is enabled for this service).
If you want Aspire to create the Service Bus for you, use the AddAzureServiceBus as you have seen earlier.
Azure Service Bus vs Other Azure Messaging Services
Azure offers three messaging services. They are often confused because their names sound similar.
Azure Service Bus is an enterprise message broker with guaranteed delivery, dead-letter queue, sessions, and ordering.
Use it for reliable service-to-service communication β commands, events, and workflows between microservices.
This is what we use in this post.
Azure Event Hubs is a high-throughput event streaming platform built for millions of events per second.
It has no dead-letter queue and no per-message acknowledgement.
Use it for telemetry pipelines, log ingestion, and analytics workloads.
Azure Event Grid is a reactive eventing service for Azure infrastructure events.
It is push-based and serverless.
Use it to react to Azure resource changes β for example, when a blob is uploaded to Storage or a VM is deallocated.
If you need reliable delivery with retries and a dead-letter queue β use Service Bus.
If you need massive-throughput event streaming β use Event Hubs.
If you need to react to Azure resource events β use Event Grid.
Hope you find this newsletter useful. See you next time.
You can download source code for this newsletter for free
Covers everything: C#, ASP.NET Core, EF Core, system design β answer each question first, reveal the solution, and a test after every chapter proves it stuck. Finish, and you earn a verifiable certificate for your LinkedIn.
Not sure where you stand? Take the free .NET Interview Run:
Find out your real level β Junior to Senior+
A realistic mock .NET interview β across 13 areas of C#, .NET, ASP.NET Core and System Design
No credit card required. When you finish, you get a personalized report: your level, your strongest and weakest areas, and where to focus next β the perfect way to benchmark yourself before diving into the Playbook.