When building data-intensive applications with Entity Framework Core, you will eventually face a critical performance challenge: inserting, updating, or deleting thousands of records efficiently.

The standard EF Core approach using SaveChanges() works perfectly for small datasets or even a few hundred rows, but it becomes a significant bottleneck when you need to process thousands of records.

There is a better solution: Entity Framework Extensions provides high-performance bulk operations that can process thousands of records in less than a second. You still use the same code you have in EF Core with a new set of DbContext extension methods.

In this post, we will explore:

• Why inserting 10,000+ rows with EF Core quickly becomes a performance bottleneck
• What happens under the hood when you call SaveChanges() and why it slows down at scale
• How Entity Framework Extensions solve bulk data problems without rewriting your EF Core code
• How to import data at scale using BulkInsert to insert thousands of records in a single operation
• How to apply mass updates with BulkUpdate to efficiently modify large datasets
• How to clean up data with BulkDelete to remove large volumes of records in one database round-trip
• How to perform upserts and sync external systems using BulkMerge with minimal code
• How to run full data synchronization jobs using BulkSynchronize to keep databases in sync reliably

Let's dive in.

You are building an IoT platform that collects telemetry data from thousands of devices every minute. Each device sends temperature readings, humidity levels, and status updates.

Your system needs to insert 50,000 telemetry records into the database every few minutes.

You start with the standard Entity Framework Core approach:

csharp
1app.MapPost("/telemetries", async (IotDbContext dbContext, InsertTelemetryRequest request) =>
2{
3   var telemetryEntries = request.MapToTelemetryEntries();
4   
5   dbContext.Telemetries.AddRange(telemetryEntries);
6   await dbContext.SaveChangesAsync();
7
8   return Results.Ok();
9});

Using the standard SaveChanges() approach, this operation could take several seconds or even minutes and consume excessive memory. This is a common problem when working with Entity Framework Core at scale.

EF Core is excellent for typical CRUD operations, but it was not designed for bulk data operations. When you need to insert, update, or delete thousands of records, the standard SaveChanges() method becomes a performance bottleneck.

Entity Framework Extensions is a library that solves this problem. It extends EF Core with high-performance bulk operations that can handle large datasets efficiently.

All examples in this post were tested on a SQL Server database using the IoT telemetry application.

To understand why EF Core struggles with bulk operations, you need to understand what happens when you call SaveChanges().

Entity Framework Core was designed with a specific workflow in mind: track changes to entities, detect what changed, and generate SQL statements to persist those changes. This workflow is excellent for typical CRUD operations, but it becomes a bottleneck when dealing with thousands of records.

Let's break down what happens step by step when you call SaveChanges() with 10,000 telemetry records.

Step 1: Change Detection

When you call SaveChanges(), EF Core first runs change detection. The change tracker examines every tracked entity to determine what changed since it was loaded or added.

Step 2: SQL Statement Generation

After detecting changes, EF Core generates SQL statements for each entity that needs to be persisted.

For inserts, EF Core generates individual INSERT statements:

sql
1INSERT INTO Telemetries (TelemetryId, DeviceId, ComponentId, Value, Quality, CollectedAt, ReceivedAt)
2VALUES (@p0, @p1, @p2, @p3, @p4, @p5, @p6);
3
4INSERT INTO Telemetries (TelemetryId, DeviceId, ComponentId, Value, Quality, CollectedAt, ReceivedAt)
5VALUES (@p7, @p8, @p9, @p10, @p11, @p12, @p13);
6
7-- ... repeated 10,000 times

EF Core batches these statements to reduce round-trips, but it still generates a separate INSERT statement for each entity. The batching helps, but it doesn't eliminate the overhead of generating and executing thousands of individual statements.

Step 3: Parameter Binding

For each SQL statement, EF Core creates parameters for every column value. With 10,000 telemetry records and 7 columns each, that's 70,000 parameters that need to be created, bound, and sent to the database.

The database provider (SQL Server, PostgreSQL, etc.) needs to parse these parameters, validate them, and prepare the execution plan. This adds significant overhead.

Step 4: Database Execution

Even though EF Core batches statements together, the database still needs to execute each INSERT statement individually.

In SQL Server, EF Core uses a MERGE statement to insert multiple rows at once. However, EF Core is limited by the SQL Server parameter limit (2,100 parameters per batch), which restricts the number of rows or columns in a single batch. Performance drops quickly when entities have many columns.

Step 5: Identity Value Retrieval

If your entity has an identity column (auto-increment primary key), EF Core needs to retrieve the generated value for each inserted record.

Step 6: Change Tracker Update

After the database operations complete, EF Core updates the change tracker to mark all entities as Unchanged. This involves iterating through all 10,000 entities again and updating their state.

What About Raw SQL?

You might think about using raw SQL with a single INSERT statement:

csharp
1var sql = "INSERT INTO Telemetries (TelemetryId, DeviceId, ComponentId, Value, Quality, CollectedAt, ReceivedAt) VALUES ";
2var values = string.Join(", ", telemetryBatch.Select(t => 
3    $"('{t.TelemetryId}', '{t.DeviceId}', '{t.ComponentId}', {t.Value}, {(int)t.Quality}, '{t.CollectedAt:yyyy-MM-dd HH:mm:ss}', '{t.ReceivedAt:yyyy-MM-dd HH:mm:ss}')"));
4
5await dbContext.Database.ExecuteSqlRawAsync(sql + values);

This approach is faster than using SaveChanges(), but it has serious problems:

• SQL injection risk - Building SQL strings from user input is dangerous
• No type safety - You lose compile-time checking
• No navigation properties - You can't work with related entities
• Maintenance burden - You need to update SQL strings when your model changes

If you try to introduce parameters into the raw SQL string, Microsoft.Data.SqlClient will generate multiple insert statements instead of a single batch. And performance will drop even more than with EF Core SaveChanges.

The Solution: Bulk Operations

What you need is a way to insert thousands of records efficiently while keeping the benefits of EF Core: type safety, navigation properties, and maintainability.

This is exactly what Entity Framework Extensions provides. It uses database-specific bulk operations optimized for inserting, updating, deleting, and synchronizing large datasets.

Entity Framework Extensions library extends EF Core with high-performance bulk operations. It solves the performance problems we discussed while keeping all the benefits of working with EF Core.

The library provides methods like BulkInsert, BulkUpdate, BulkDelete, BulkMerge, and BulkSynchronize that work directly with your existing EF Core entities and DbContext.

Let's see how it transforms the telemetry insertion example:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var telemetryBatch = await ReceiveTelemetryFromDevices(); // 10,000 records
5
6await dbContext.BulkInsertAsync(telemetryBatch);
7
8// Execution time: 250ms

The regular SaveChanges took 4.2 seconds to complete. Entity Framework Extensions reduced the execution time to 250ms. That's a 16x difference.

How It Works Under the Hood

Entity Framework Extensions uses database-specific bulk operations that are optimized for handling large datasets. The exact implementation varies by database provider, but the general approach is the same.

For SQL Server, the library uses SqlBulkCopy, which is the exact mechanism that SQL Server Integration Services (SSIS) uses for high-performance data loading.

Here's what happens:

Step 1: Data Preparation

The library reads your entity collection and extracts the property values into an in-memory data structure optimized for bulk operations. This is much faster than EF Core's change detection because it doesn't need to track changes or compare values.

Step 2: Bulk Transfer

Instead of generating individual INSERT statements, the library uses the database's native bulk insert mechanism. For SQL Server, this means using SqlBulkCopy to stream data directly to the database in a binary format.

The database receives the data as a stream and can process it much more efficiently than individual statements:

• Batch index updates instead of updating indexes for each row
• Write to the transaction log in larger chunks
• Use minimal logging in certain scenarios
• Optimize memory usage and buffer management

Step 3: Minimal Overhead

Because the library bypasses EF Core's change tracker, it eliminates most of the overhead we discussed in the previous section. There's no change detection, and no parameter binding for individual records.

Key Benefits

The library provides several significant benefits beyond just performance:

Works with Your Existing Code - You don't need to rewrite your data access layer or change your entity models. The bulk operations work with the same entities and DbContext you already use.

To start using Entity Framework Extensions, install the NuGet package:

bash
1dotnet add package Z.EntityFramework.Extensions.EFCore

Then add the using statement to your code:

csharp
1using Z.EntityFramework.Extensions;

The library extends your DbContext with bulk operation methods. You can call these methods directly on your context:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4// Bulk insert
5await dbContext.BulkInsertAsync(entities);
6
7// Bulk update
8await dbContext.BulkUpdateAsync(entities);
9
10// Bulk delete
11await dbContext.BulkDeleteAsync(entities);
12
13// Bulk merge (upsert)
14await dbContext.BulkMergeAsync(entities);
15
16// Bulk synchronize
17await dbContext.BulkSynchronizeAsync(entities);

When to Use Bulk Operations

You should consider using bulk operations when:

• You need to insert, update, or delete more than 1000 records at once
• Performance is critical for your use case
• You're processing batch jobs or data imports
• You're synchronizing data between systems
• You're experiencing timeout issues with standard EF Core write operations

The BulkInsert method is the most commonly used bulk operation in Entity Framework Extensions. It allows you to insert thousands of records into the database in a single, optimized operation.

With just one line of code, you can use the BulkInsert method to insert large batches of IoT devices or sensor data:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var telemetryBatch = await ReceiveTelemetryFromDevices(); // 10,000 records
5
6await dbContext.BulkInsertAsync(telemetryBatch);

This single line replaces the entire AddRange and SaveChangesAsync workflow.

The method automatically:

• Detects the entity type from the collection
• Maps entity properties to database columns
• Uses the optimal bulk insert mechanism for your database provider

Working with Related Entities

In a real IoT system, you might need to insert devices along with their components. The Device and Component entities have a parent-child relationship:

csharp
1public class Device
2{
3    public Guid DeviceId { get; set; }
4    public string Name { get; set; } = string.Empty;
5    public string DeviceType { get; set; } = string.Empty;
6    public string Manufacturer { get; set; } = string.Empty;
7    public string SerialNumber { get; set; } = string.Empty;
8    public string FirmwareVersion { get; set; } = string.Empty;
9    public string HardwareVersion { get; set; } = string.Empty;
10    public DeviceStatus Status { get; set; }
11    public DateTime LastSeenAt { get; set; }
12    public DateTime RegisteredAt { get; set; }
13    public string Configuration { get; set; } = string.Empty;
14}
15
16public class Component
17{
18    public Guid ComponentId { get; set; }
19    public Guid DeviceId { get; set; }
20    public ComponentType ComponentType { get; set; }
21    public string Name { get; set; } = string.Empty;
22    public string Capability { get; set; } = string.Empty;
23    public string Unit { get; set; } = string.Empty;
24    public string StateValue { get; set; } = string.Empty;
25    public ComponentState State { get; set; }
26    public bool IsActive { get; set; }
27    public DateTime LastUpdatedAt { get; set; }
28}

IncludeGraph option in Entity Framework Extensions lets you bulk insert an entire object graph without manually saving each level:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var devices = await ImportDevicesWithComponents(); // 5,000 devices
5
6await dbContext.BulkInsertAsync(devices, options => options.IncludeGraph = true);

Except for IncludeGraph, Entity Framework Extensions provides various configuration options to customize its behavior. You can configure batch size, timeout, column mappings, and more.

Here is how you can customize Entity Framework Extensions for real-world scenarios.

Real-World Performance Comparison

Let's look at actual performance numbers for different operation sizes on SQL Server:

1. 1,000 Records

   • EF Core SaveChanges: 95.16 ms
   • BulkInsert: 24.10 ms
   • BulkInsertOptimized: 10.15 ms
   • Speedup: ~9x

2. 10,000 Records

   • EF Core SaveChanges: 698.82 ms
   • BulkInsert: 80.35 ms
   • BulkInsertOptimized: 28.24 ms
   • Speedup: ~25x

3. 100,000 Records

   • EF Core SaveChanges: 7,168.83 ms
   • BulkInsert: 1,261.36 ms
   • BulkInsertOptimized: 531.94 ms
   • Speedup: ~14x

Performance improvement increases with dataset size.

If you don't need to return identity or other output values after insertion, you can use the BulkInsertOptimized method from Entity Framework Extensions. BulkInsertOptimized is the fastest bulk operation for large datasets.

Under the hood, Entity Framework Extensions uses a temporary table when outputting values. Instead, BulkInsertOptimized uses the BulkCopy strategy directly into the destination table.

You can explore the full benchmark results in the official Entity Framework Extensions repository.

The BulkUpdate method in Entity Framework Extensions allows you to update thousands of records efficiently without loading them into EF Core's change tracker. This is essential when you need to apply updates to large datasets based on external data or business rules.

The Problem with Standard EF Core Updates

The standard EF Core approach to updating multiple records requires loading them into memory, modifying their properties, and calling SaveChanges:

csharp
1var deviceIds = await GetDeviceIdsForStatusUpdate(); // 5,000 device IDs
2
3var devices = await dbContext.Devices
4    .Where(d => deviceIds.Contains(d.DeviceId))
5    .ToListAsync();
6
7foreach (var device in devices)
8{
9    device.Status = DeviceStatus.Offline;
10    device.LastSeenAt = DateTime.UtcNow;
11}
12
13await dbContext.SaveChangesAsync();

This approach has the same problems we discussed earlier: change tracking overhead, individual UPDATE statements, and slow execution time.

Using BulkUpdate

A common scenario in IoT systems is synchronizing device data with an external system. You receive a file or API response with updated device information and need to apply those updates to your database.

With BulkUpdate, you prepare your entities with the updated values and pass them to the method:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4// Receive updated device data from external system
5var externalDeviceData = await FetchDeviceDataFromExternalApi(); // 10,000 devices
6
7// Map external data to your entity model
8var devicesToUpdate = externalDeviceData.Select(ext => new Device
9{
10    DeviceId = ext.Id,
11    FirmwareVersion = ext.Firmware,
12    HardwareVersion = ext.Hardware,
13    Configuration = ext.Config,
14    LastSeenAt = ext.LastContact
15}).ToList();
16
17await dbContext.BulkUpdateAsync(devicesToUpdate, options =>
18{
19    options.ColumnPrimaryKeyExpression = d => d.DeviceId;
20    options.ColumnInputExpression = d => new
21    {
22        d.FirmwareVersion,
23        d.HardwareVersion,
24        d.Configuration,
25        d.LastSeenAt
26    };
27});

You can create entity instances with just the primary key and the properties you want to update, or read them from the database.

You can also specify which properties to ignore when updating:

csharp
1context.BulkUpdate(customers, options => options.IgnoreOnUpdateExpression =
2    c => new { c.RegisteredAt, c.Manufacturer, c.SerialNumber, c.Configuration } );

This pattern is very efficient for data synchronization scenarios. You don't need to load existing records, compare values, or track changes. You simply specify what to update, and the Entity Framework Extensions library handles the rest.

The BulkDelete method in Entity Framework Extensions allows you to delete thousands of records efficiently without loading them into memory or tracking them in EF Core's change tracker. This is essential for data cleanup operations, archiving old records, and removing decommissioned IoT devices.

The Problem with Standard EF Core Deletes

The standard EF Core approach to deleting multiple records requires loading them into memory and calling SaveChanges:

csharp
1var cutoffDate = DateTime.UtcNow.AddMonths(-6);
2
3var oldTelemetry = await dbContext.Telemetries
4    .Where(t => t.CollectedAt < cutoffDate)
5    .ToListAsync(); // Loads 100,000 records into memory
6
7dbContext.Telemetries.RemoveRange(oldTelemetry);
8await dbContext.SaveChangesAsync();

This approach has serious problems when deleting large datasets:

• High memory consumption - Loading 100,000 records into memory can consume hundreds of megabytes
• Slow execution - EF Core generates individual DELETE statements for each record
• Change tracker overhead - All entities are tracked, adding CPU and memory overhead

Using BulkDelete

With BulkDelete, you can delete records by providing just their primary keys. A common scenario is receiving a list of IDs to delete from an external system:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var deviceIdsToDelete = await GetDecommissionedDeviceIds(); // 100,000 device IDs
5
6var devicesToDelete = deviceIdsToDelete.Select(id => new Device
7{
8    DeviceId = id
9}).ToList();
10
11await dbContext.BulkDeleteAsync(devicesToDelete, options =>
12{
13    options.ColumnPrimaryKeyExpression = d => d.DeviceId;
14});

This deletes 100,000 devices in less than a second without loading any entity data into memory.

The BulkMerge method in Entity Framework Extensions performs upsert operations: it inserts new records and updates existing ones in a single operation. This is essential when synchronizing data with external systems where you don't know which records already exist in your database.

Let's explore how to use BulkMerge to synchronize device data, import telemetry from external sources, and keep your database in sync with external APIs.

Without BulkMerge, you need to manually check which records exist and then insert or update accordingly.

With BulkMerge, you provide your data, and the Entity Framework Extensions library handles everything:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var externalDevices = await FetchDevicesFromExternalApi(); // 10,000 devices
5
6var devicesToMerge = externalDevices.Select(ext => new Device
7{
8    DeviceId = ext.DeviceId,
9    Name = ext.Name,
10    DeviceType = ext.Type,
11    Manufacturer = ext.Manufacturer,
12    Status = ext.Status,
13    LastSeenAt = ext.LastContact
14}).ToList();
15
16await dbContext.BulkMergeAsync(devicesToMerge);

Under the hood, BulkMerge uses a temporary table approach:

1. Creates a temporary table in the database
2. Inserts your data into the temporary table
3. Performs a SQL MERGE operation (or equivalent) that:
   • Inserts rows that don't exist in the target table
   • Updates rows that already exist
4. Cleans up the temporary table

This is pretty efficient even for massive datasets.

Configuring the Merge Key

By default, BulkMerge uses the primary key to determine if a record exists. You can specify a different key:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var externalDevices = await FetchDevicesFromExternalApi();
5
6var devicesToMerge = externalDevices.Select(ext => new Device
7{
8    SerialNumber = ext.SerialNumber, // Use serial number as the key
9    Name = ext.Name,
10    DeviceType = ext.Type,
11    Status = ext.Status
12}).ToList();
13
14await dbContext.BulkMergeAsync(devicesToMerge, options =>
15{
16    options.ColumnPrimaryKeyExpression = d => d.SerialNumber;
17});

This is useful when your external system uses a different identifier than your primary key.

The BulkSynchronize method in Entity Framework Extensions performs a complete synchronization between your source data and the database. Unlike BulkMerge, which only inserts and updates records, BulkSynchronize also deletes records that exist in the database but are missing from your source data.

This is essential when you need to maintain an exact mirror of an external system in your database. If a device is removed from the external system, it should also be removed from your database.

The Difference Between BulkMerge and BulkSynchronize

BulkMerge performs upsert operations: it inserts new records and updates existing ones, but it never deletes anything. This is perfect when you want to add or update data without removing existing records.

BulkSynchronize performs a full synchronization: it inserts new records, updates existing ones, and deletes records that are no longer present in your source data.

Let's look at a scenario to understand the difference:

Your database currently has these devices:

• Device A, Device B, Device C

Your external system returns:

• Device A (updated), Device B (unchanged), Device D (new)

BulkMerge result:

• Device A (updated), Device B (unchanged), Device C (still there), Device D (inserted)

BulkSynchronize result:

• Device A (updated), Device B (unchanged), Device D (inserted)
• Device C is deleted because it's not in the source data

A common scenario is synchronizing your device registry with an external device management system. The external system is the source of truth, and your database should always reflect exactly what's in that system.

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4// Fetch the complete device list from the external system
5var externalDevices = await FetchAllDevicesFromExternalApi(); // 10,000 devices
6
7var devicesToSync = externalDevices.Select(ext => new Device
8{
9    DeviceId = ext.DeviceId,
10    Name = ext.Name,
11    DeviceType = ext.Type,
12    Manufacturer = ext.Manufacturer,
13    SerialNumber = ext.SerialNumber,
14    Status = ext.Status,
15    LastSeenAt = ext.LastContact
16}).ToList();
17
18await dbContext.BulkSynchronizeAsync(devicesToSync);

After this operation:

• New devices from the external system are inserted
• Existing devices are updated with the latest data
• Devices that exist in your database but are missing from the external system - are deleted

How BulkSynchronize Works Under the Hood

BulkSynchronize uses a temporary table approach similar to BulkMerge, but with an additional deletion step:

1. Creates a temporary table in the database
2. Inserts your source data into the temporary table
3. Performs a SQL MERGE operation that:
   • Inserts rows that don't exist in the target table
   • Updates rows that already exist
4. Deletes rows from the target table that don't exist in the temporary table
5. Cleans up the temporary table

This is much more efficient than manually comparing records and performing separate insert, update, and delete operations.

Synchronizing with Filters

In many scenarios, you don't want to synchronize all records in a table. You might want to synchronize only a subset based on certain criteria.

For example, you might want to synchronize only devices of a specific type:

csharp
1// @nuget: Z.EntityFramework.Extensions.EFCore
2using Z.EntityFramework.Extensions;
3
4var externalSensors = await FetchSensorsFromExternalApi(); // Only sensor devices
5
6var sensorsToSync = externalSensors.Select(ext => new Device
7{
8    DeviceId = ext.DeviceId,
9    Name = ext.Name,
10    DeviceType = "Sensor", // Only "Sensor" devices should be synchronized
11    Manufacturer = ext.Manufacturer,
12    Status = ext.Status
13}).ToList();
14
15await dbContext.BulkSynchronizeAsync(sensorsToSync, options =>
16{
17    options.SynchronizeKeepIdentity = true;
18    options.ColumnSynchronizeDeleteKeySubsetExpression = d => d.DeviceType;
19});

The ColumnSynchronizeDeleteKeySubsetExpression option ensures that only sensor devices are deleted during synchronization. Other device types in your database remain untouched.

Building data-intensive applications with Entity Framework Core can be challenging, especially in terms of performance.

Not anymore, if you use Entity Framework Extensions library.

Entity Framework Extensions is a commercial library with a free trial, which you can use to see if it's right for your project. The library requires a license for production use. See more information on licensing here.

For commercial use, the time saved in development and the improved application performance typically pay for the license within the first month of use.

Try the free trial to see how these methods can transform your bulk data operations from a performance bottleneck into one of the fastest parts of your application.

Many thanks to ZZZ Projects for sponsoring this blog post.

Hope you find this newsletter useful. See you next time.