OutOfMemoryException from downloading in batch mode.

Mar 11, 2014 at 1:22 AM
Edited Mar 11, 2014 at 11:59 PM

For some reason I am receiving a OutOfMemoryException - Insufficient memory to continue the execution of the program.

I am using the Windows Phone 8 emulator.

I am have batching enabled set to 2048KB.

I can sync small data fine using this method (5,000 rows) when I attempt to sync larger data:
300,000 rows I experience this error.

Mar 11, 2014 at 3:07 AM
Edited Mar 11, 2014 at 3:08 AM
I think the code for batching is implemented through recursion so the memory/list data builds up and is never disposed/garbage collected causing a OutOfMemoryException.

I will investigate, but I feel that is the issue...

Any feed back is welcome.
Mar 11, 2014 at 2:16 PM
Edited Mar 11, 2014 at 2:16 PM
Hi Xela,

Interesting. I will check too.
Did you have test your scenarion on W8 or only on WP8 ?

The OutOfMemoryException occurs on the emulator or on the server side ?

Mar 11, 2014 at 8:39 PM
Edited Mar 11, 2014 at 11:57 PM
Hi Mimetis

Only Windows Phone 8, it occurs on the emulator. I assume it would be fine on Windows 8 as it has more memory, that is unless you use larger data then you should run into the same issue.

I check the memory usage after each change set and see it goes up around 20MB after each change set until it hits the 150MB total memory.

I have restructured the code to run off a while loop instead of recurring methods and it seems to be garbage collecting everything.

So far everything is running correctly.

Before change(After each change set):


After change:


Example code:
            CacheRequestResult results = null;
                CacheRequest request = new CacheRequest
                    Format = this.ControllerBehavior.SerializationFormat,
                    RequestType = CacheRequestType.DownloadChanges,
                    KnowledgeBlob = this.localProvider.GetServerBlob()
                results = await this.cacheRequestHandler.ProcessCacheRequestAsync(request, null, cancellationToken);
                if (results.ChangeSet != null)
                    statistics.TotalDownloads += (uint)results.ChangeSet.Data.Count;
                    await this.localProvider.SaveChangeSet(results.ChangeSet);
            while (!results.ChangeSet.IsLastBatch && !cancellationToken.IsCancellationRequested);
            return statistics;

Mar 12, 2014 at 2:49 PM
Edited Mar 12, 2014 at 3:16 PM
Hi Alex;

Thx for your solution

Im currently investigate the out of memory bug you have resolved with this code.
To be sure, what is you method you used to check the memory usage on your emulator ?

I currently try to reproduce the same behavior before make any correction
For now, I use this code to track the memory pressure during download :
Debug.WriteLine("App Cur Mem : "   Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage / 1024 / 1024);
Debug.WriteLine("App Mem Lim : "  Microsoft.Phone.Info.DeviceStatus.ApplicationMemoryUsageLimit / 1024 / 1024);
Debug.WriteLine("Device Mem  : "  Microsoft.Phone.Info.DeviceStatus.DeviceTotalMemory / 1024 / 1024);
Mar 13, 2014 at 1:01 AM
Edited Mar 13, 2014 at 1:24 AM

Yep I am using the same memory Property's as you to debug. The memory builds up when the ProcessCacheRequestResults method is being run but after all the recursion ends it frees it up.

In your ProcessCacheRequestResults method I just edited the following to show the memory usage with a messagebox to popup the result each change set:
else // It means its an Download response
                    Debug.Assert(e.ChangeSet != null, "Completion is not for a download request.");

                    // Increment the refresh stats
                    if (e.ChangeSet != null)
                        statistics.TotalDownloads += (uint)e.ChangeSet.Data.Count;

                        await this.localProvider.SaveChangeSet(e.ChangeSet);
/********************* Changes ******************/
                        string mem= "Current: " + (DeviceStatus.ApplicationCurrentMemoryUsage / 1000000).ToString() + "MB\n" +  "Memory Limit: " + (DeviceStatus.ApplicationMemoryUsageLimit / 1000000).ToString() + "MB";
/******************** End Changes *******************/

                        // Dont enqueue another request if its been cancelled
                        if (!cancellationToken.IsCancellationRequested)
                            if (!e.ChangeSet.IsLastBatch)
                                // Enqueue the next download
                                statistics = await this.EnqueueDownloadRequest(statistics, cancellationToken);
Mar 13, 2014 at 11:30 AM
Thx Alex

I ve found the bug in the recursion process beetween the ProcessCacheRequestResults method and the EnqueueDownloadRequest method (and by the way, the EnqueueUploadRequest method too)
I will publish the correction in the next release on codeplex and nuget

And I ve made a bunch of performance améliorations in the SQLite core implementation (you will see a significant performance amelioration in the insert batch part)

Mar 13, 2014 at 12:38 PM
Edited Mar 13, 2014 at 1:07 PM
Cool the new increased performance for the batch inserts will be very useful, previously it would take around 2hours to sync a empty sqlite database with 300k rows :)

Any estimate on when you will release the next update? ;)
Mar 13, 2014 at 2:16 PM
Just checkin the source code would be enough for you or you need the full nuget packages ?

i will check in the source code as soon as possible, maybe today or tomorrow

Mar 13, 2014 at 8:36 PM
Yeah the source check in is good enough for me :)

Thanks for contributing your free time to developing this tool kit.


Mar 20, 2014 at 11:36 AM
Hi Alex;

I ve made a big checkin.
You can grab the source or make an upgrade from the Nuget package.

So here are the corrections:
  • Correction of the memory leak in Batch mode
  • Correction on the batch size (was incorrect before. When you set 2048, it wasn't 2 mo, but 20 ...)
  • Significantly performance improvement (batch mode insert are improved from 10 minutes to 40 sec on my test machine)
  • Change the SQLite wrapper from SQLite-Winrt to SQLitePCL (could impact your dev part)
If you could test on your environment and make a feedback, it would be great :)

Mar 20, 2014 at 9:42 PM
Hey Mimetis,


Huge performance increase and no longer get the memory leak on large data in batch mode.

300,000 rows synced now takes only 4 minutes! Very happy with that speed increase :)


Mar 21, 2014 at 12:43 AM
From 2 hours to 4 minutes ...
That's .... good :)


Mar 26, 2014 at 10:39 PM
Hey Mimetis,

When using a cancellation token or locking the device cancels the sync, but then when I try to sync from the previous state it throws an error.

Does syncing from a previous canceled state work for you?


Apr 22, 2014 at 8:45 AM
Hi Alex,

I will check this issue as soon as possible !

Apr 23, 2014 at 10:07 AM
Hi Alex

Do you have any sample available for me to reproduce the bug ?
Or maybe do you know in which part of the code the Issue occurs ?

Apr 27, 2014 at 9:52 PM
Edited Apr 27, 2014 at 9:53 PM
I think the error has to do with the serverblob.

The error I recieve on WP8(Not tested on WindowsRT) is:

The input stream is not a valid binary format. The starting contents (in bytes) are: 00-00-00-05-00-00-00-00-00-00-00-01-00-00-00-00-00 ...
at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, IMethodCallMessage methodCallMessage)
at Microsoft.Synchronization.Services.SyncBlob.DeSerialize(Byte[] syncBlob)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob, Guid batchCode, Guid nextBatchSequenceNumber)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
at Microsoft.Synchronization.Services.DownloadChangesRequestProcessor.ProcessRequest(Request incomingRequest)
at Microsoft.Synchronization.Services.SyncService`1.ProcessRequestForMessage(Stream messageBody)

The server blob being sent is:


I think the error is from an invalid serverblob?
Apr 28, 2014 at 9:51 PM
Edited Apr 28, 2014 at 9:52 PM
Actually I just downloaded the latest version and don't seem to have this issue anymore :)
May 21, 2014 at 1:34 PM
I ve made a correction for this bug in the las version (

May 21, 2014 at 9:47 PM
Edited May 21, 2014 at 10:10 PM
Sweet, that last update was a false positive as I found it again but forgot to reply.

The new SyncProgressEvent is great too :D

Thanks for the fix.
May 22, 2014 at 8:49 AM
Yep :)
Dont have made any documentation about SyncProgressEvent, but I will write something .. later :)

By the way, updating only the service would be fine to correct the "Input Stream Not Valid ..." error
Unfortunately, if you have some customers blocked because of this error, You should have to make a full re - sync to restore the whole database (Mainly because the server files are already deleted)

May 22, 2014 at 9:54 PM
Ok, I'll try update my service as well and re-sync all the devices.

The SyncProgressEvent in my opinion was easy enough(for me) to implement into my project it is a great feature, may need documentation for others though.
I was actually going to put something like that in myself, but you done it for me ;)

Oct 17, 2014 at 8:28 PM

I am trying to use the lates version of the Sync Framweork toolkit in a windows 8.1 application with SQLite (SyncClient.SQLite.

When I try to sync a large database I always bet an Out of memory exception (with the property config.SetDownloadBatchSize(2* 2048);

The memory taken by IIS start to increase and then I get the exception.

I tried to look the source code to modify the code as discussed here but I cannot find the part to change.

the exception I am getting is :

Cannot enumerate changes at the RelationalSyncProvider for table 'SFM_IPOS_DISPENSER'. Check the inner exception for any store-specific errors.
at Microsoft.Synchronization.Data.DbSyncBatchProducer.DequeueBatch()
at Microsoft.Synchronization.Data.RelationalSyncProvider.ConsumeBatchFromProducer(DbSyncScopeMetadata scopeMetadata)
at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChanges(DbSyncScopeMetadata scopeMetadata, DbSyncSession DbSyncSession, UInt32 memoryBatchSize)
at Microsoft.Synchronization.Data.RelationalSyncProvider.GetChangeBatch(UInt32 batchSize, SyncKnowledge destinationKnowledge, Object& changeDataRetriever)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
at Microsoft.Synchronization.Services.DownloadChangesRequestProcessor.ProcessRequest(Request incomingRequest)
at Microsoft.Synchronization.Services.SyncService`1.ProcessRequestForMessage(Stream messageBody)

Exception of type 'System.OutOfMemoryException' was thrown.
at System.Object.MemberwiseClone()
at System.Array.Clone()
at Microsoft.Synchronization.SyncId.get_RawId()
at Microsoft.Synchronization.SyncIdFormat.ValidateArgument(SyncId id, String argumentName, String errorMessage)
at Microsoft.Synchronization.SyncKnowledge.Contains(SyncId replicaId, SyncId itemId, SyncVersion changeVersion)
at Microsoft.Synchronization.Data.RelationalSyncProvider.GetRowEnumerationState(DbDataReaderHandler readerHandler, SyncKnowledge knowledgeToCompare)
at Microsoft.Synchronization.Data.RelationalSyncProvider.EnumerateChangesInBatchesInternal(Object batchProducer)

Thanks in advance for the help :)