OutOfMemoryException from downloading in batch mode.

Mar 11 at 12:22 AM
Edited Mar 11 at 10:59 PM
Hey,

For some reason I am receiving a OutOfMemoryException - Insufficient memory to continue the execution of the program.

I am using the Windows Phone 8 emulator.

I am have batching enabled set to 2048KB.

I can sync small data fine using this method (5,000 rows) when I attempt to sync larger data:
300,000 rows I experience this error.

Cheers,
Alex
Mar 11 at 2:07 AM
Edited Mar 11 at 2:08 AM
I think the code for batching is implemented through recursion so the memory/list data builds up and is never disposed/garbage collected causing a OutOfMemoryException.

I will investigate, but I feel that is the issue...

Any feed back is welcome.
Coordinator
Mar 11 at 1:16 PM
Edited Mar 11 at 1:16 PM
Hi Xela,

Interesting. I will check too.
Did you have test your scenarion on W8 or only on WP8 ?

The OutOfMemoryException occurs on the emulator or on the server side ?

Sébastien
Mar 11 at 7:39 PM
Edited Mar 11 at 10:57 PM
Hi Mimetis

Only Windows Phone 8, it occurs on the emulator. I assume it would be fine on Windows 8 as it has more memory, that is unless you use larger data then you should run into the same issue.

I check the memory usage after each change set and see it goes up around 20MB after each change set until it hits the 150MB total memory.

I have restructured the code to run off a while loop instead of recurring methods and it seems to be garbage collecting everything.

So far everything is running correctly.

Before change(After each change set):

37MB
55MB
78MB
96MB
114MB
132MB

After change:

37MB
57MB
62MB
61MB
73MB
63MB
59MB

Example code:
            CacheRequestResult results = null;
            do
            {
                CacheRequest request = new CacheRequest
                {
                    Format = this.ControllerBehavior.SerializationFormat,
                    RequestType = CacheRequestType.DownloadChanges,
                    KnowledgeBlob = this.localProvider.GetServerBlob()
                };
                results = await this.cacheRequestHandler.ProcessCacheRequestAsync(request, null, cancellationToken);
                if (results.ChangeSet != null)
                {
                    statistics.TotalChangeSetsDownloaded++;
                    statistics.TotalDownloads += (uint)results.ChangeSet.Data.Count;
                    await this.localProvider.SaveChangeSet(results.ChangeSet);
                }
            }
            while (!results.ChangeSet.IsLastBatch && !cancellationToken.IsCancellationRequested);
            return statistics;
Cheers,

Alex
Coordinator
Mar 12 at 1:49 PM
Edited Mar 12 at 2:16 PM
Hi Alex;

Thx for your solution

Im currently investigate the out of memory bug you have resolved with this code.
To be sure, what is you method you used to check the memory usage on your emulator ?

I currently try to reproduce the same behavior before make any correction
For now, I use this code to track the memory pressure during download :
Debug.WriteLine("App Cur Mem : "   Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage / 1024 / 1024);
Debug.WriteLine("App Mem Lim : "  Microsoft.Phone.Info.DeviceStatus.ApplicationMemoryUsageLimit / 1024 / 1024);
Debug.WriteLine("Device Mem  : "  Microsoft.Phone.Info.DeviceStatus.DeviceTotalMemory / 1024 / 1024);
        
Sebastien
Mar 13 at 12:01 AM
Edited Mar 13 at 12:24 AM
Hey,

Yep I am using the same memory Property's as you to debug. The memory builds up when the ProcessCacheRequestResults method is being run but after all the recursion ends it frees it up.

In your ProcessCacheRequestResults method I just edited the following to show the memory usage with a messagebox to popup the result each change set:
else // It means its an Download response
                {
                    Debug.Assert(e.ChangeSet != null, "Completion is not for a download request.");

                    // Increment the refresh stats
                    if (e.ChangeSet != null)
                    {
                        statistics.TotalChangeSetsDownloaded++;
                        statistics.TotalDownloads += (uint)e.ChangeSet.Data.Count;

                        await this.localProvider.SaveChangeSet(e.ChangeSet);
                        
/********************* Changes ******************/
                        string mem= "Current: " + (DeviceStatus.ApplicationCurrentMemoryUsage / 1000000).ToString() + "MB\n" +  "Memory Limit: " + (DeviceStatus.ApplicationMemoryUsageLimit / 1000000).ToString() + "MB";
MessageBox.Show(mem);
/******************** End Changes *******************/

                        // Dont enqueue another request if its been cancelled
                        if (!cancellationToken.IsCancellationRequested)
                        {
                            if (!e.ChangeSet.IsLastBatch)
                            {
                                // Enqueue the next download
                                statistics = await this.EnqueueDownloadRequest(statistics, cancellationToken);
                            }
                        }
                        else
                        {
                            cancellationToken.ThrowIfCancellationRequested();
                        }
                    }
                }
          }
Coordinator
Mar 13 at 10:30 AM
Thx Alex

I ve found the bug in the recursion process beetween the ProcessCacheRequestResults method and the EnqueueDownloadRequest method (and by the way, the EnqueueUploadRequest method too)
I will publish the correction in the next release on codeplex and nuget

And I ve made a bunch of performance améliorations in the SQLite core implementation (you will see a significant performance amelioration in the insert batch part)

Sebastien
Mar 13 at 11:38 AM
Edited Mar 13 at 12:07 PM
Cool the new increased performance for the batch inserts will be very useful, previously it would take around 2hours to sync a empty sqlite database with 300k rows :)

Any estimate on when you will release the next update? ;)
Coordinator
Mar 13 at 1:16 PM
Just checkin the source code would be enough for you or you need the full nuget packages ?

i will check in the source code as soon as possible, maybe today or tomorrow

Seb
Mar 13 at 7:36 PM
Yeah the source check in is good enough for me :)

Thanks for contributing your free time to developing this tool kit.

Cheers,

Alex
Coordinator
Mar 20 at 10:36 AM
Hi Alex;

I ve made a big checkin.
You can grab the source or make an upgrade from the Nuget package.

So here are the corrections:
  • Correction of the memory leak in Batch mode
  • Correction on the batch size (was incorrect before. When you set 2048, it wasn't 2 mo, but 20 ...)
  • Significantly performance improvement (batch mode insert are improved from 10 minutes to 40 sec on my test machine)
  • Change the SQLite wrapper from SQLite-Winrt to SQLitePCL (could impact your dev part)
If you could test on your environment and make a feedback, it would be great :)

Sebastien
Mar 20 at 8:42 PM
Hey Mimetis,

Thanks!!

Huge performance increase and no longer get the memory leak on large data in batch mode.

300,000 rows synced now takes only 4 minutes! Very happy with that speed increase :)

Cheers,

Alex
Coordinator
Mar 20 at 11:43 PM
From 2 hours to 4 minutes ...
That's .... good :)

Cheers

Seb
Mar 26 at 9:39 PM
Hey Mimetis,

When using a cancellation token or locking the device cancels the sync, but then when I try to sync from the previous state it throws an error.

Does syncing from a previous canceled state work for you?

Cheers,

Alex.
Coordinator
Apr 22 at 7:45 AM
Hi Alex,

I will check this issue as soon as possible !

Sebastien
Coordinator
Apr 23 at 9:07 AM
Hi Alex

Do you have any sample available for me to reproduce the bug ?
Or maybe do you know in which part of the code the Issue occurs ?

Sebastien
Apr 27 at 8:52 PM
Edited Apr 27 at 8:53 PM
I think the error has to do with the serverblob.

The error I recieve on WP8(Not tested on WindowsRT) is:

ErrorDescription=System.Runtime.Serialization.SerializationException
The input stream is not a valid binary format. The starting contents (in bytes) are: 00-00-00-05-00-00-00-00-00-00-00-01-00-00-00-00-00 ...
at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, IMethodCallMessage methodCallMessage)
at Microsoft.Synchronization.Services.SyncBlob.DeSerialize(Byte[] syncBlob)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob, Guid batchCode, Guid nextBatchSequenceNumber)
at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
at Microsoft.Synchronization.Services.DownloadChangesRequestProcessor.ProcessRequest(Request incomingRequest)
at Microsoft.Synchronization.Services.SyncService`1.ProcessRequestForMessage(Stream messageBody)

The server blob being sent is:

serverBlob=AAEAAAD/////AQAAAAAAAAAMAgAAAFlNaWNyb3NvZnQuU3luY2hyb25pemF0aW9uLlNlcnZpY2VzLCBWZXJzaW9uPTEuMC4wLjAsIEN1bHR1cmU9bmV1dHJhbCwgUHVibGljS2V5VG9rZW49bnVsbAUBAAAAK01pY3Jvc29mdC5TeW5jaHJvbml6YXRpb24uU2VydmljZXMuU3luY0Jsb2IFAAAAIDxDbGllbnRLbm93bGVkZ2U+a19fQmFja2luZ0ZpZWxkIDxDbGllbnRTY29wZU5hbWU+a19fQmFja2luZ0ZpZWxkHDxJc0xhc3RCYXRjaD5rX19CYWNraW5nRmllbGQaPEJhdGNoQ29kZT5rX19CYWNraW5nRmllbGQaPE5leHRCYXRjaD5rX19CYWNraW5nRmllbGQHAQADAwIBC1N5c3RlbS5HdWlkC1N5c3RlbS5HdWlkAgAAAAkDAAAABgQAAAAkMzllMDg0YTktMDBjYi00MGViLWJjYTYtZWY4ZDc5NDc5ZjM1AAT7////C1N5c3RlbS5HdWlkCwAAAAJfYQJfYgJfYwJfZAJfZQJfZgJfZwJfaAJfaQJfagJfawAAAAAAAAAAAAAACAcHAgICAgICAgJFjlA/sEKGTKSRAzwUPemEAfr////7////UgEAdJmr4kKgqx/Reoj7DQ8DAAAA0QAAAAIAAAAFAAAAAAAAAAEAAAAAAAAABQAAEAAAAAI54ISpAMtA67ym7415R581wdn2nid1QDGVQDVd0BvOjQAAABgAABABKAIAAAEAAAAVAAAAAgAAAAEAAAAAAAAAAQAAAAEAAAABAAAAAAA8RvYAAAAXAAAAAQAAABYAAAACAAMAAAAAAQA9AGMAbwBuAG4AZQBjAHQAaQB2AGkAdAB5AF8AZQB4AHQAcgBhAGMAdABfAGcAZQBvAC0AMQA5ADkAMQAAAAAAAAAAAAAAABkBAAAAAAs=

I think the error is from an invalid serverblob?
Apr 28 at 8:51 PM
Edited Apr 28 at 8:52 PM
Actually I just downloaded the latest version and don't seem to have this issue anymore :)
Coordinator
May 21 at 12:34 PM
I ve made a correction for this bug in the las version (3.8.4.3)

Sebastien
May 21 at 8:47 PM
Edited May 21 at 9:10 PM
Sweet, that last update was a false positive as I found it again but forgot to reply.

The new SyncProgressEvent is great too :D

Thanks for the fix.
Coordinator
May 22 at 7:49 AM
Yep :)
Dont have made any documentation about SyncProgressEvent, but I will write something .. later :)

By the way, updating only the service would be fine to correct the "Input Stream Not Valid ..." error
Unfortunately, if you have some customers blocked because of this error, You should have to make a full re - sync to restore the whole database (Mainly because the server files are already deleted)

Sebastien
May 22 at 8:54 PM
Ok, I'll try update my service as well and re-sync all the devices.

The SyncProgressEvent in my opinion was easy enough(for me) to implement into my project it is a great feature, may need documentation for others though.
I was actually going to put something like that in myself, but you done it for me ;)

Alex