Sync a big database takes up all my memory machine

Oct 20, 2014 at 6:18 PM
Hello,

I am trying to use Sync Framework toolkit in our project in a windows 8.1 app with Sqlite (client side) and a service deployed in Azure.

I have use Frabrickam Universal as example and I am able to sync the only problem is our DB is quite big and ideally need to be synched fully to the client.

When I try to sync Isee the memory in IIS increasing constantly untill does not reach the limit and windows closes all apps to free resources.

Sometimes (usually when I am close to the limit) I get OutOfMemoryException,

I have tried everything is in the forum (including using the Batch Download) but I cannot keep the memory low.

I want to add I am using Azure Emulator in my machine but this happens even using Frabrikam code if I point my DB.

Please any help would be really appreciated.

Regards

Nic
Coordinator
Oct 21, 2014 at 9:52 AM
First of all, you have to use the Bacth size mode to "fragment" your the traffic during the download to device operation.
Maybe something like 2048 (approximatively 2mo) should be correct.
be careful that the max size of each row form your server may be under this limit (2 mo)

By the way, the upload process from device to server doesn't have any batch mode. It's a one operation with all datas to upload.
I assume that the rows from the device shouldn't be to hight (in size) so the batch mode doesn't work in this way.

Can you share with me your server code, and maybe a backup of your database to let me do some tests ?

Sebastien
Oct 21, 2014 at 10:16 AM
Hi Mimetis,


Thanks very much for your answer.


So yes I am using the bacth size mode to fragment the db (I have set it to be something like 20 MB because I have read somewhere a guy having the same problem as me fixed it putting a big Batch mode) but IIS basically starts to fill all the memory up and Windows 8 starts closing applications and kill the process.


I cannot share the db sadly without the asking the client but I was able to sinc one table with 47062 but this is one of 52 tables I have to sync and the data in all other tables are almost the same size ☹


I do not have particular requirements in timing (it can get even 40 60 mins the first sync it is not a problem) the problem is IIS fills the memory and sometimes I get outOfMemoryException.


My server is configured like this

config.ServerConnectionString = @"Data Source=.\SQLEXPRESS;Initial Catalog=xxxxxxUat-2014-10-9-16-13;Integrated Security=true;";

config.SetEnableScope("DefaultScope");

// 

// 

// TODO: Optional.

// config.SetBatchSpoolDirectory(@"E:\SpoolDir");

config.SetDefaultSyncSerializationFormat(Microsoft.Synchronization.Services.SyncSerializationFormat.ODataJson);

config.SetConflictResolutionPolicy(Microsoft.Synchronization.Services.ConflictResolutionPolicy.ServerWins);

config.SetDownloadBatchSize(250*1024);

 

config.UseVerboseErrors = true;

config.EnableDiagnosticPage = true;

I have also seen in the code that the SetDownloadBatchSize is divided by 10 so if I put 2*1024 actually my batch size is not 2 MB….. Is this correct?



I do not understand why IIS keep so much memory and never releases it ☹


Thanks
Oct 21, 2014 at 11:43 AM
I have run another test with settings 2*1024

After reaching 6GB of memory used by IIS I got this exception:
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>/LM/W3SVC/1273337584/ROOT-1-130583644844113468</AppDomain><Exception><ExceptionType>System.OutOfMemoryException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Exception of type 'System.OutOfMemoryException' was thrown.</Message><StackTrace>   at System.Data.DataTable.NewRowFromBuilder(DataRowBuilder builder)
   at System.Data.DataTable.NewRow()
   at Microsoft.Synchronization.Data.RowSorter.AddSortedRowToDataTable(DataTable curDataTable, SortedRow sortedRow)
   at Microsoft.Synchronization.Data.RowSorter.&amp;lt;PullSortedBatches&amp;gt;d__0.MoveNext()
   at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.SaveBatchesAndReturnFirstBatch(RowSorter rowSorter)
   at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
   at Microsoft.Synchronization.Services.DownloadChangesRequestProcessor.ProcessRequest(Request incomingRequest)
   at Microsoft.Synchronization.Services.SyncService`1.ProcessRequestForMessage(Stream messageBody)
   at SyncInvokeProcessRequestForMessage(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp;amp; outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
   at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
   at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.CompleteParseAndEnqueue(IAsyncResult result)
   at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.HandleParseIncomingMessage(IAsyncResult result)
   at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)
   at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)
   at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.ProcessHttpContextAsync()
   at System.ServiceModel.Channels.HttpChannelListener`1.BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)
   at System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
   at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)</StackTrace><ExceptionString>System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Data.DataTable.NewRowFromBuilder(DataRowBuilder builder)
   at System.Data.DataTable.NewRow()
   at Microsoft.Synchronization.Data.RowSorter.AddSortedRowToDataTable(DataTable curDataTable, SortedRow sortedRow)
   at Microsoft.Synchronization.Data.RowSorter.&amp;lt;PullSortedBatches&amp;gt;d__0.MoveNext()
   at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.SaveBatchesAndReturnFirstBatch(RowSorter rowSorter)
   at Microsoft.Synchronization.Services.SqlProvider.SqlSyncProviderService.GetChanges(Byte[] serverBlob)
   at Microsoft.Synchronization.Services.DownloadChangesRequestProcessor.ProcessRequest(Request incomingRequest)
   at Microsoft.Synchronization.Services.SyncService`1.ProcessRequestForMessage(Stream messageBody)
   at SyncInvokeProcessRequestForMessage(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp;amp; outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp;amp; rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
   at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
   at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
   at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
   at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.CompleteParseAndEnqueue(IAsyncResult result)
   at System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.HandleParseIncomingMessage(IAsyncResult result)
   at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)
   at System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginProcessInboundRequest(ReplyChannelAcceptor replyChannelAcceptor, Action dequeuedCallback, AsyncCallback callback, Object state)
   at System.ServiceModel.Channels.HttpChannelListener`1.HttpContextReceivedAsyncResult`1.ProcessHttpContextAsync()
   at System.ServiceModel.Channels.HttpChannelListener`1.BeginHttpContextReceived(HttpRequestContext context, Action acceptorCallback, AsyncCallback callback, Object state)
   at System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
   at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
   at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)</ExceptionString></Exception></TraceRecord>
If I bring the Diaf console it shows this:

Diagnostics
Diagnostic Test Result Comments
Sync Framework Runtime PASSED None
Connection to SQL Server PASSED None
Database Provisioning PASSED None
Batching Folder Exists? PASSED None
Write access to batching folder PASSED None
ClientAccessPolicy.xml/CrossDomain.xml files None


Service Configuration
Setting Configured Value
Scope Name defaultscope
Default Conflict Resolution ServerWins
Default Serialization Format ODataJson
Verbose Error Response true
Batching Directory C:\Users\nic\AppData\Local\dftmp\Resources\78f258c7-c68c-47c6-bd00-94191945d3c7\temp\temp\RoleTemp
Download Batch Size 204

As you can see the batch size is 204 KB not 2 MB...

Am I using a worng ersion? I have downloaded it from Git for version 3.8.6.0...

I have asked if it is possible provide you a DB as well.

I really appreciate your help thanks alot

Nic
Oct 22, 2014 at 8:50 PM
Hello I am still trying to find a solution to my problem and today I made some progress (after one week of desperation).


I have modified my DB in order to introduce filtering and the memory now is getting released (I do not knw why it does not when I try to full sync the db though but for my app is good even filtering).

When I have provisioned all my table ( I made a couple of tests not with all table but only with some at the beginning) the synching works and the memory does not rise more than 7000 MB but after a bit I got asynchronization exception regarding a not valid Blob (a blob cannot start with 000000000000000000000-000000000001........) exception and I am not able to resume the sync any more. Another option for me would be split the sync in 2 or more parts. I was thinking to create lets say 2 Template scopes and sync first with one and then with the second.

I would like to avoid this because as far I have seen this procedure would generate 2 SQLite db in the client and then I would have to maintain 2 db with all the problems that this could cause

Another problem of this is that in my db a client can have many retailers ond for each retailers there are others informations related to them in other tables so I would lneed to take with the first sync all the retailers for a user and then for each retailers get all the related info...

Does anyone knows how it is possible to fix the blob exception I am facing? What class in the toolkit generates this? I have seen in another post someone else was facing the same problem but it was written that the bug has been fixed so I am not sure why I am getting it:(

Thanks again for any help

Nic