That’s why for us at Resco, the synchronization functionality and its adoption have always been a priority, as it enables mobile workers and office teams to work better together.

The recent update of Resco solutions takes this to the next level again.

The new JSON storage of questionnaires brings a lot of possibilities for improving synchronization’s speed and efficiency.

That’s why we have put together the most up-to-date information and a short guide so you can make the best out of it.

In this article you will learn about:

• What were the previous storage options and how they worked
• How the new JSON stores data
• Comparison of storage space used by the new and legacy JSON formats
• What are the recommended actions for using JSON
• How to create questionnaires with the correct setup

The legacy storage options

Record-based storage

The first option was record-based, where each newly completed questionnaire included its own questionnaire record, question records, and question group records.

Over time, this could cause a lot of records to be stored in the database, and could lead to a long synchronization time and even server timeouts.

That could lead to potential integrity problems, such as incomplete questionnaire. This would happen when the synchronization was interrupted while only a part of the question records were transferred.

Template dependency

This option creates a distinction between the core and template-dependent fields.

Core fields cannot be duplicated for other questionnaires and posses unique information as ‘value’, ‘score’, and ‘answeredon’ fields.

Template-dependent fields are ones that can be used for multiple questionnaires. That means the data doesn’t need to be stored for each completed questionnaire, which saves a lot of space.

Such fields are ‘resco_styles’, ‘resco_localization’, ‘resco_rules’ and ‘resco_autoreport’, that usually tend to take the most storage.

This method improved upon the records-based storage and decreased the amount of space needed, but also makes the questionnaire not complete without the template, and cannot run properly without it.

Note that template dependency comes hand in hand with template versioning, which addresses the problem with template changes.

Legacy JSON

With this storage method, we serialized all the fields presented in the ‘question’ and ‘question_group’ records into a one JSON string.

The data string was stored in ‘resco_serializedanswers’ when a new questionnaire was created. This method eliminated the need for question and ‘question_group’ records.

The origin JSON improved the storage significantly, but it was still storing all the fields, some of which were not necessary (if the template dependency was not turned on). The format used standard .NET serialization and was not optimal as far as storage was concerned

Recommended actions for the older storage methods

Currently, you can use all of the mentioned methods to store questionnaires, but the status will change with the update in December, 2023, where we will be deprecating legacy JSON and record-based storage.

Recommended actions regarding the legacy storage formats are:

• If you don’t have any integration with third-party tools that parse JSON data, we recommend switching to the new JSON right away to start benefiting from a reduced database footprint and shorter sync times
• • If you do have an integration with third-party tools that work with questionnaire answers in legacy JSON format, check it’s working with the latest version and adapt your processes during the transition period
• In either case, you have the option to convert your existing questionnaire data to the new JSON format with Resco developed tool called Questionnaire Converter. To learn more technical details, please, contact Resco support

How does the new JSON storage option work

The newest storage option is made to be modular by giving you the option of selecting which information should be stored and which shouldn’t. It also includes versioning and dependency settings that are mandatory for this method.

There are two options available, Minimal JSON and Flexible JSON:

Minimal JSON

This option stores the least amount of information, just key-value pairs, where the key is question name and the value is its answer.

Also, questions containing a default value can be excluded from the storage, as the answer is known and can be reconstructed.

With this method you can achieve the most efficient storage possible yet.

Let’s see some example where we have a question on the one side and answer on the other. Firstly, we have a questionnaire form filled:

This is how it will look in the JSON string format (note that the actual string is stored as just one line):

As you can see, there is only essential information about the questions, groups, and answers.

Flexible JSON

Using the Flexible JSON method, you can customize the format by choosing which information is stored.

It also improves on the problems of the original JSON as it omits almost all of the information that can be reconstructed from the template.

Let’s see some example using the same questionnaire data from the Minimal JSON:

As you can see, the amount of information here almost doubled from the minimal JSON, and it also includes data such as score of the question answer, lookup, and option set labels.

Size comparison of different storage methods

Firstly, we need to look at the impact of the template dependency for storage-based method. As you can see from the graph below, it decreases the storage required to about 8% of the original one.

In this example, there are 2300 questionnaires answered with 21 question groups and 183 questions for each one.

Next, we can have a look at Minimal JSON. Looking at the same scenario and given that all of the questions are answered, it looks something like this:

The Minimal JSON improves the storage 6 times, but a more realistic scenario, where not all of the questions are answered, would be around 10 times.

In the next graph, we can see the difference between Flexible JSON and Record-based method. The scenario is the same as with the previous example:

With Flexible JSON there are more fields present in the string, but the storage efficiency is still about 5 times better.

Here you can see all four scenarios together:

How to create a questionnaire with answers stored in JSON format

Now we can have a look at how you can create your questionnaire with these settings enabled to help you with decreasing your storage requirement.

Currently, when you create a questionnaire, the settings are set to default for the new JSON method with versioning and dependency enabled (and these settings are required for the method).

We recommend keeping these settings on, because as mentioned previously, we are planning to deprecate the other legacy methods by the end of 2023.

When you open your questionnaire and set up your question and question groups, you can have a look at the JSON string in the preview mode:

With this setup, you have a handy overview of how the storage file is going to look like and what information is included.

How can you further optimize Minimal and Flexible JSON?

It’s important to note that with both Minimal and Flexible JSON, length of the questions names has a significant impact on storage.

It’s especially valuable to consider this if you are managing a large number of questionnaires.

For example, imagine a questionnaire with 183 questions where a single question contains an average of 18 characters.

Now, let’s say we would be able to optimize the length to just 1 character per question, which would save over 3K characters per questionnaire.

If you would be able to shorten the names across 2300 questionnaires per day, you could save approximately 150MB of storage per month.

You can achieve this while designing your questionnaire for each of the questions:

What’s next?

With the introduction of the new JSON storage option, you can choose to store only the information you need and reduce the size of your database, resulting in shorter synchronization times.

And with the deprecation of older storage methods in December 2023, it’s important to make the switch as soon as possible. Check out Resco’s Wiki for more information on the Resco Inspections data model and JSON storage for questionnaires.

If you would like to learn more about how to optimize sync speed for Inspections, take a look at synchronization training provided by Resco Support Team Lead Timotej Leško.

Príspevok How to speed up questionnaire sync and optimize storage use with the new JSON (+Comparison with the legacy formats) zobrazený najskôr Resco.

With access to real-time insights and metrics, companies and their field workers can make informed decisions quickly, allowing them to respond faster than ever before.

But synchronization can be the make-or-break technology for your whole digital initiative.

With megabytes or even gigabytes of data, it can be tricky to set up and it’s a challenge for situations when moving between online & offline locations.

Resco Mobile App Development Toolkit offer several tools helping you to maximize the efficiency of synchronization.

Let’s take a look at the most important ones and their specifics.

Tools for analyzing

Sync Log

The Sync log is a powerful tool. You can use it to analyze the time spent on each synchronization step and adjust it based on the requirements.

For example, you notice that the sync is taking too long and find out that one entity is downloaded without restrictions, even when the user doesn’t need all the data.

The most common way of figuring out the cause of any issues is to look into the log from the device. You can find it in the “About“ section of the app.

The option “Diagnostic Sync Logs” must be enabled in the Woodford project, under “Settings” in “Configuration”, or as “Log Sync Details” in the Mobile CRM app’s Setup.

We recommend using this option only if you’re in the process of fine tuning. Enable this to ensure that your log includes as much detail as possible. However, in general, don’t leave this option enabled in production.

After publishing the project and synchronizing with the app, you will see the changes.

Scroll down to the latest synchronization (which you can find thanks to the date and time in its name), where you will see what entities have been synchronized, how many records are there for each entity, and other processes which happened during synchronization.

You can find detailed information about the structure of our log files and an in-depth analysis on our wiki: Log files – Resco’s Wiki

Sync Dashboard

The Sync Dashboard is our monitoring tool for administrators that collects information about every synchronization. It presents them in an understandable way with charts and the ability to see each synchronization log individually.

It also shows when there are any errors or warnings present. The administrator can see what exactly caused them by clicking the icon and solve the issue at hand.

You can find more information about how to set up the Sync Dashboard in your environment and how to use it on our wiki: Sync Dashboard – Resco’s Wiki

Storage Analyzer

This tool helps with deeper analysis by showing statistical data about the app’s local storage.

You can find it in the setup of your app:

You can see multiple reports focusing on database storage and binary documents (blob storage).

Here we are providing detailed information about what each component takes into storage for you to look at and adjust as necessary.

You can find detailed explanations for each of the Storage Analyzer reports our wiki: Storage analyzer – Resco’s Wiki

Audit

You can do two types of auditing: server auditing and mobile auditing.

Server Auditing

It can be quite helpful to see what changes were made to records or entities to help determine additional steps to improve synchronization, and when tracking other issues.

You can enable auditing on your back-end server.

For more information, see our wiki article: Auditing – Resco’s Wiki

Mobile Auditing

This type of auditing shows changes in the app itself. Here you can find out how often users create records and what changes they make.

Auditing – Resco’s Wiki

That creates individual records for each change you want to track, so we suggest using maintenance in Woodford to prevent unnecessary cluttering.

Maintenance – Resco’s Wiki

Fiddler

This is an external free tool that we regularly use when trying to analyze the traffic between the server and the app. You can use it too for our Windows (store and desktop) apps.

It helps with figuring out what data was synchronized with the server and it’s also handy for finding bottlenecks.

You can find more information on how to set up Fiddler on our wiki: Fiddler – Resco’s Wiki

For a brief introduction to Fiddler Classic including some practical tips and tricks, enroll in the Fiddler Classic course in Resco Academy.

Tools for configuration

Sync filter

When you want to limit what data is being shown in the application, Sync filter is just the tool for that.

With it, you can set each entity to show only the records relevant to a user and make the synchronization process as quick and smooth as possible.

When selecting most entities in Woodford, you will find that they have a sync filter.

You can set up your conditions for the filter, and after a full synchronization (deleting data in the app’s setup tab and synchronizing), you will see the changes in the app.

It is also possible to edit the filter directly by exporting it as an XML file and adjusting it yourself.

You can find more information about the Sync Filter on our wiki: Sync Filter – Resco’s Wiki

Also, we have plenty of examples on how to utilize the sync filter here: Sync Filter examples – Resco’s Wiki

Advanced Sync Setup

When you want to go into more in-depth synchronization, you can set up the behavior of the Sync Downloader and configure how the records are downloaded.

This is an XML configuration file that can be found in the Woodford project under “Settings” inside “Configuration” on the top of the panel named „Sync Config“.

You can modify the download process to your liking based on many parameters.

For an exact and detailed explanation of how to work with Advanced sync setup, visit our wiki page: Advanced sync setup – Resco’s Wiki

Blob Storage

In Resco Cloud, you can you can use external storage for binary documents to save space in your database (such as Azure Storage or AmazonS3).

Blob storage takes this into account and downloads the documents from this storage during synchronization.

For more information on how to set it up in your environment, look into our wiki article: Blob storage for Resco Cloud – Resco’s Wiki

Document Filter

For when you don’t want to download all of your files into your app, but want to have them present on-demand (with internet connection) you can use this blob filter.

It can be found in the Woodford project under “Setting” inside “Configuration” on top of the „Blob Filter“ panel.

You can set varying max sizes for different file formats based on specific entities and much more.

For examples and more information about this tool, go to our wiki page: Document filters – Resco’s Wiki

Woodford Config Setup

Last but certainly not least is the Woodford configuration setup. It contains an amazing number of specific configurations for setting up your environment to your needs.

This is present in every Woodford Project under “Settings” on the left panel.

To pick a few specific configuration settings that are worth mentioning here:

• Max Upload Attachment Size – The maximum size of an attachment (documents, notes, email attachments) in bytes that can be created, captured or attached in the app.
• Background Download – Always download records in the background. If True, foreground sync switches to background after executing the upload & customization update. No impact on background sync
• Auto Sync – Set up an automatic sync
• Max Sync Records – Max number of records per entity to download. This is a hard limit that violates data integrity (a more restrictive sync filter is a better way to reduce app data)
• Show Sync Warnings – Show sync warnings alert after sync completion or just silently write them to the sync log (in case of multiple warnings, only the first one is shown)
• # Records/Upload Request – Max number of upload requests sent in one batch to the server

For more information about each setting, see our wiki article explaining the whole Configuration tab: Configuration – Resco’s Wiki

How to start

Fortunately, Resco solutions provide a range of tools that help optimize and manage data synchronization.

But if we can recommend what should be the first step in making your sync better, it’s enrolling to Resco Academy courses.

They provide an in-depth look at data synchronization and insights on how to maximize efficiency when using different tools.

These courses will be a great place to start:

• Introduction to Resco Architecture
• Woodford Basics
• Synchronization & Sync Dashboard
• Logs and troubleshooting

If you are ready to enroll in one of these courses, you can register to Resco Academy for free.

Príspevok 10 tools to improve your synchronization speed and management zobrazený najskôr Resco.