Synchronization - reference catalog

useReferenceSync embeds offline a reference dataset that the application browses without modifying: products, categories, third parties, contacts, along with their images and PDF documents.

It complements useSyncClient, which is transactional and works in its own database. Both often coexist in the same application: the catalog comes down through useReferenceSync, business writes go up through useSyncClient or through a business action queue.

How it differs from useSyncClient

The decisive advantage of the latter is that your screens keep querying your own tables, with your indexes and your queries. Nothing changes in the read code when a list goes offline.

Getting started

import { useReferenceSync } from '@cap-rel/smartcommon';
import { useMyModuleDb } from 'src/db';
import { APP_VERSION } from 'src/utils/constants/vite';

const ENTITIES = [
    { objectType: 'product', store: 'products' },
    { objectType: 'category', store: 'categories', cleanOrphans: true }
];

const DOCUMENTS = [
    {
        objectType: 'product',
        store: 'productDocuments',
        fk: 'product_id',
        doctypes: ['image', 'pdf']
    }
];

export const useCatalogSync = () => {
    const db = useMyModuleDb();

    return useReferenceSync({
        db,
        appVersion: APP_VERSION,
        entities: ENTITIES,
        documents: DOCUMENTS,
        metaStore: 'syncMeta'
    });
};

The metadata store must exist in your Dexie schema, as a plain key / value store:

const db = useDb({
    name: 'myModule',
    version: 1,
    stores: {
        products: 'id, ref, label',
        categories: 'id, label',
        productDocuments: '++local_id, product_id, server_id, type',
        syncMeta: 'key'
    }
});

Options

entities

{ objectType: 'product', store: 'products', mapper: mapProduct, cleanOrphans: false }

Each type is pulled in pages of 500 through GET sync/pull, with its own delta marker stored under the lastSyncAt_<objectType> key. That marker is written only once every page has gone through: an interruption mid-way makes the whole pass start over rather than leaving a gap.

cleanOrphans costs a full pull on every synchronization. Only enable it on small reference datasets, and only if the server does not already publish its exclusions in the delete list.

documents

{
    objectType: 'product',
    store: 'productDocuments',
    fk: 'product_id',
    doctypes: (prefs) => [prefs.syncImages && 'image', prefs.syncPdfs && 'pdf'].filter(Boolean),
    enabled: (prefs) => prefs.syncProductDocuments
}

Beware of one trap: a module may register its own syncable types server-side, for instance capfullpos_product to filter the sellable catalog. Those types apply to entities, but the document controller only knows the native types. The two lists do not copy each other.

Blobs are fetched as ZIP bundles rather than one by one, falling back to individual downloads for oversized files. This is what avoids flooding the server with one request per thumbnail when a session opens.

Row written locally:

{
    local_id,            // auto-incremented
    [fk]: object_id,     // product_id, category_id...
    server_id, type, filename, relative_path, mime_type,
    blob, size, synced_at, server_updated_at
}

The thumb doctype, when the server allows it, returns only the thumbnail instead of the full-resolution original. On a grid of tiles, the difference in embedded weight is considerable.

dataFeeds

For dictionaries and configuration blocks, which do not go through the sync engine but through a plain GET:

{ key: 'paymentModes', endpoint: 'syncdata/payment-modes', store: 'paymentModes', clearBefore: true }

Returned values

How a pass runs

1. Client registration (idempotent): POST sync/register
2. Entities, in declaration order: paginated GET sync/pull
3. Documents: metadata, local comparison, ZIP bundle download
4. Data feeds: one GET per feed
5. Writing the lastSyncAt marker

An error on one step is recorded in the result and does not interrupt the following ones. Two exceptions stop everything immediately:

• a cancellation, triggered by resetSync during a pass;
• a 403, which raises a ForbiddenSyncError. The stop is immediate and deliberate: insisting on a series of rejected requests gets the application blacklisted by application firewalls.

A pass also refuses to start while offline, and refuses to overlap a pass already in progress.

Showing progress

const CatalogSyncOverlay = () => {
    const { isSyncing, syncProgress, lastSyncAt, error, syncNow } = useCatalogSync();

    if (!isSyncing) {
        return <button onClick={syncNow}>Update the catalog</button>;
    }

    return (
        <div>
            <p>{syncProgress?.step ?? 'Preparing'}</p>
            {syncProgress?.total > 0 && (
                <progress value={syncProgress.current} max={syncProgress.total} />
            )}
        </div>
    );
};

syncProgress goes back to null at the end of a pass. On entities, current and total both hold the number of items written so far: progress is a rising volume, not a percentage, because the total is unknown until the last page.

Trap: reading during a first synchronization

The lastSyncAt_<objectType> markers are only written after a complete success. A screen that checks for the marker to decide whether it may read locally will therefore show an empty list for the whole first pass, even though rows are landing as they come. Test the content of the store, not the marker.

See also

• Offline synchronization - choosing a mechanism
• Data storage - useDb and Dexie stores
• API requests - useApi
• Mapping Dolibarr - React - declaring a syncable object