Strategies for transferring mongoDB data between environments

This document describes the strategies that can be used when cloning Sage X3 environments or transferring elements from one environment to another. For relational database transfers associated with a Sage X3 endpoint, a copy tool already exists. But for administration data stored in mongoDB, the existing tools are not relevant, and other strategies are required. These strategies are described here.

This page is not supposed to describe all the possibilities for a mongoDB, but just to give some basic information about the most frequent operations. A complete mongoDB reference manual is available on mongoDB site.

Complete transfer

In this first scenario, a complete staging environment has to be transferred to the customer environment.
The correct tools to use are the mongoDB tools mongodump and mongorestore.

These tools are available in the 'bin' subdirectory of mongoDB installation path, and their syntax is rather simple:

Data extraction

The extraction of data uses the mongodump tool. The mongoDB database must of course be running to launch this command. The syntax (from a command line) is the following:

mongodump --db syracuse --out directory

In Sage X3 context, the first argument --db syracuse corresponds to the name of the mongoDB database and directory is the path of a directory that is created to store the extracted data. In this directory, a subdirectory syracuse(the name of the database) is created.

In this directory, for every collection (i.e., administrative entity called <entity>), two files are created, called:

  • <entity>.metadata.json
  • <entity>.bson

The first file contains metadata (index definition), the second contains binary JSON data.

Every entity that has to be imported on another mongoDB instance uses data and metadata from these two files.

Data integration

The integration of data uses the mongorestore tool. The mongoDB database must of course be running to launch this command. The syntax (from a command line) is the following:

mongorestore --db syracuse --drop directory

The --drop option is not mandatory; it allows you to delete a collection before integrating it.

All the collections present as 'json' and 'bson' files in the directory are integrated.

It is important not to transfer the host collection. This technical collection is automatically created and the content includes UUIDs that make it not transportable.

Transfer of some entities only

All entities cannot be extracted and integrated. Some entities refer to other entities where all references are based on UUIDs, although these references appear with their name in the administrative interface. Therefore, the collection cannot be transferred separately, but as consistent groups of information.

Two tools are available to integrate consistent administration data: