Data Access → Drivers → Memory

The Memory native data access driver allows applications to manipulate spatial data (both vector and raster) in main memory. Besides the conteiner classes there are also many utility classes and functions for helping load data from other drivers and to manipulate them in main memory as well. This is a special driver built as a module (not as a plugin) and embedded by several driver implementations.

This section describes the design and implementation issues of this driver and how it can be used by application developers.

The In-Memory driver implements some of the data access foundation classes as showned in the following class diagram.

The above diagram means that you can load a shapefile to main memory by just doing the same operations you do when loading it to a PostGIS backend.

The following code snippet shows how to get an in-memory data source:

te::da::DataSource* ds = te::da::DataSourceFactory::open("MEM", "");

This driver has two operation modes:

• shared mode: each dataset managed by the data source may be shared by multiples instances of datasets retrieved from the transactor interface.
• non-shared mode: each dataset managed by the data source is cloned by the data source transactor when a client ask for a dataset or make a query.

These modes are set during connection time by specifying some parameters (see next section).

The te::mem::DataSource class is capable of managing in-memory datasets. Each dataset must have an unique name. Only in-memory datasets can be managed by an in-memory data source.

The main capabilities of a In-Memory data source are:

• ACCESS_POLICY: read and write.
• TRANSACTION: doesn't supported.
• DATASET_PERSISTENCE: supported.
• <color red>TODO</color>

• MAX_DATASETS: can be used to set the maximum number of datasets to be handled by the data source. If not informed the dafault used value is 1024. (Optional)
• OPERATION_MODE: if this parameter is set to NON-SHARED each time a dataset is retrieved from the transactor a new clone is created otherwise a shared copy of the original dataset is used. The default (even if omitted) is SHARED. (Optional)

The In-Memory driver also implements all the raster foundation classes as showned in the following class diagram.

The above implementation allows one to manipulate raster data in main memory. For example, to read a raster stored in a GeoTIFF file and load it all to the memory we can do the following:

std::map<std::string, std::string> rinfo;
 
rinfo["MEM_SRC_RASTER_DRIVER_TYPE"] = "GDAL";
rinfo["SOURCE"] = "/data/geotiff/mosaico_brasil.tiff";
 
te::rst::Raster* inraster = te::rst::RasterFactory::make("MEM", rinfo);

The internal in-memory raster implementation can be built over a single and contiguous memory-block or it can be split over several data blocks organized in tiles (or blocks). This will depend on the client choices during the in-memory raster creation (see next section). One benefit of spliting the raster in tiles is that it allows the manipultaion of raster with high dimensions in a fragmented memory computer. Another potential benefit by spliting the raster in tiles is to make a better use of main CPU cache during raster traversal (avoiding cache miss).

The open method in the RasterFactory class works slightly different from the other drivers. It will use the informed parameters to read the data from an input raster or an input data buffer. The input raster may be in the system file (like a regular GeoTiff) or it can be retrieved by any raster driver (like a PostGIS raster). In this case, you must provide enough information in the raster info parameter so that the memory implementation can access the source raster. The In-Memory raster driver accepts the following additional parameters:

• MEM_SRC_RASTER_DRIVER_TYPE: the identifier of the driver used for reading the input raster. If ommited the default raster driver is used. (Optional)
• MEM_TILED_RASTER: if this parameter is true the memory raster will use a tile (block) strategy. This means that internally the whole raster is split into different data blocks (called tiles). (Optional)
• MEM_RASTER_TILE_WIDTH: the width (in pixels) of the tiles if MEM_TILED_RASTER is set to true. If omitted and MEM_TILED_RASTER is set to true it will use the same block size as the input raster. If this parameter is informed the block height must also be present. (Optional)
• MEM_RASTER_TILE_HEIGHT: the height (in pixels) of the tiles if MEM_TILED_RASTER is set to true. If omitted and MEM_TILED_RASTER is set to true it will use the same block size as the input raster. If this parameter is informed the block width must also be present. (Optional)

The parameters above can be used for instance to transfer a GeoTIFF from a file to an in-memory raster:

std::map<std::string, std::string> rinfo;
 
rinfo["MEM_SRC_RASTER_DRIVER_TYPE"] = "GDAL";
rinfo["SOURCE"] = "/data/geotiff/mosaico_brasil.tiff";
rinfo["MEM_TILED_RASTER"] = "TRUE";
rinfo["MEM_RASTER_TILE_WIDTH"] = "50";
rinfo["MEM_RASTER_TILE_HEIGHT"] = "50";
 
te::rst::Raster* myraster = te::rst::RasterFactory::open("MEM", rinfo);

There is another set of parameters that can be used to create an empty in-memory raster when using the make method from RasterFactory class. The rInfo parameter can have the following entries if grid and band properties are omitted:

• MEM_RASTER_NROWS: the number of rows of the new raster.
• MEM_RASTER_NCOLS: the number of cols of the new raster.
• MEM_RASTER_DATATYPE: the datatype of the bands.
• MEM_RASTER_NBANDS: the number of bands of the new raster.
• MEM_RASTER_SRID: the id of the spatial reference system of the new raster. (Optional)
• MEM_RASTER_RES_X: the pixel resolution along x-axis. If the bounding box is omitted it is assumed a value of 1.0. (Optional)
• MEM_RASTER_RES_y: the pixel resolution along y-axis. If the bounding box is omitted it is assumed a value of 1.0. (Optional)
• MEM_RASTER_MIN_X: the bounding box lower-left x. (Optional)
• MEM_RASTER_MIN_Y: the bounding box lower-left y. (Optional)
• MEM_RASTER_MAX_X: the bounding box upper-right x. (Optional)
• MEM_RASTER_MAX_Y: the bounding box upper-right y. (Optional)

The parameters above can be used to create an empty raster as follows:

std::map<std::string, std::string> rinfo;
 
rinfo["MEM_RASTER_NROWS"] = "100";
rinfo["MEM_RASTER_NCOLS"] = "100";
rinfo["MEM_RASTER_DATATYPE"] = te::common::Convert2String(te::dt::UCHAR_TYPE);
rinfo["MEM_RASTER_NBANDS"] = "1";
 
te::rst::Raster* myraster = te::rst::RasterFactory::make("MEM", 0, std::vector<te::rst::BandProperty*>(), rinfo);

If grid and band properties are informed:

// define grid
te::rst::Grid* grid = new te::rst::Grid(100, 100);
 
// define band properties
std::vector<te::rst::BandProperty*> bprops;
bprops.push_back(new te::rst::BandProperty(0, te::dt::UCHAR_TYPE));
 
// define tiles information (in this case, only one block used)
bprops[0]->m_blkh = grid->getNumberOfRows();
bprops[0]->m_blkw = grid->getNumberOfColumns();
bprops[0]->m_nblocksx = 1;
bprops[0]->m_nblocksy = 1;
 
te::rst::Raster* myraster = te::rst::RasterFactory::make("MEM", grid, bprops, std::map<std::string, std::string>());

Another way to create a raster is by providing an external data buffer in the h parameter. In this case one must indicate that this parameter is an external buffer:

• MEM_IS_DATA_BUFFER: if this parameter is set to true the memory raster will assume that the void* parameter is a pre-allocated buffer that can be used by the raster. In this case the client must describe the buffer with the following parameters:
  • MEM_BUFFER_NROWS: the number of rows of the raster represented in the buffer. (Mandatory if MEM_IS_DATA_BUFFER is set to true)
  • MEM_BUFFER_NCOLS: the number of cols of the raster represented in the buffer. (Mandatory if MEM_IS_DATA_BUFFER is set to true)
  • MEM_BUFFER_DATATYPE: the datatype of the bands. (Mandatory if MEM_IS_DATA_BUFFER is set to true)
  • MEM_BUFFER_NBANDS: the number of bands of the raster represented in the buffer. (Mandatory if MEM_IS_DATA_BUFFER is set to true)
  • MEM_BUFFER_SRID: the id of the spatial reference system of the raster represented in the buffer. (Optional)
  • MEM_BUFFER_RES_X: the pixel resolution along x-axis. If the bounding box is omitted it is assumed a value of 1.0. (Optional)
  • MEM_BUFFER_RES_y: the pixel resolution along y-axis. If the bounding box is omitted it is assumed a value of 1.0. (Optional)
  • MEM_BUFFER_MIN_X: the bounding box lower-left x. (Optional)
  • MEM_BUFFER_MIN_Y: the bounding box lower-left y. (Optional)
  • MEM_BUFFER_MAX_X: the bounding box upper-right x. (Optional)
  • MEM_BUFFER_MAX_Y: the bounding box upper-right y. (Optional)

For example:

std::map<std::string, std::string> rinfo;
 
rinfo["MEM_IS_DATA_BUFFER"] = "TRUE";
rinfo["MEM_BUFFER_NROWS"] = "100";
rinfo["MEM_BUFFER_NCOLS"] = "100";
rinfo["MEM_BUFFER_DATATYPE"] = te::common::Convert2String(te::dt::UCHAR_TYPE);
rinfo["MEM_BUFFER_NBANDS"] = "3";
 
void* mybuffer = new unsigned char[100 * 100 * 3];
 
te::rst::Raster* myraster = te::rst::RasterFactory::make("MEM", 0, std::vector<te::rst::BandProperty*>(), rinfo, mybuffer);

Optionally, you can set the parameter FORCE_MEM_DRIVER in the rinfo, as the following:

std::map<std::string, std::string> rinfo;
 
// define the parameter in raster info
rinfo["FORCE_MEM_DRIVER"] = "TRUE";
 
// define grid
te::rst::Grid* grid = new te::rst::Grid(100, 100);
 
// define band properties
std::vector<te::rst::BandProperty*> bprops;
bprops.push_back(new te::rst::BandProperty(0, te::dt::UCHAR_TYPE));
 
// create raster without "MEM" as the first parameter
te::rst::Raster* myraster = te::rst::RasterFactory::make(grid, bprops, rinfo);

It is also possible to create an in-memory Raster from another opened raster (whatever driver has created it):

std::map<std::string, std::string> rinfo;
 
rinfo["SOURCE"] = "/data/geotiff/mosaico_brasil.tiff";
 
// open one raster (input)
te::rst::Raster* inraster = te::rst::RasterFactory::open(rinfo);
 
// create the new in-memory Raster from the input
te::rst::Raster* myraster = te::rst::RasterFactory::make("MEM", 0, std::vector<te::rst::BandProperty*>(), std::map<std::string, std::string>(), inraster);

Notice that the new in-memory raster may take the ownership of the informed void* parameter depending on the deleter parameter. If one provides a deleter the memory raster will use it at the end of its scope.

To create a raster in memory with a single block, and data type 4bits, the code is the following:

// define grid for a 32x32 raster
te::rst::Grid* grid = new te::rst::Grid(32, 32);
 
// define band properties
std::vector<te::rst::BandProperty*> bprops;
bprops.push_back(new te::rst::BandProperty(0, te::dt::R4BITS_TYPE)); // this data type defines the 4bits raster
 
// define tiles information (in this case, only one block used)
bprops[0]->m_blkh = grid->getNumberOfRows();
bprops[0]->m_blkw = grid->getNumberOfColumns();
bprops[0]->m_nblocksx = 1;
bprops[0]->m_nblocksy = 1;
 
// create raster in memory
te::rst::Raster* mem4bitsraster = te::rst::RasterFactory::make("MEM", grid, bprops, std::map<std::string, std::string>());
 
// fill raster (values in range [0, 15] are allowed for a 4bits raster)
for (unsigned r = 0; r < mem4bitsraster->getNumberOfRows(); r++)
  for (unsigned c = 0; c < mem4bitsraster->getNumberOfColumns(); c++)
    if ((c > 5 && c < 25) && (r > 10 && r < 20))
      mem4bitsraster->setValue(c, r, 15);
    else
      mem4bitsraster->setValue(c, r, 3);
 
// clean up
delete mem4bitsraster;

-------------------------------------------------------------------------------
Language          files     blank   comment      code    scale   3rd gen. equiv
-------------------------------------------------------------------------------
C++                  14       750       296      2181 x   1.51 =        3293.31
C/C++ Header         15       670       605       678 x   1.00 =         678.00
-------------------------------------------------------------------------------
SUM:                 29      1420       901      2859 x   1.39 =        3971.31
-------------------------------------------------------------------------------

• The memory DataSetItem class is used by the following drivers: OGR, PostGIS, SQLite.
• Rever DataSetTypePersistence::update(te::dt::Property* oldP, te::dt::Property* newP)
• We could improve this driver by keeping a list of opened In-Memory data sources and attaching each data source to a name (a new connection parameter):
  • this way we can implement create, drop, exists, …

TO BE DONE