About Datasets

Overviews

About creating and using LeasePak Datasets

About DataSets - LeasePak data exported from the LLDB

A dataset is an integrated group of data files, stored in a single directory on the application host, containing all of the data contained in a LLDB at a point in time. A dataset is also called a snapshot, after the db_snapshot utility which creates them, both referring to the "moment in time" aspect of datasets.

There are three types of data files in a dataset:

Housekeeping files

These include lastsnapshot.txt, which describes the database and dataset contained in the snapshot; terminators.txt, which specifies the field and record terminators used in the production of the dataset files. It is required by the db_restore utility in order to read the data files correctly.
CISAM files

CISAM files are located in the $udatadirectory of a LeasePak environment, and are technically "part of the LLDB," though they are non-relational. While not always present, they are critically important to certain LeasePak server processes when they are present. These files have .dat and .idx filename extensions.
Export files

Export files contain the data of an LLDB found actually stored in the database itself. They are in an export format called "bcp" (bulk copy). Each file contains rows from only one table in the database. Each row is terminated by the record terminator specified in the terminators.txt file, and within each record, the columns of data are terminated by the field terminator specified in terminators.txt.

These export files have the file extension .bcp or .pbcp. .bcp files ostensibly contain all of the data from one particular table. .pbcp files contain only part of the data from that table. The basename is usually the name of the table, but may contain elements that attempt to specify what portions of the table were exported, depending on circumstances..

The export files may have an additional filename extension, consisting of a period plus a number represented in decimal notation, one or more digits in length. Files with these extensions are called segments, and each segment contains a subset of the data in the source database table. Fully restoring a dataset with segmented files requires all of the segments to be loaded into the database.

Locations of Datasets

Datasets are placed by LeasePak in either each build directory (see Directory Trees) under dsets, or in the $TOPDIR/datasets directory. The datasets in dsets are restricted to only the level7 and seed datasets, which are the "official" datasets of LeasePak. They are read-only.

Datasets created at Netsol customer sites are stored in $datasets. It is possible to snapshot to and restore from datasets stored elsewhere on the application host's filesystem, by using the full pathname (starting with '/') as the dataset parameters of db_snapshot and db_restore.

About the level7 and seed datasets

Level7 is a dataset that was the product of a long series of regression test suites. Starting from an "empty" LeasePak database, these tests input customers and applications and vendors and assets and leases, and ran the essential EOP updates, and these became the data in the level7 dataset. It's value to NetSol, and thus to our clients, is that it represents a common starting point for teaching and for trouble-shooting. It is often possible to demonstrate a bug that occurs in production environments on the level7 data. This makes it far easier to troubleshoot in the absence of a client's multi-gigabyte database.

Seed is the "empty" LeasePak database. It contains only one portfolio, the LeasePak codes table, the message table, and a few other essential tables to set up a new instance of LeasePak, with no individual data. It is the "empty" database from which the level7 dataset was grown, and from which every new instance of LeasePak is grown.

ENHANCEMENT IN V6.5A

Query-Based Datasets

Beginning in LeasePak version 6.5a, customer sites on the Oracle platform now have the ability to create partial exports of database tables. Using the -q queryspec parameter with db_snapshot, the utility will export rows from specified tables according to a user-prepared file of SQL select statements, with output going to .pbcp files. Multiple queries may be stored in a single .bcpqry file, and particular queries may be executed by including their ID on the command line along with their .bcpqry filename.

At the same time, db_restore was enhanced to handle multisegment .bcp files
Please see the full documentation on these new capabilities at Custom Bcp Queries. This is for the Oracle DBMS only.