Data Conversion

Converting and loading data into a BSI database can be accomplished using several different methods. You can perform the conversion and load yourself, you can use the translation features in BSI to help convert your data, IMS can assist you in the process, or you can use a combination of these options. The best conversion option for you can be determined by the size and content of your data, how much translation is necessary, your available staffing with the proper expertise, and budget constraints.

Size Matters

The first consideration is the size of your database. The BSI data entry process is limited to adding 10,000 records at a time. If the number of specimen records you wish to add is small (less than 200,000 or so), then the data can be divided up into 10K batches and added to the system. This is the preferred method as it forces all of the new data to go through the standard data entry verification process. If the number of records that you have to import is above 500,000 you should consider asking IMS to import it directly into the database. IMS will then write programs to validate the data and bulk load it into the database. We will then run post conversion checks on the data to ensure that all the data will pass the data entry verification routines.

Content & Cleanliness

The second consideration is the content of your data. Is your data stored with multiple specimens on each record? If so it will need to be expanded to a single specimen on a record. How much data will need to be translated to match the BSI data codes that you have selected? Will data need to be extracted from some fields and created for others?

Associated with content is the error level present in your data. The majority of the time in data conversion is not spent in analyzing, converting, or loading data. Most is spent in the correction of erroneous data detected during the conversion process. In many cases this requires contacting the original collection centers or primary investigators to request additional information on resolving discrepancies. The investigation process (e-mails, programmatic corrections, data reloads) typically requires a significant amount of labor. Since BSI will not allow data into the database that violates any of the data validation criteria (remember you want clean data after the conversion), any errors must be corrected before the data can be loaded into the database.

Cost – The Bottom Line

The final considerations are staff and budget. You might not have the IT resources to convert your data in an acceptable time frame. No problem! IMS offers experienced staff that will walk you through the conversion process and load your data into BSI. This process includes analyzing your data, preparing a data conversion plan for your review, creating custom programs to convert your data, and the execution of those programs along with other conversion plan processes to load your data into BSI. All of this work is performed at our very affordable T&M rates. Existing BSI customers have found that having IMS convert the data is less expensive than doing it in house.

While working to develop a data conversion plan, IMS will also work with you to identify customizations needed for your new BSI database, as well as help you define your BSI-related workflow. See some case studies for examples of past data conversions.