Managing data for Continuous Integration

Topics: Managing Data
Coordinator
Sep 4, 2014 at 7:27 PM
What do you think is the best way to manage data for Continuous Integration?

Some scenarios:
  1. Deploy Reference data to a clean test environment
  2. Update Reference data in an existing environment
  3. Move reference data from one environment to the other
  4. Update/Delete transaction data due to changing requirements
Some options:
  1. Manual updates using UI or OOB Import/Export
  2. Custom PowerShell Cmdlets
  3. CRM Data Migration Tool & Deployment Packages
  4. ETL Tools - SSIS/Scribe/Informatica/etc...
Let me know your thoughts?
Coordinator
Sep 4, 2014 at 7:35 PM
For Reference and Configuration Data:
  1. The OOB import/export manual or automated doesn't seem to be the best option. Relationships are hard to define. Also you have to reply on Guids that can be different between various environments.
  2. The new CRM Data Migration tools with 2013 SDK SP1 seems to be a good option here. The issue is that it doesn't yet provide API to export data. However this can be easily imported using a Deployment Package with PowerShell. Can manages your keys and relationship which doesn't have to be Guids
  3. ETL tools seem to be an overkill
For Transaction Business Data:
  1. This type of operations will vary between each release. A manual or automated OOB import or export seem to be a good option. Where this is not possible manual actions or custom powershell cmdlets can help.
  2. For complex scenarios ETL packages can be a good option
  3. Why not write a workflow that you can run on demand manually or though PowerShell to do this?
  4. Bulk Edit can also work for simple one off updates.
Developer
Sep 6, 2014 at 4:43 PM
In a perfect world CI would also include automated tests which requires a set of fixed data to run against.
Its also very practicable if you can easily provide test data for a new developer machine or even refresh the test instance.

May scenario one and two are the same because you can just delete all the data and and just start a re-import.
  1. Why do you think OOB is not good for Relationships? You can define mappings e.g. on the name field which should be unique.
  2. There are different options. a) You script your data with the "Add-XrmEntity" cmdlet or b) we write a Cmdlet which takes an OOB Export (The data as well as the schema) and imports this. The cool thing about option two is, you can edit your test data offline.
  3. There is the ConfigurationMigration and the PackageDeployer where the PackageDeployer has Cmdlets and the ConfigurationMigration does not. As far as I know the PackageDeployer uses just the OOB import (an data file and a schema file). I not used the ConfigurationMigration so far. Do you have any experience with it?
  4. Agree with the ETL
What do you mean with transaction data?