Importing Data into MS CRM - Overview

MS CRM is a great place to permanently consolidate data silos, which could be in Excel, Access, external SQL databases or any other data source, but there are also many useful reasons for bringing data into CRM periodically or just having CRM accessing external data stores.

Integrating MS CRM with the rest of the world is a passion of mine and pulling data into CRM, driving CRM with external data and sharing data from CRM to drive external processes is key to powerful integration.

I put CRM data exchanges into a number categories.

1. One Shot: Initial Imports from systems replaced by CRM.
2. Scheduled Imports: Ongoing data pulled from another system.
3. Synchronization: Synchronization with an external database.
4. Business Process drivers: External data is consulted to start business processes within CRM.
5. Integration: CRM and another application share data with each other.
6. External Control: CRM drives external processes

What I would like to do is examine these topics and share some of the tools and parts of my import framework over the next few months.

I think that this is a good point to mention Scribe. I have used Scribe, but none of my tools or examples will be for Scribe.  Scribe has an excellent reputation and is a valuable tool for many. They also offer numerous connectors for moving data between CRM and other mainstream applications. I highly recommend Scribe if you are not a developer familiar with the CRM SDK.

Since I consider myself a .NET developer first, I have found that working with the SDK gives me a lot more low level control and flexibility, and none of my code or methods will require the purchase of a 3rd party tool.

Development side note: While developing my data import framework which uses CRM web services, I had to work with MS to find out why my application kept crashing.  It turned out that it was importing records so quickly that the web services were overloading the TCP/IP Sockets on the W2003 Server. The fix was to modify the registry settings to both drastically increase the number of Sockets available and to reduce the recycle time that that Sockets would sit idle before being reclaimed for reuse. This may not have been a big deal in the grand scheme of things, but that didn't stop me from having a brief Tim Allen moment when I found out that I completely blew by the default settings for W2003 :)