RFC822 Import to MS CRM email activity utility

When migrating from Goldmine and some other CRM systems to MS CRM there is sometimes a need to import historical email.  This is frequently stored in a SQL database in rfc822 format. RFC822 is an email format standard.

What I am providing is a single class along with a file from the MS CRM SDK that this class currently uses. This single class is consolidated from a number of other classes in my import framework and is intended to give you a nearly complete rfc822 import solution that you can modify for your needs.  Every place that you need to make a modification according to your particular system has a //TODO: comment explaining what you need to adjust particular to your configuration. In addition assume that there will be some tweaking required, and make sure to do thorough debugging and testing to verify that your email records are being properly generated.

You can download the source code here.

There are 2 steps to using this class.

Part 1: Initialize a new RFC822ToEmailActivity object.

var emailConversion = new RFC822ToEmailActivity("Contoso"); // Your Orgname

Part 2: Calling the CreateEmailFromRfc822Record() method

CreateEmailFromRfc822Record() should be called in your loop that is reading the rfc822 records

var emailEntity = emailConversion.CreateEmailFromRfc822Record(     rfc822Record, // This is the string holding the whole 822 record     localEmailDomainName, // This is the local email domain name ex. onecrmpro.com      emailOwnerId, // systemUserId of email entity owner     relatedEntityName, // contact or account or....     relatedEntityId // id of the entity to regard the email to
)

1. Calls parsing method
2. Creates email attachments from pathnames returned from parsing method.
3. Writes to an error log file with all attachments that are not found and other errors.
4. Makes any other changes to the email entity before saving it.
5. Sets Email State as completed (sent or received) based on local domain name.

Called Internally: ParseRFC822() parsing and email activity creation.

1. Parse an rfc822 record
2. Create an email activity based on that record.
3. Compare email addresses, both To: and From: with a local domain name to decide if the email is incoming or outgoing.
4. Create ActivityParties for To:, CC:, BCC:, From: related to CRM users, Contacts, Accounts and Leads in that match order precedence.
5. Return a list of pathnames to any attachments.

RFC882 Email Attachments

Products like Gold Mine are storing email bodies in the database but they only have pathnames to the attachments. These attachment paths are frequently on the end user’s local computer which can be unreliable. These attachments may have been renamed, moved or deleted.

Hint1: Create a single shared folder on a file server with a separate tree branch for each computer that has attached files in the email history and modify the pathnames programmatically to match the structure you create on the file server.

Example:
S:\importattachments\C\Documents and Settings\mkovalcson\My Documents

Hint2: Comment out or create a test mode where nothing is written into the CRM database, but missing attachments are still written to a log file where you can see what isn’t found and decide how important it is to hunt attachments down before the final import.

You may want to look at this before you get started.
MS CRM Data Import (Cleaning up the Mess)

MS CRM 4.0 Entity Mapping and Hidden Mappings

Microsoft has a powerful way to manage automatic attribute mappings from entities created from other entities. Typically we see this in the QOI process where an Opportunity is created, and then a Quote, Order or Invoice can be created from the Opportunity and much of the information is automatically copied from the Opportunity to that Quote, Order or Invoice.

Entity Mapping

If you open up your Opportunity from Customize Entities and look at 1:N Relationships, you can see the opportunity_quotes relationship.

If you double click on that you will see mappings. This shows you the names of each source opportunity attribute that is “Mapped” to a target quote attribute when a Quote is created from an opportunity.

This is very useful if you want to add custom fields and have those automatically propagated too or if you want to disable any default mappings.

So what about mapping the opportunityproduct? or quotedetail, salesorderdetail, and invoicedetail for that matter?

Hidden Entity Mapping

Step 1. You need the EntityMapId for the relationship that you want to map.

You can manually scan the EntityMapBase table matching against the SourceEntityName and TargetEntityName.

Or you can write a select like the one below to get a nice list of Guids to copy and paste into a temporary form.

Select TargetEntityName, EntityMapId from dbo.EntityMapBase
where SourceEntityName = 'opportunityproduct'

Step 2. Take the following URL and replace your server name and the EntityMapId of your choice to gain access to the rest of the mappingList editors.

http://localhost:5555/Tools/SystemCustomization/Relationships/Mappings/mappingList.aspx?mappingId=3A116CD4-A5EE-DD11-BDF0-0003FFEB167C

CRM Demos, SSD Drives and Virtual Machines

When doing a demo you never want the software to look slow, and when doing CRM development against virtual machines performance is always critical to being efficient.

The Hardware
Samsung recently released their new 256Gb SSD drive and for me this drive signals that SSD drives have become mature. Intel has had very fast SSD drives available, but they were not very large. Like most other SSD drives they suffered from a condition where over time the drive would drop dramatically in performance. Samsung’s new drive has algorithms to minimize this to the point of it being of it being much less noticeable while still performing very similarly to the Intel SSD’s in access time and throughput.

So hoping that this drive lived up to my expectations, I replaced my laptop’s 7200 rpm 160Gb drive with Samsung’s new 256Gb SSD drive. My main concern was that running my VM’s off the system drive would still be unacceptable.

Virtual Machines
Before: I have run my vhd files from separate Firewire 800 Raid 0 drives. Keeping the system drive separate from the VHD was critical to getting good virtual machine performance.

After: The MS CRM 4.0 VHD 2009 now loads up in a mere 14 seconds while still running on the system drive versus 52 seconds on my external Firewire 800 drive. VS 2008 loads up after a clean boot(nothing cached) in 17 seconds and closes in 2 seconds compared with 57 and 13 seconds on my old drive.

Developing against a VM while on battery
This allows real development work to be done while running on battery power. The new SSD drive uses less power than the original drive did, and lugging around external drives for demos is now unnecessary.

As I write this the new Samsung drive is currently available at Dell as part 341-9999 for $699.99.  For this you get 256Gb at the speed of a 10,000 rpm RAID system that fits in a laptop while sipping at your batteries. In some ways, especially average access time, it actually blows an expensive and bulky RAID system away.

Even MORE Speed Possible
I’ve seen reports twin drive laptops getting transfer rates of 300 MB/sec with two of these Samsung drives in a RAID 0 configuration. For comparison a single Samsung drive has a transfer rate of 200MB/sec which is still a lot faster than the 60Mb/sec of a 7200rpm internal laptop drive my less than a year old laptop was using.

The Installation
The replacement was easy. I used my Acronis backup software to make a complete disk backup to one of my external RAID drives. For my Dell M6300 I had to remove 4 screws, pull the drive out. Remove two more screws for the surrounding case. Screw the new drive into the case, slide it into the computer and put the 4 attaching screws back in. Boot from the Acronis CD. It recognizes all of my external drives USB and Firewire. Use the Acronis software to “add a drive” and partition it. Then restore the backup, remove the CD and reboot. The backup and restore(much faster) took me about 2.5 hours. Everything else took about 15 minutes.

MS CRM 4.0 Generate SSRS reports by webservice ( pdf , HTML , CSV , etc..)

Below is a Generic SSRS web service reporting method that generates a byte[] perfect for use in an annotation or email attachment. The example method above it shows how to call this method and how to create an annotation attachment.

You can combine this with my post on driving email templates and generating email attachments to create some very powerful solutions. If you need a primer on activity parties see my post on creating activity parties.

Generating SSRS reports by web service is a very valuable tool. I frequently use this to generate custom reports from custom web applications running in iframes such that the report is run with the permissions of the user running the report. To take advantage of this the reports must be run against MS CRM’s filtered views.

To Generate an SSRS report by web service do the following:

1. Add a report execution web reference to your solution. Make sure this link is accessible. The url should be something like the following:
http://<SRS_Server_Name>/ReportServer/ReportExecution2005.asmx

2. Build up the parameters that you need to generate the report.

3. Select the Report Name of your Report

4. Specify the Type of Report

5. Specify the Device information ( formatting )

The example method CallReport() below demonstrates these steps. It in turn calls GenerateSRSbytes which makes the actual web service call.

using System.Net;
using Microsoft.Win32;
using ReportExecutionService;


public class SRSReporting
{

    public void CallReport()
    {
        string reportServiceUrl = "http://<SRS_Server_Name>/ReportServer/ReportExecution2005.asmx";

        // Create the Report Service Url from the registry
        RegistryKey key = Registry.LocalMachine.OpenSubKey("Software\\Microsoft\\MSCRM", false);

        if (key != null)
        {
            reportServiceUrl = (string)key.GetValue("SQLRSServerURL") + @"/ReportExecution2005.asmx";
        }

        // Report parameter Name and Value fields are strings.

        string reportParameter = "type";
        string reportValue = "annual";

        // Create as many parameters as you need.
        var parameters = new[]
         {
             new ParameterValue {Name = "entityid", Value = "0031a390-0d42-de11-9aa9-0003ff517b20"},
             new ParameterValue {Name = reportParameter, Value = reportValue}
         };

        // Pathname to the report from Reporting Services, where TestReport is the rdl uploaded using http://<SRS_Server_Name>/Reports
        // Make sure that you set the Data Source properly
        // Make sure to set security properly
        string reportName = "/AutomatedCRMReporting/TestReport";

        // Specify what type of report you want to create HTML,PDF, CVS, Excel, Word, XML, MHTML, Image
        string reportFormat = "PDF";

        // Specify the device information to control the output of your report.
        //For device information See http://msdn2.microsoft.com/en-us/library/ms155397.aspx
        string deviceInformation = "<DeviceInfo><DpiX>240</DpiX><DpiY>240</DpiY><StartPage>0</StartPage></DeviceInfo>";

        // Generate a report in a format for a CRM annotation or email attachment
        byte[] generatedReport = GenerateSRSbytes(reportName, parameters, reportFormat, deviceInformation, reportServiceUrl, null, null, null);

        // An example attachment created from an SRS report
        //var newAnnotation = new annotation
        //  {
        //      objectid = CrmTypes.CreateLookup(regardingEntity, regardingId),
        //      objecttypecode = CrmTypes.CreateEntityNameReference(EntityName.contact.ToString()),
        //      isdocument = CrmTypes.CreateCrmBoolean(true),
        //      ownerid = CrmTypes.CreateOwner(EntityName.systemuser.ToString(), ownerId),
        //      filename = reportFileName,
        //      documentbody = Convert.ToBase64String(generatedReport),
        //      filesize = CrmTypes.CreateCrmNumber(generatedReport.Length),
        //      subject = reportFileName
        //  };

        //service.Create(newAnnotation);
    }


    /// <summary>
    /// Generates an SSRS Report and returns a byte[] for a CRM attachment
    /// </summary>
    /// <param name="reportPath">Name of the report.</param>
    /// <param name="parameters">The parameters.</param>
    /// <param name="outputFormat">The output format.</param>
    /// <param name="deviceInformation">The device information. See http://msdn2.microsoft.com/en-us/library/ms155397.aspx</param>
    /// <param name="ReportServiceUrl"></param>
    /// <param name="userName">if null, DefaultNetworkCredentials are used</param>
    /// <param name="passWord">if null, DefaultNetworkCredentials are used</param>
    /// <param name="domainName"></param>
    /// <returns></returns>
    public static byte[] GenerateSRSbytes(string reportPath, ParameterValue[] parameters, string outputFormat, string deviceInformation, string ReportServiceUrl, string userName, string passWord, string domainName)
    {
        string encoding;
        string mimeType;
        string extension;
        string[] streamIDs;
        string SessionId;
        string historyID = null;
        Warning[] warnings;

        // By default the Report will run with the permissions of the AD authenticated User.
        var rs = new ReportExecutionService.ReportExecutionService
        {
            Credentials = System.Net.CredentialCache.DefaultNetworkCredentials,
            Url = ReportServiceUrl
        };


        // Impersonate credentials if they are specified. 
        if (userName != null && passWord != null)
        {
            if (domainName == null)
            {
                rs.Credentials = new NetworkCredential(userName, passWord);
            }
            else
            {
                rs.Credentials = new NetworkCredential(userName, passWord, domainName);
            }
        }

        // Set timeout in seconds of the report takes a long time.
        //rs.Timeout = 600000;

        ExecutionHeader execHeader = new ExecutionHeader();
        rs.ExecutionHeaderValue = execHeader;

        var execInfo = new ExecutionInfo();
        execInfo = rs.LoadReport(reportPath, historyID);

        rs.SetExecutionParameters(parameters, "en-us");

        SessionId = rs.ExecutionHeaderValue.ExecutionID;

        // Render Report
        return rs.Render(outputFormat, deviceInformation, out extension, out mimeType, out encoding, out warnings, out streamIDs);

    }
}

Happy Coding!

MS CRM 4.0 Many to Many Query helper

Rather than trying to explain how to query a many to many relationship, I thought it might be easier for many people to take a working generic method that retrieves a collection of entities from a Many to Many relationship and modify it as needed.

When a many to many relationship is created CRM creates a link table. Default naming conventions could give you a name like this

new_new_entity1_new_entity2

which is a link table between new_entity1 and new_entity2

The three entities involved are the following:

• linkTableEntity - new_new_entity1_new_entity2
• filterEntity – new_entity1
• put your conditions against this
• returnedEntity – new_entity2
• specify your return attributes against this

The arguments to call this method are the following:

• service = ICRMService
  
• linkTableEntityName= "new_new_entity1_new_entity2"
  
• filterEntityName = "new_entity1"
• filterEntityIdName = "new_entity1id"
• filterAttribute = "new_name"
• filterValue = "my entity’s name"
  
• returnedCollectionEntityName = "new_entity2"
• returnedCollectionIdName = "new_entity2id"
• returnedAttibutes = null  //null returns all columns
  or returnedAttributes = new[]{"new_entity2id","new_name"}

/// <summary>
/// Generic Method to retrieve a collection of all DynamicEntities related to another Entity in a Many to Many relationship
/// 
/// 
/// </summary>
/// <param name="service"></param>
/// <param name="linkTableEntityName"></param>
/// <param name="filterEntityName"></param>
/// <param name="filterEntityIdName"></param>
/// <param name="filterValue"></param>
/// <param name="returnedCollectionEntityName"></param>
/// <param name="returnedCollectionEntityIdName"></param>
/// <param name="filterAttribute"></param>
/// <param name="returnedAttributes">if null, AllColumns() is used</param>
/// <returns></returns>
public static BusinessEntityCollection RetrieveEntityCollectionFromManyToMany(ICrmService service, 
    string linkTableEntityName,
    string filterEntityName, string filterEntityIdName, string filterAttribute,  string filterValue,
    string returnedCollectionEntityName, string returnedCollectionEntityIdName, string[] returnedAttributes)
{
    // Selection against linked Filter Entity
    var con = new ConditionExpression
    {
        AttributeName = filterAttribute,
        Operator = ConditionOperator.Equal,
        Values = new[] { filterValue }
    };

    var filter = new FilterExpression
    {
        FilterOperator = LogicalOperator.And
    };
    
    filter.AddCondition(con);

    // the Entity that you are filtering with
    var filterLinkEntity = new LinkEntity
       {
           LinkToEntityName = filterEntityName,
           LinkFromAttributeName = filterEntityIdName,
           LinkToAttributeName = filterEntityIdName,
           LinkCriteria = filter
       };
    
    // The linktable Entity that CRM generates
    var linkTableEntity = new LinkEntity
      {
          LinkToEntityName = linkTableEntityName,
          LinkFromAttributeName = returnedCollectionEntityIdName,
          LinkToAttributeName = returnedCollectionEntityIdName
      };

    linkTableEntity.LinkEntities.Add(filterLinkEntity);

   
    // The Entity returning the results
    var expression = new QueryExpression
    {
        EntityName = returnedCollectionEntityName
    };

    // Set columns being returned
    if (returnedAttributes == null)
    {
        expression.ColumnSet = new AllColumns();
    }
    else
    {
        expression.ColumnSet = new ColumnSet(returnedAttributes);
    }

    expression.LinkEntities.Add(linkTableEntity);

    var request = new RetrieveMultipleRequest {Query = expression, ReturnDynamicEntities = true };

    var response = (RetrieveMultipleResponse)service.Execute(request);
    
    return response.BusinessEntityCollection;
}

Accessing and Controlling the CrmForm from an iframe.

If you write custom web applications in iframes that depend on the data in the parent form, it is important to run on up to date information to avoid user confusion.

Important! This requires that the Restrict cross-frame scripting box for your iframe is NOT checked.

Accessing attribute values on the Parent Form can be done simply by adding a parent.document in front of your normal attribute names. This gets you the values currently on the form, but doesn't guarantee those values have been saved.

var myAttributeValue = parent.document.crmForm.all.new_attribute.DataValue;

Parent Form Save "if modified" This is a nice method to call because if there are no modified values on the form it won't do anything and your application can continue on it's merry way.

parent.document.crmForm.Save();

Force a Parent Save Sometimes you want to force a save so that a Plug-in is called to do some additional calculations even though no attributes have been modified on your current form. The recalculate on the Opportunity, Quote, Order, and Invoice forms is a good example.

Note: The SubmitCrmForm method is not a MS supported method, but it hasn't changed in the last couple CRM revisions.

SubmitCRMForm( Mode, Validate, ForceSubmit, closeWindow)

Modes:
1 = Save, 2 = SaveAndClose, 7 = Send,
58 = SaveAsCompleted, 59 = SaveAndNew

Example: In this example I am forcing a save so that a plug-in is called on the Pre Update for the quote to create a new total value from the quote details that were filled in when the Quote was created from an Opportunity. These new fields were  mapped between the detail products see Mapping Note below. The iframe contains a button to do something that requires all of the data on the quote be up to date.

function RecalculateQuote() 
{ 
   if ( parent.document.crmForm.all.new_totalamount.DataValue == null)
   {    
    parent.document.crmForm.SubmitCrmForm(1, true, true, false);   
    return false;      
    } 
    
    parent.document.crmForm.Save();
}

The above JavaScript is made to work with an OnClientClick so that a return false will not generate a postback.

Why do we need a return false option?  If the validation fails, a message is generated for the user, the save is not performed, and you do not want your action to run.

The data has already been filled in, then the crmFrom.Save() will check to make sure that any modified fields on the form have been saved.

<asp:Button ID="RunMyNeatFeature" CssClass="button" runat="server" Text="Go" 
OnClick="RunMyNeatFeature_Click"  
OnClientClick="javascript:return RecalculateQuote();"/>

For the button the OnClick will PostBack and run the functionality I care about only if the RecalculateQuote() is not false.

This safeguards the user from running the functionality with different data than what is on the screen when they press the "Go" button.

Mapping Note: For mapping details open SQL Management Studio and look at the EntityMapBase table to find the EntityMapId that has the SourceEntityName and TargetEntityName that you are interested in. Add your guid to the end of the URL below and it will bring up a form to map attributes between the Source and Target Entities.

http://yourcrmservername:5555//tools/systemcustomization/relationships/mappings/mappinglist.aspx?mappingId=

Techniques for Driving MS CRM Email Templates

The email templates in MS CRM can be leveraged as a powerful building block when generating emails programmatically.

Tips:

• Generate your HTML in another editor and then paste it into the template editor to add data slugs.
• Associate your template with the entity that you want the most information from.
• Use Template Naming conventions that organize things and make programmatic template selection easier.

For an example lets say that we have product specific Quote Letters.

Using my helper method from a previous blog article, grab an email template matching a specific name based on the product.

// Get appropriate Template
BusinessEntityCollection templates = 
   h.GetAllEntitieswithFilter(EntityName.template.ToString(), new[] { "templateid" }, new[]{"title"}, new[]{emailTemplateTitle});

Then create a new InstantiateTemplateRequest using the template id and point it at an entity with the information that you want merged in. This must be the same type that you associated your template with when you created it.

var instTemplate = new InstantiateTemplateRequest
     {
         TemplateId = thisTemplate.templateid.Value,
         ObjectId = quoteId,
         ObjectType = EntityName.quote.ToString()
     };


// Execute the request to create an email message from the template.
var instTemplateResponse = (InstantiateTemplateResponse)h.service.Execute(instTemplate);

var newEmail = (email)instTemplateResponse.BusinessEntityCollection.BusinessEntities[0];

The result of the InstantiateTemplateResponse is an email entity collection. For this example we just have one email.

From here you can regard the email to a completely different entity if you want. It is only important that the ObjectId and ObjectType be set for the merge process. After that is complete this is just another email with a bunch of fields pre-filled. Now you can have your way with it like any other email that you created from scratch. This is critical to making the best use of email templates.

Additional Merging
If you are calculating other numbers or pulling information from another data source outside of CRM, there is no reason to limit merging to what can be represented as data slugs.

In this example I am using putting additional information into the email body with a simple string replace using an agreed upon naming convention to represent additional merge fields.

// If template had #EmbedLicense# in body replace it with the license information
newEmail.description = newEmail.description.Replace("#EmbedLicense#", embedFile);

From here you can add attachments as shown here.

Proofing before Email is Sent
Another useful thing to do if you are doing this in a web application or a client Winform application is save the email and then open it in a new window for final review before it is sent. To do that generate a URL like the following with your orgname and email id passed in. You can get the CRM server base URL from the registry as shown here which you need for your web service reference, and then use the following: CrmServerBaseUrl = CrmServiceUrl.Substring(0, CrmServiceUrl.ToLower().IndexOf("mscrmservices"));

 // Open newly created email activity at this Url.
var emailUrl = h.CrmServerBaseUrl + "/"+orgname+"/activities/email/edit.aspx?id={" + emailId + "}";

From a WinForm application you can pop a CRM screen up like this.

For a web application register some JavaScript in the code behind that sets the value of the new email's URL.

 // Sets Javascript variables to envoke Window Open to new Explorer Window with body onload event
const string SETQUERYSTRING_SCRIPT = "Email_String";

if (!Page.ClientScript.IsClientScriptBlockRegistered(SETQUERYSTRING_SCRIPT))
{
   Page.ClientScript.RegisterClientScriptBlock(GetType(), SETQUERYSTRING_SCRIPT, string.Format(@"<script language=""javascript"">emailRecordUrl = '{0}';</script>", emailUrl));
}

Then in the aspx file use the following JavaScript code to open the new window. Call the script from the onload of the body.

   <script type="text/javascript">

    var emailRecordUrl = "";   
    
    function UpdateCRMFormGenerateEmail()
    {
       if (emailRecordUrl != "")
       {
           window.open( emailRecordUrl, 
               "_blank", "toolbar=no,scrollbars=yes,resizable=yes");
           emailRecordUrl = "";
       }   
   }

    </script>
</head>
<body onload="UpdateCRMFormGenerateEmail();">