MS Dynamics Rest OData API with bearer token(OAuth2.0) using Script Component in SSIS

What is Issue?

There are many connectors available to connect MS Dynamics 365 CRM online in SSIS but all are having their own limitations and licensing constraints while using. So how can we avoid using these connectors and utilize what is available in SSIS by default?

The Reason-

SSIS provides connection managers to connect all available databases but when it comes for MS Dynamics 365 CRM Online, you don’t have any database in your control. So the other way remain is the Rest API call. But there is also no Rest API Connector by default available in SSIS.

Solution-

The solution for this issue is script component available in SSIS. It provides you the scripting ability and you can do what ever you want with script.

I am using the same to connect the MS Dynamics 365 CRM Online OData Rest API with OAuth 2.0 authentication method.

Let’s go step by step-

  1. Add Script component in Data Flow and select as the Source-

2. Double click on Script Component ->”input and Output” and Add some output columns with some proper data types, so that you can add data in output column fetched from API like below-

3. After adding output column click on “Script” on left hand side menu and click on Edit script-

4. It will open another visual studio window with “VstaProject”-
Add the references –
1. Microsoft.CSharp
2. System.Net
3. System.Net.Http
4. System.Web.Extensions

5. Now open the main.cs and add below sample code and change the variables and queries as per your need-
(Below is the example to fetch “estimatedclosedate” and “opportuinityid” from opportunity entity using Rest API.)

#region Help:  Introduction to the Script Component
/* The Script Component allows you to perform virtually any operation that can be accomplished in
 * a .Net application within the context of an Integration Services data flow.
 *
 * Expand the other regions which have "Help" prefixes for examples of specific ways to use
 * Integration Services features within this script component. */
#endregion

#region Namespaces
using System;
using System.Collections.Generic;
using System.Data;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Web.Script.Serialization;
#endregion

/// <summary>
/// This is the class to which to add your code.  Do not change the name, attributes, or parent
/// of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    #region Help:  Using Integration Services variables and parameters
    /* To use a variable in this script, first ensure that the variable has been added to
     * either the list contained in the ReadOnlyVariables property or the list contained in
     * the ReadWriteVariables property of this script component, according to whether or not your
     * code needs to write into the variable.  To do so, save this script, close this instance of
     * Visual Studio, and update the ReadOnlyVariables and ReadWriteVariables properties in the
     * Script Transformation Editor window.
     * To use a parameter in this script, follow the same steps. Parameters are always read-only.
     *
     * Example of reading from a variable or parameter:
     *  DateTime startTime = Variables.MyStartTime;
     *
     * Example of writing to a variable:
     *  Variables.myStringVariable = "new value";
     */
    #endregion

    #region Help:  Using Integration Services Connnection Managers
    /* Some types of connection managers can be used in this script component.  See the help topic
     * "Working with Connection Managers Programatically" for details.
     *
     * To use a connection manager in this script, first ensure that the connection manager has
     * been added to either the list of connection managers on the Connection Managers page of the
     * script component editor.  To add the connection manager, save this script, close this instance of
     * Visual Studio, and add the Connection Manager to the list.
     *
     * If the component needs to hold a connection open while processing rows, override the
     * AcquireConnections and ReleaseConnections methods.
     * 
     * Example of using an ADO.Net connection manager to acquire a SqlConnection:
     *  object rawConnection = Connections.SalesDB.AcquireConnection(transaction);
     *  SqlConnection salesDBConn = (SqlConnection)rawConnection;
     *
     * Example of using a File connection manager to acquire a file path:
     *  object rawConnection = Connections.Prices_zip.AcquireConnection(transaction);
     *  string filePath = (string)rawConnection;
     *
     * Example of releasing a connection manager:
     *  Connections.SalesDB.ReleaseConnection(rawConnection);
     */
    #endregion

    #region Help:  Firing Integration Services Events
    /* This script component can fire events.
     *
     * Example of firing an error event:
     *  ComponentMetaData.FireError(10, "Process Values", "Bad value", "", 0, out cancel);
     *
     * Example of firing an information event:
     *  ComponentMetaData.FireInformation(10, "Process Values", "Processing has started", "", 0, fireAgain);
     *
     * Example of firing a warning event:
     *  ComponentMetaData.FireWarning(10, "Process Values", "No rows were received", "", 0);
     */
    #endregion

    private String clientID, secretKey, resource, oauthToken, apiUrl, tokenURL;
    private HttpClient httpClient;

    /// <summary>
    /// This method is called once, before rows begin to be processed in the data flow.
    ///
    /// You can remove this method if you don't need to do anything here.
    /// </summary>
    public override void PreExecute()
    {
        base.PreExecute();
        /*
         * Add your code here
         */
        apiUrl = "https://xxxxxx.crm8.dynamics.com/api/data/v8.2/";
        clientID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx"; // Client id to generate Bearer Token
        secretKey = "xxxxxxxxxxxxxxxxxxxx"; //Secret Key
        resource = "https://xxxxxx.crm8.dynamics.com"; //resource
		tokenURL="https://login.microsoftonline.com:443/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx/oauth2/token/"
        GetBearerToken();
    }

    private void GetBearerToken()
    {
        HttpClientHandler clientHandler = new HttpClientHandler();
        clientHandler.Proxy = WebRequest.GetSystemWebProxy();
        clientHandler.Proxy.Credentials = CredentialCache.DefaultCredentials; // or new NetworkCredential("username","password","DOMAIN");
        clientHandler.UseProxy = true;
        httpClient = new HttpClient(clientHandler);
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
        HttpContent requestContent = new FormUrlEncodedContent(new Dictionary<string, string>
            {
                {"grant_type","client_credentials"},
                {"client_id",clientID},
                {"client_secret",secretKey},
                {"resource",resource}
            }
        );
        HttpResponseMessage response = httpClient.PostAsync(tokenURL, requestContent).Result;

        String responseString = response.Content.ReadAsStringAsync().Result;

        JavaScriptSerializer serialiser = new JavaScriptSerializer();
        dynamic apiResult = serialiser.DeserializeObject(responseString);

        oauthToken = (String)apiResult["access_token"];
    }

    /// <summary>
    /// This method is called after all the rows have passed through this component.
    ///
    /// You can delete this method if you don't need to do anything here.
    /// </summary>
    public override void PostExecute()
    {
        base.PostExecute();
        /*
         * Add your code here
         */
    }

    public override void CreateNewOutputRows()
    {
        /*
          Add rows by calling the AddRow method on the member variable named "<Output Name>Buffer".
          For example, call MyOutputBuffer.AddRow() if your output was named "MyOutput".
        */
        String query = "$select=opportunityid,estimatedclosedate";
        String crmRestQuery = apiUrl + "opportunities?" + query;

        Boolean isDone = false;
        while (isDone == false)
        {
isDone = true;
            HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, crmRestQuery);

            //add header parameters
            request.Headers.Add("Prefer", "odata.include-annotations=\"*\"");//for formatted values
            request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            request.Headers.Add("Authorization", "Bearer " + oauthToken);

            //send request
            HttpResponseMessage response = httpClient.SendAsync(request).Result;
            String responseString = response.Content.ReadAsStringAsync().Result;

            JavaScriptSerializer serialiser = new JavaScriptSerializer();
            serialiser.MaxJsonLength = 50 * 1000000;	// Increased to deserialize large number of fields 
            dynamic apiResult = serialiser.DeserializeObject(responseString);

            //--------process dynamic apiResult

            object[] result = null;
            //get only values from response
            foreach (var item in apiResult)
            {
                object key = item.Key;
                if ((string)key == "value")
                    result = (object[])item.Value;
              if ((string)key == "@odata.nextLink")
                {
                    isDone = false;
                    crmRestQuery = (string)item.Value;
                }
            }
            //array of object to list
            List<object> res = result.ToList<object>();
            //for each record in result
            foreach (var item in res)
            {
                //add row of output columns for each record
                Output0Buffer.AddRow();

                Dictionary<string, object> obj = (Dictionary<string, object>)item;
                //all attributes from Record (object)
                foreach (KeyValuePair<string, object> item1 in obj)
                {
                    if (item1.Key == "opportunityid")
                        Output0Buffer.ID = (String)item1.Value;
                    if (item1.Key == "estimatedclosedate")
                        Output0Buffer.EstCloseDate = (String)item1.Value;
   if (item1.Key == "estimatedclosedate@OData.Community.Display.V1.FormattedValue")
                        Output0Buffer.EstCloseDateFormatted = (String)item1.Value;
                }
            }
        }
    }
}

In case of API using FetchXML you can refer the below code-
We can use pagingcookie for 5000+ records

    public override void CreateNewOutputRows()
    {
        /*
          Add rows by calling the AddRow method on the member variable named "<Output Name>Buffer".
          For example, call MyOutputBuffer.AddRow() if your output was named "MyOutput".
        */
        string query = @"<fetch {0} version='1.0' output-format='xml-platform' mapping='logical' distinct='false'>
                                         <entity name='opportunity'>
                                            <attribute name='opportunityid' />
                                            <attribute name='estimatedclosedate' />
                                          </entity>
                                        </fetch>";

        int page = 1;
        var cookie = string.Format("page='{0}'",page);

        string xml = string.Format(query, cookie);
        string crmRestQuery = apiUrl + "opportunities?fetchXml=" + xml;

        bool isDone = false;
        while (isDone == false)
        {
            isDone = true;
            HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, crmRestQuery);
            //add header parameters
            request.Headers.Add("Prefer", "odata.include-annotations=\"*\"");//for formatted values
            request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            request.Headers.Add("Authorization", "Bearer " + oauthToken);

            //send request
            HttpResponseMessage response = httpClient.SendAsync(request).Result;
            string responseString = response.Content.ReadAsStringAsync().Result;

            JavaScriptSerializer serialiser = new JavaScriptSerializer();
            serialiser.MaxJsonLength = 50 * 1000000;	// Increased to deserialize large number of fields 
            dynamic apiResult = serialiser.DeserializeObject(responseString);

            //--------process dynamic apiResult

            object[] result = null;
            //get only values from response
            foreach (var item in apiResult)
            {
                object key = item.Key;
                if ((string)key == "value")
                    result = (object[])item.Value;
                //when Rest API using fetchXML 
                if ((string)key == "@Microsoft.Dynamics.CRM.fetchxmlpagingcookie")
                {
                    isDone = false;
                    page++;

                   //retrieve pagingCookie value
                   string xmlVal = WebUtility.UrlDecode((WebUtility.UrlDecode((string)item.Value)));
                   int Start = xmlVal.IndexOf("pagingcookie=\"", 0) + "pagingcookie=\"".Length;
                   int End = xmlVal.IndexOf("\" istracking=\"False\"", Start);
                   
                    //proccess the pagingCookie value to support in API URL
                    string strVal= xmlVal.Substring(Start, End - Start).Replace("<", "%26lt;").Replace(">", "%26gt;").Replace("\"", "%26quot;").Replace("'", "%26apos;").Replace("&", "%26amp;");
                    cookie = string.Format("page='{0}' paging-cookie='{1}'",page, strVal);
                    
                    //modify the query for next page with paging cookie
                    xml = string.Format(query, cookie);
                    crmRestQuery = apiUrl + "opportunities?fetchXml=" + xml;

                }

            }
            //array of object to list
            List<object> res = result.ToList<object>();
            //for each record in result
            foreach (var item in res)
            {
                //add row of output columns
                Output0Buffer.AddRow();

                Dictionary<string, object> obj = (Dictionary<string, object>)item;
                //all attributes from Record (object)
                foreach (KeyValuePair<string, object> item1 in obj)
                {
                    if (item1.Key == "opportunityid")
                        Output0Buffer.ID = (String)item1.Value;
                    if (item1.Key == "estimatedclosedate")
                        Output0Buffer.EstCloseDate = (String)item1.Value;
                    if (item1.Key == "estimatedclosedate@OData.Community.Display.V1.FormattedValue")
                        Output0Buffer.EstCloseDateFormatted = (String)item1.Value;
                }

            }
        }

6. Build the VstaProject and close the VstaProject Visual Studio window.

7. Click on Ok button in Script Component Properties window.

8. Save the SSIS project and execute the Data flow Task.

9. When you add Data viewer on output of Script component, you will get desired output-

I used client_credentials method to generate Bearer Token, you can use the other methods and modify GetBearerToken() method.

The all the parameters I used as hardcoded, you can manage it by passing as input/output parameters.

Hope this will help…
Enjoy Ms CRM!!!

Follow on Facebook- FB: MSCRM16Tech

Get SharePoint Document details for particular record in MS CRM using FetchXML

What is Issue?

There are many cases while implementing MS CRM where clients ask to perform some operation in MS CRM based on the document uploaded in SharePoint. But how to get SharePoint document details in MS CRM other than “Documents” associated view? If you try to fetch records using advanced find or querying sharepointdocument entity directly, it will show you error. So how we can query and get the document details using code either from client side or server side?

The Reason-

MS CRM Supports only the associated view for Document i.e. sharepointdocument entity. MS CRM do not support direct querying on all sharepointdocument.

Solution-

There is a way to fetch SharePoint document details for particular record in MS CRM.

If you try to query on Document entity in MS CRM using Advanced find you will get the error-” SharePoint document entity does not support any view other than associated view”.

Now this seems trouble to get sharepoint document details. Don’t worry, as error itself says, the sharepoint document does not support any view other than associated view, We can keep building query in advance find itself.

Suppose I want to get documents details for one of my Opportunity in MS CRM , So I am building the query in the following way-
1. Look for : Document
2.Use Saved View :All Sharepoint Documents
3.If you want, you can edit the columns
4. Select Regarding(Opportunity) contains data
5. Select sub query for opportunity record(for which you want to get details)

If you click on results, it will still show you same error. Don’t worry, download the fetchXML for the query we built.

<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false">
  <entity name="sharepointdocument">
    <attribute name="documentid" />
    <attribute name="fullname" />
    <attribute name="relativelocation" />
    <attribute name="sharepointcreatedon" />
    <attribute name="filetype" />
    <attribute name="absoluteurl" />
    <attribute name="modified" />
    <attribute name="sharepointmodifiedby" />
    <attribute name="title" />
    <attribute name="readurl" />
    <attribute name="editurl" />
    <attribute name="author" />
    <attribute name="sharepointdocumentid" />
    <attribute name="ischeckedout" />
    <attribute name="locationid" />
    <attribute name="iconclassname" />
    <order attribute="relativelocation" descending="false" />
    <link-entity name="opportunity" from="opportunityid" to="regardingobjectid" link-type="inner" alias="ad">
      <filter type="and">
        <condition attribute="opportunityid" operator="eq" value="{83779C3E-xxxx-xxxx-xxxx-0CD3BEAFE160}" />
      </filter>
    </link-entity>
  </entity>
</fetch>

When I tried to use the fetchXML in Javascript query, I got the Sharepoint Document details in json as below-

As sharepoint document entity supports only associated view, it requires regarding object to query. So if you try to remove and link-entity tag from fetchXML, it will not return any record in response.

Now we have fetchXML which we can utilize in Javascript or C# based on requirement.

Hope this will help…
Enjoy Ms CRM!!!

Follow on Facebook- FB: MSCRM16Tech

ReadOnly Subgrids in MS CRM

What is Issue?

While working on MS CRM projects I had requirement of readOnly subgrids in MS CRM. You can achieve this to some extent by hiding buttons on subgrids. But if the requirement is like links should not be clickable for some of the security roles or for some conditions, it is not possible using out-of-box subgrids. That time we had no other option than going with html and JavaScript web resources for this.

The Reason-

This can not be achieve using out-of-box subgrids controls.

Solution-

There are 2 solution for this-
1. Go with HTML webresource.
2. Create Your own Custom Control using PCF (PowerApps Control Framework)

I created the completely read-only subgrid with pagination functionality similar to out-of-box subgrid.

Taking my current scenario-
I have to make subgrid completely read-only if the “More Contact Required?”= NO
Or else the subgrid will be normal subgrid.

So what I did –

  1. On form I added 2 similar subgrids with different Names.
  2. Added script on change of “More Contact Required?” field to show required subgrid.
  3. Changed the control for subgrid which is visible when “More Contacts Required?”= NO
    Open Form Editor -> Select Subgrid -> Change Properties -> Controls Tab ->Add Control -> Selected my own custom Control ->ReadOnlySubgridPage -> Click Add
  4. Save Form and Publish.

Now I found it so simple as I can make any subgrid readonly with my custom control.

You can download the Solution from here and import in Dynamics 365 to use the control.

If you want to check source code download from here.

To work with source code you need some of the basic setup in your computer-
1. Nodejs installed
2. VS or Visual Studio Code installed
3.Understanding of below commands-

pac pcf init --namespace yourNamespace --name yourControlName --template dataset
npm install
npm run build
npm start
pac solution init --publisher-name yourPubliserName --publisher-prefix yourPrefix
pac solution add-reference --path "pcfproj folder path"
msbuild /t:restore
msbuild

Hope this will help…
Enjoy Ms CRM!!!

Follow on Facebook- FB: MSCRM16Tech

Color Picker Custom Control for MS Dynamics CRM

What is Issue?

One of my previous projects had requirement of color picker in MS CRM to provide users flexibility to select any color to can display notifications(using HTML) on their records. But we had no other option than going with html and JavaScript web resources for this.

The Reason-

MS CRM don’t provide any color picker field out-of-box.

Solution-

As I already had a blog written for color picker using web resources and spectrum libraries-Color Picker in MS CRM ?

But it is web resource which needs much efforts to match to field styles in crm.

I have created the custom pcf control (PowerApps Custom Control). This control can be used for the single line text fields.

Depends on browser it appears in supported way-

On Browsers other than IE-

This slideshow requires JavaScript.

In IE it will appear as –

This slideshow requires JavaScript.

If you want to use it you form you can download ready custom control from here-ColorPickerSolution.zip

How to use it-
1. Import downloaded solution in CRM-
ColorPicker5

2. Create single line of Text field in any entity you want and add it on form.
3. when adding to form select field, click on change field property and in controls tab add custom control “ColorPicker”-

This slideshow requires JavaScript.

4.Save and publish the form.
Now the control is available and ready on form to use.

 

You can also download the source code from here-ColorPickerSource.zip

To work with source code you need some of the basic setup in your computer-
1. Nodejs installed
2. VS or Visual Studio Code installed
3.Understanding of below commands-


pac pcf init --namespace yourNamespace --name yourControlName --template field
npm install
npm run build
npm start
pac solution init --publisher-name yourPubliserName --publisher-prefix yourPrefix
pac solution add-reference --path "pcfproj folder path"
msbuild /t:restore
msbuild

Hope this will help…
Enjoy Ms CRM!!!

Follow on Facebook- FB: MSCRM16Tech

Compare Security Roles available in MS CRM Organization

What is Issue?

Many times in organization many security roles got created but it becomes difficult to find out differences between them. Checking one by one is time consuming and complicated task.

The Reason-

No direct way available to compare 2 security roles.

Solution-

you can use the tool created by mscrm16Tech as below-
It provides view of all security roles available in Organization and allows to to compare them with each other.

When you install and open the application in your system you will get below window which will help you in configuration of MS CRM connection and database connection if needed. You can connect to CRM using CRM SDK Login window. This feature is available in latest version of software only(v1.0.0.28 and above).

newConfigScreen

After configuring the app and clicks on OK, Select the application “Compare Security Roles” in next window  –

newApplicationList

Now you need to load the available security Roles in both the list  from connected CRM Organization.
Select security role from each list and click compare.

You also have option to check security roles for entities for some publisher only(custom entities).

The output loaded in grid will  privileges on entities available to from respective security roles selected in lists.

Please use below image for reference-

App4Screen

Please follow the images to work with the app.

You can download MSCRMTechTool Here- MSCRM16Tech Tool

Hope this will help…
Enjoy Ms CRM!!!

Follow on Facebook- FB: MSCRM16Tech