Unfortunately, and as far as I can see, there is no straight forward way to do bulk records deletion on Dynamics CRM 4.0 using the out of the box functionality and interface of Dynamics CRM 4.0.

To bulk delete records in Dynamics CRM 4.0, you have the following main options:

• Get a third party tool or CRM add-on to bulk delete records. This option is a straight forward one but you might have to pay for purchasing or using the tool. It may also have security issues. I would not recommend it to my clients as most probably the tool is created by a small company or an individual which I don’t know. Hence, it will be rather difficult to put this tool on a live Production environment or client server. Let alone adding it to CRM Online or to a CRM hosted solution by a partner.
• Use CRM SDK to write a .NET application (or a .NET console application) that will run and delete all records for a specified entity or entities. This is a more robust way of doing it, but it may take longer time and is probably not suitable for people who do not come from .NET development background.
• Use Scribe Insight. This is what this post is about really.. Using Scribe Insight to bulk Delete Dynamics CRM records.

Please Note: This is a work around. It is not supported by Scribe and the advice in this post is provided as is with no warranty. I have tried it and it works perfectly but can not guarantee it will have the same acceptable results in any other environment.

Here is what you need to do:

1. Create a new Scribe workbench DTS (or Job). Point to your usual source file (even a sample one) and point to CRM: either IFD Forms for hosted CRM or direct connection.
2. Configure the targe: Create one delete step on the target.
3. Make sure that the option to “Allow multiple record matches on updates/deletes” is ticked under the All steps tab.
4. Under Step control tab, leave failure to go to next row but change all the success records (Success (0), Success (1) and Success (>1) ) to End Job. Select success radio button at the bottom and write a message to your log such as: “All records Deleted”.
5. No Data links are important as you are only deleting.
6. On the Lookup link, just make the lookup condition impossible. Such as: where Account Name = 123456789 or whatever.
7. Run the DTS.

The Job will read the first source line. Will then try to find this record at the target (remember it is update/delete). Since we have setup the lookup link to look for something “impossible to find”, the result of the update will be Success (0).

Once this happens, Scribe will go and delete all records for your chosen entity (or CRM table). This will be a complete bulk delete of all CRM records using Scribe.

Remember, it’s a work around… that works.

After a bit of research, I found out that there several UK post code suppliers. The major supplier is of course as expected Royal Mail. Royal Mail provides a service called Postcode Address File (PAF®). The problem though is that Royal Mail will not give you any application, they will just give you some raw data on a CD and they will be sending you updates for this data every month, 3 months, 6 months and even some daily changes depending on your choice. The data is great and I think it does have almost all Britain’s postcode areas and addresses. Regular updates also ensures that the information is up to date. The raw data though needs to be manipulated by an application. Some hard coding will be required which is not too difficult to do but if you are not an experienced developer or have no development background, this will be an issue for you. I will try to address the coding of such functionality in Microsoft .NET (C# and VB.NET) in a later post.

Alternatively, I found out about quite a few online companies that provide solutions for the raw data. So, basically they create web-services which manipulate Royal Mail (PAF ) raw data and all you need to do is to write simple code to return the details you require. So you pass on the post code to them for example and they send you back the full address details. Many of them will even give you example code making your life easier. If you do not need this for a website, you can always opt for their other services such as an online application for you that you can login to and get the address information you require.

Pricing of these post code data providers varies with most of them if not all providing the service using credits. So each credit gives you one address information request. Credits are sold in packs with prices averaging around £50 for 750 credits pack. Do not buy the first one you see, ask around and shop around. Royal Mail though has a different licence structure with prices starting from £85 for a single workstation and £2 for every extract up to £££thousands of pounds for a multi-organisation multi-server licences. 

If you need more information, recommendation and links to the best providers of this post code and address information services, please get in touch and I will get back to you. I have now managed to nail down the best providers and got good offer from them. Just get in touch!

Again, I have looked online and could barely see any information or help in this regard (probably my mistake as I didn’t search properly!). I found out that the key for moving Scribe project files including source across to a new location or a new server, the key is always the QETXT.ini file. This file is vital for pointing the DTS to which source file it should be looking at. QETXT.ini files have all the field names, so mapping S1 to the name that you have chosen for the first source field. It also has the source file name, the ODBC name and the table name. From there you can do almost everything.

When you move the files across, you will obviously need to re-point to the new source location, but by editing QETXT.ini, you will be able to put back all the source (S1 to field name) mappings, point to the new source file name, ODBC name and everything else.

This has proved very efficient and have worked now with my whole deployment.

One more important piece of advice, always try to get a dedicated folder for every source file. So if you have more than one DTS jobs, make sure that the source for each DTS job is in a separate dedicated folder. This will ensure you have separate QETXT.ini file for each one of them, hence, you can easily update the information inside it. It will still work with one large QETXT.ini file but it’s always better to separate the sources and their associated QETXT.ini files. You can always manually split this file into source specific files and put each source in a separate folder later on (which is what I have done after ”inventing” this best practice of having separate source folders!).

Have you ever looked in Scribe Insight for a way to rename a source file before processing it. Scribe console creates collaborations where integration processes can be configured so that they wait for a file to be added to a specific location and then run a specified job. Once the file is added the job will run and process the file. Also, Scribe DTS jobs can only be setup to process a source file that has its name always fixed and unchanged. So a DTS can be setup to process a source file named: customersdata.txt. It will never run if another source file is added to the location the Scribe console is looking at. In this case, if you get a source file with the date (and time) stamp in its name will need to rename it so the the DTS can detect it and run. So if the source file comes with time and date stamp that varies every day (for example: customers_1453_21092009.txt), you will need to rename “customers_1453_21092009.txt” to “customersdata.txt” only to get the DTS to work.

After some research, I found out that you can only do this using pre and post processing commands step of the process. Every integration process has step 2 in it called pre and post processing commands. In this step, you can specify a pre-processing and post-processing commands or scripts. This feature lets you specify a pre and post processing file that can do this renaming for you. Accepted pre and post files are: *.vbs, *.js, *.vbe, *.bat, *.cmd, *.exe, *.com

You will then create a pre-processing script that finds all files that start with “customers*” in our example and rename it to customersdata.txt which is the source file name the job is expecting. Post processing can be to rename the file to something else so that you keep a record of processed files.

In step 3 of the Integration process, Scribe also gives you two options (in the form two check boxes) that I find very useful. You can either select to delete the source file after processing or you can select to rename it. So the source file will be processed and renamed to something like customersdata.L1.txt. Unfortunately Scribe doesn’t give you the choice to choose the new name of the file. Hence, you will need to write a post processing script again for renaming it afterwards if you are after a specific file name.

I’m not sure if there is any other way of renaming source files (also called event files by Scribe) before processing them or specifying a new name for them after processing.

The quick answer is yes. SCRIBE is a good application that can be used to provide an integration between Microsoft Dynamics CRM and SAP.

If you want the report I created studying the strength of SCRIBE and the possibility of using it to do such integration, please request it via a comment on this page and I will email you the document. I will also email you a technical specification document that SCRIBE has sent me which details such integration. This technical document is not publicly available on their website as far as I know but you can always request it directly from them.

My technical report/document also has some examples of case studies in which Microsoft CRM system has been integrated successfullywith SAP. It also lists some white papers and technical documents/documentation that has covered the subject in general and the specific integration between SAP and CRM using SCRIBE.

The document does not include any reference to the client, the exact project specification or any information that could be confidential. It’s just simple facts and findings on SCRIBE and the possibility of using it for SAP and CRM integration.

*****  Updated 18/02/2010:

The document is now available on Scribe Insight blog as a guest blog post: http://blog.scribesoft.com/2010/02/guest-post-crmsap-integration-using-scribe.html  .. 

I can still send you the document if you want, just request it via a comment below please.

****** Mohamed Ibrahim Mostafa

I thought about a list of general questions that most (if not all) consultants working on any integration solutions will need to have complete answers for before starting the design phase, let alone the development phase of the project.

In my opinion, the list of questions are as follows (not in real order – just a braindump!):

• How many environments do you have? Development, Test and Live? (recommended) or is the project is still in development so you can use live environment for development? Where will be the test environment later on?
• Is this a direct integration or in-direct integration? Is this an instant, event driven integration or a periodic scheduled integration between two systems? Are their queues for data to be migrated?
• What backup and restore operations can you do? The ability to Backup and restore data is vital.
• What integration application or tool are you going to use or is available? SCRIBE, SQL Server Integration Services, Web Services (Microsoft .NET Web Services), console applications, plugins? What SDK will you need? CRM SDK and CRM API for example?
• The Environment structure: How many physical servers? Where are these servers located? Where is the integration tool or application installed?
• How and When can I get access to the environment? Access to all servers is required including access to all databases and to all applications. For example: Access to Microsoft Dynamics CRM application (via webclient) is essential to confirm that data imported to a CRM has been migrated successfully.
• What type and format of extracts and data imports? CSV (Comma separated Values), XML, i-Doc, sql flat files, batch files, etc…
• Where will the extracts be imported? directly using the tool or via an FTP server? Is an SFTP server required?
• Are their duplicates? If so where, and what classifies a duplicate?
• Are there data entry standards for each application in the overall integrated system?
• Are there fields that are required in each system part of this integration?
• Are there fields that aren’t used?
• Are there any fields with null values?
• What relationship does the data has? are there fields which are dependant on others?
• What are the primary and forign keys of all tables in each system that will be part of the integrated system? Any field that does not allow null, business required (and preferably business recommended) must have a data upon migration (Defauls can be used then).
• Overall high level mapping between the different systems.
• What is the value, length, and format of fields/columns in the source system? What is the corresponding value, length and format in the target system?
• Are there any Pick Lists? A cross reference is required to map source and target values.
• What Data validation is required and is acceptable by the client and the project stakeholders?
• Differencing: What are the business rules for differencing? What data does not need to be updated and when? what data is needed to be updated based on the business requirements?
• using Default values for all required fields and columns in the target system to avoid causing any errors.

This is the list I have thought of so far. I will keep on updating this list as and when I think of something important that needs to be considered.

Let me know if you have any comments or feedback on these questions and tell me whether or not are these questions helpful.

Thanks for reading.