biztalk archiving folders

BizTalk Server Best practices, Tips, and Tricks: #6 Maintaining Archiving Folders

Welcome once again to another BizTalk Server Best practices, Tips, and Tricks blog post! In my previous blog post, I talked about some essential tips and tricks for BizTalk Server administrators:

And for BizTalk Server Developers:

Today I’m going to speak about another critical Best practice, Tips and Tricks for BizTalk Server administrators and developers: Maintaining Archiving Folders

biztalk archiving folders

This is a task that is the responsibility of administrators as well as developers! For sure, administrators need to implement some monitoring and cleaning operations, but developers need to report and help administrators identify all of these folders.

#6 Maintaining Archiving Folders

It is very common in integration scenarios to see messages being archived locally into a hard drive – either by using a pipeline component like:

or by simply using the default functionalities in BizTalk Server, like Send Port Filters:

This can happen for several reasons, for example:

  • The messages need to be archived for legal reasons (normally (recommended) we use an “external” repository – not a local hard drive in the server)
  • Because it is practical and valuable to have the files so they can easily be reprocessed in the case of failures
  • Or because we are troubleshooting an issue and want an easy way to see the message that is arriving or leaving, for example.

The common problem with some of these approaches is that we usually tend to implement the archiving mechanism and we forget to implement a monitoring and cleaning mechanism. We end up with several problems after a while:

  • The most critical –> disk full, causing processes to fail or other problems that are even more critical
  • Having many files in a folder will also cause performance problems when we are trying to perform operations on it. The lookup time of a directory increases proportionally to the square of the number of entries, and performance seems to drop between 1000 and 3000 files – the importance is not the exact number itself but knowing that a high number may cause problems and performance degradation while trying to write new files on the archive folder.

It is very easy to understand that if you have an integration process that processes hundreds of messages, you can quickly get this kind of problem.

Needless to say that it is extremely important that you implement monitoring and maintenance tasks from day one… or once again, you will completely forget until it is too late.

So what are your options?

If you have monitoring tools like BizTalk360, you can take advantage of the existing monitoring capabilities that this tool will provide to you. One of that is the capability to monitor local or remote folders like share folders, FTP or SFTP. You can check more about this feature here: Introduction to File Locations Monitoring in BizTalk360, but in resume, you can set up a local folder – in this case, we will use a local folder – and configure a Thresholds rule with the metric File Count to monitor like:

  • If the File Count is greater than 1000, raise an alert. If you receive such an alert, you can even trigger a  PowerShell script to do the cleanup. Read this blog to understand how this works.

You can also use a separate PowerShell script, and here we basically can do everything we want. We can monitor any folder we want and notify the teams if we want. But most importantly, we can implement cleaning tasks or maintenance tasks to avoid performance problems or prevent the disks from being full. To run this script, you could consider Automated Tasks in BizTalk360. 

For example, this script:

4
5
6
7
8
9
10
11
12
13
14
15
16
17
foreach ($directory in $Directories)
{
    $zipFile = "E:\BackupFileWarehouse\" + $directory.FullName.Substring($directory.FullName.LastIndexOf(":")+2).Replace("\","_") + "_" +  (get-date -f yyyyMMdd_hhss).ToString() + ".zip"
 
    $filelist = get-childitem $directory.FullName |
                                where-object {$_.LastWriteTime -le (get-date).AddHours(-$Hours)} |
                                where-object {-not $_.PSIsContainer}
 
    if($filelist.Count -gt 0)
    {
        $filelist | format-table -hideTableHeaders FullName | out-file -encoding utf8 -filepath lastmonthsfiles.txt
                    & 'C:\Program Files (x86)\7-Zip\7z.exe' a $zipFile `@lastmonthsfiles.txt
 
        $filelist | %{Remove-Item $_.FullName }
        rm lastmonthsfiles.txt
    }
}

THIS POWERSHELL IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND.

Allows you to:

  • Define the local folders you want to monitor
  • Define the number of hours, minutes, or days that you want to archive the files, leaving only the recent ones in the folder.
    • If the creation time of the file is older than this parameter, then it will be archived and deleted from the folder.
  • For each folder, it will check if files exist to be archived
    • If so, it will create a zip file, using the 7-zip tool, in a different location that you can define.
    • Otherwise, it will not create any archiving file.

Of course, if this script will not move/create the archived files in a shared location or something similar, then you will need an additional mechanism to maintain and delete these zip files. This is just an example of how simple we can do these tasks.

You can download Manage messages being archived locally into the hard drive with PowerShell from GitHub here: Manage messages being archived locally into the hard drive with PowerShell