Upgrading an Enterprise Managed File Transfer System

The following document describes the requirements gathering and infrastructure design process that I did to meet my customer's needs.  All diagrams and requirements documents were created by me.

The Integration Services Group (ISG), maintained a sprawling Managed File Transfer system called Interchange that they had developed in-house over the course of five years.  All interdepartmental system to system (S2S) file transfers passed through Interchange though the majority of transfers were handed off to IBM MQ FTE.  SFTP was provided to external customers who couldn’t connect over MQ FTE.  Ad hoc transfers between employees (known internally as P2P transfers) was handled by a custom written website accessible through any web browser. Transfers from systems to persons (S2P) and from persons to systems (P2S) were also supported. (Collectively, the infrastructure to support P2P, S2P, and P2S transfers was known as P* or P-Star.) Customer endpoint integration was handled by a custom written agent called the ICA.  Management of the core ICAs, support infrastructure for the ICAs and the SFTP infrastructure made their MFT product complex and fragile.

By having the ICA on almost every customer’s servers including ISG's own core servers, they endured huge maintenance headaches and considerable latency in delivering new transfer routes to customers.

In initial meetings with ISG, they indicated that they wished to further separate the infrastructure for their S2S transfers from their P2P transfers and eliminate the SFTP transfers and the ICA.  They also indicated that costs associated with their on-site S2S and P2P servers were unacceptably high and they were open to cloud-based alternatives.  Further discussions resulted in a list of approximately 200 requirements covering security, deployments, maintenance capabilities, error recovery and others.  Given their success with MQ FTE, ISG indicated that they wished to stay with MQ FTE for all their internal S2S transfers and extend MQ FTE to their external customers.

The following diagrams describe the first revision of the proposed P* and S2S architecture.  Note that file transfer flow goes from left to right. 

Figure 1 describes the initial plan for separating P* from S2S.

Figure 2 describes a refined proposal after talking to various P* vendors about their capabilities.  Common capabilities between P* vendors were incorporated into this revision.  Prior to this version, ISG had not considered the possibility of a cloud client that a customer could install themselves.  Discussions with ISG indicated that some internal P2S and S2P customers may not be comfortable with installing a new file transfer client on their servers, so an FTE-Cloud bridge server was introduced to allow those clients continued S2P and P2S transfers.  Discussions with the P2P cloud vendors indicated that their cloud client had post-transfer steps to ensure that files were cleanly handed off between the cloud client and MQ FTE.  ISG wanted to minimize as much as possible the number of folder watches in the new system as it was a significant cause of headaches under the old system.

IPT stands for Internet Pass Through.  It’s an IBM product that permits MQ messages to traverse a firewall.  The lack of IPT is what previously prevented ISG from offering MQ FTE transfer services to external customers.

Partway through the requirements process, ISG changed their service delivery model.  They realized that they were making a great deal of work for themselves by being in the integration business and wanted out.  Even with the elimination of the ICA and SFTP transfers, ISG was still on the hook for going to each customers location and figuring out such mundane details as file paths, file system permissions, OS users, OS user permissions, FTE agent updates, compounded with non-technical customers.  Obviously, all that configuration work slows down ISG and their customers.

To decrease this load and provide bright line boundaries of responsibility with their customers, ISG decided to move to a service model akin to the United States Postal Service. Under the “Mailbox Model”, ISG customers would come to a standardized integration point housed on ISG’s servers to drop off and retrieve their files instead of dropping off files on their own servers.  The distinction is subtle but significant.

Using ultralight Docker containers combined with a distributed file storage system were proposed as a way to support rapid, nay, automatic provisioning of new mailboxes. The Mailbox Model would move ISG from the integration business to the storage business.

Without reference to possible implementations using Docker, Figure 3 describes the general pattern for file flow under the Mailbox Model.  Clients connect over whatever protocol they have chosen and deposit their files.  After dropping a trigger file to signal they are finished, the MFT agent then transfers the files to the destination agent.  If the source and destination message stores happen to reside on the same Docker host along with the MFT server, then all transfers are basically in-memory and extremely fast.

ISG envisioned that customers would go to a website, authenticate themselves, specify the endpoint protocols they wanted, click “Build It” and in a few minutes have IPs, ports and credentials for their transfers. With this website in mind, optimizing transfers between message stores becomes easy since the source and destination Message Stores could be collocated on the same Docker host. Figure 4 roughly describes how this kind of optimization might look.

The Protocol Adapter described in Figures 3, 4 & 5 is the integration point to the customer and represents the ability to “speak” whatever protocol ISG supports and the customer chooses.  Because they can be automatically provisioned, ISG was comfortable with reintroducing SFTP support along with NFS, SMB, or FTE. 

A sister group to ISG would have implemented the Docker cloud and the above proposals would be adapted the infrastructure that they could provide.

Unaddressed Issues

These diagrams do not address how the P* cloud would integrate with the Mailbox Model Message Stores, only that they should.