Transfer extracted data and checking

As mentioned in previous post, pushing or the publish/subscribe model are preferred. Todays topic is how to transfer the extracted files.

It is straight forward when we are using publish/subscribe model when that platform is already available. It is preferred because such platform usually supports our transferring requirement. For example,

• easy resending the files
• reliable transfer status (check for the completion of the transfer)
• secured

Or, we may implement our own file transferring solution. One of the simplest way is using sftp and implement ourselves some status checking. For example

• send paired with a control count file; or
• send using temporary file name and rename after transferring completed; or
• compress the file before sending (corrupted file means sending doesn't complete)