Cloud Storage

Cloud storage connectors are native cloud storage endpoints (Remote Cloud Storage Endpoint) built into UDMG, allowing file transfers to and from major cloud providers. Cloud storage connectivity in UDMG is powered by Rclone, a widely adopted open-source library for managing files across cloud storage platforms. UDMG supports integration with the following remote cloud storage providers:

• Amazon (AWS) S3
• Google Cloud Storage (GCS)
• Microsoft Azure Blob Storage
• Microsoft Azure File Storage

Push Scenario

This configuration scenario involves a Local Filesystem to Remote Cloud Storage Pipeline, where UDMG sends files from the local filesystem to an external remote cloud storage provider.

In this configuration scenario:

1. UDMG connects to an remote cloud storage provider (defined in a Remote Cloud Storage Endpoint) as a client.
2. UDMG uploads (pushes) files from the filesystem on the local UDMG server (defined in a Local Filesystem Endpoint) and saves them to the remote cloud storage provider.

Pull Scenario

This configuration scenario involves a Remote Cloud Storage to Local Filesystem Pipeline, where UDMG

In this configuration scenario:

1. UDMG connects to an remote cloud storage provider (defined in a Remote Cloud Storage Endpoint) as a client.
2. UDMG downloads (pulls) files from the remote cloud storage provider and saves them to a filesystem on the local UDMG server (defined in a Local Filesystem Endpoint).

UDMG as Gateway

In this configuration, UDMG acts as a multi-protocol gateway to stream files between your partners and your remote cloud storage providers. UDMG supports transferring files received from all three local server Endpoints — Local SFTP Server, Local FTP Server, and Local HTTPS Server (WTC).

Pipeline Name	Source Endpoint	Destination Endpoint	Description	Transfer Triggered By
Local FTP(S) Server to Remote Cloud Storage	Local FTP Server	Remote Cloud Storage	UDMG receives files via FTP(S) and transfers them to a remote cloud storage provider.	Partner's FTP(S) client
Local HTTPS Server to Remote Cloud Storage	Local HTTPS Server (WTC)	Remote Cloud Storage	UDMG receives files via the Web Transfer Client (WTC) and transfers them to a remote cloud storage provider.	Partner actions (upload/download) performed through the WTC
Local SFTP Server to Remote Cloud Storage	Local SFTP Server	Remote Cloud Storage	UDMG receives files via SFTP and transfers them to a remote cloud storage provider.	Partner's SFTP client

Local FTP to Cloud

In this configuration:

1. UDMG hosts and operates the Local FTP Server Endpoint.
2. External partners connect to UDMG using an FTP or FTPS client.
3. Partners manage files and folders.
4. Files are streamed and saved to a remote cloud storage provider (defined in the Remote Cloud Storage Endpoint).

Local HTTPS WTC to Cloud

In this configuration:

1. UDMG hosts a Local HTTPS Server (WTC) Endpoint that exposes the WTC interface.
2. Partners connect to the WTC, typically using a URL such as: https://<fqdn>:<port>/workspace/<domain>
3. The WTC authenticates partner Accounts using credentials based on their login method.
4. After authentication, partners can browse, upload, download, share, and manage files within the directories defined in the assigned Pipelines. Files are uploaded and downloaded from the remote cloud storage provider.
5. The HTTPS listener handles all TLS encryption, ensuring secure communication between the partner's browser and the UDMG Server.

Local SFTP to Cloud

In this configuration:

1. UDMG hosts and operates the Local SFTP Server Endpoint.
2. External partners connect to UDMG using an SFTP client.
3. Partners manage files and folders.
4. Files are streamed and saved to a remote cloud storage provider (defined in the Remote Cloud Storage Endpoint).

High-Level Architecture

Streaming Transfers

Remote Cloud Storage Endpoints use a streaming transfer model by default, where data flows directly from the source to the destination through memory buffers without local staging. Benefits include:

• Performance: Single-pass transfer is faster, especially for large files.
• Disk Space: No local staging required — only memory buffers are needed.
• Simplicity: No tmpPath configuration is needed for most transfers.
• Scalability: Reduces disk I/O, making it better suited for high-volume transfer servers.
• Cost: No additional disk provisioning is required for staging.

Configurable Buffer Size

The streaming buffer size can be configured at the Configuration File.

Scenario	Buffer Size
Default	16 MB
Memory-constrained	4 MB
High-bandwidth network	32–64 MB
Very large files (>10 GB)	64 MB

Azure Files Exception

Unlike AWS S3, Microsoft Azure Blob Storage, and GCS, which support streaming writes (uploads) where the file size is unknown upfront, Microsoft Azure File Storage requires the file size to be known before writing (uploading) begins. Attempting to stream directly to Microsoft Azure File Storage results in 416 InvalidRange errors, because the Azure backend must pre-allocate the file before writing byte ranges.

To work around this limitation, UDMG Server uploads transfers use a local temp folder strategy. All incoming data is first buffered to a local temporary file. Once the write (upload) is complete and the file size is known, the file is uploaded to Microsoft Azure File Storage in a single operation. During this process, the Transfer status is changed to "Uploading".

The temp folder location is:

{WorkDir}/.udmg-internal-tmp/azure-files/

Directory Markers

UDMG sets directory-markers to true by default, ensuring cloud transfers behave like a filesystem where folders are created and persisted. To disable this and align behavior more closely with the native remote cloud storage provider, set this field to false in your Configuration JSON at the Remote Cloud Storage Endpoint.

"directory_markers": true,

info

This setting applies only to AWS S3, GCS, and Microsoft Azure Blob Storage.

For detailed information, see Rclone documentation.

Unnamed Folders

UDMG requires all cloud transfer paths to follow a valid operating system path structure. Unnamed folders, which some cloud storage providers permit, are not supported. If an unnamed folder is created, delete it directly from your cloud provider and review your configured paths at the Endpoint, Pipeline, and Transfers Scheduled API levels.

Invalid paths for the Cloud Storage Base Path field or the full path (Cloud Storage Base Path + Relative or Virtual Path + destination API) values are:

• /dir
• /
• /dir
• /dir1/
• /dir//1
• //dir/

For detailed examples of valid paths per Pipeline type, see the specific Pipeline page.

For detailed Rclone information, see the relevant forum.

For detailed cloud provider bucket or container and directory naming standards, see below:

• AWS S3
• GCS
• Microsoft Azure Blob Storage
• Microsoft Azure File Storage

Checksum

UDMG supports MD5 checksum verification for cloud transfers using the native metadata provided by the cloud provider, where available. This avoids the overhead of downloading files solely to compute a checksum locally.

Checksum availability depends on the cloud provider, the cloud service, and how the file was uploaded:

Provider	Native MD5 (Checksum)	Info
AWS S3	ETag	MD5 only for single-part uploads; empty or invalid for multipart
Azure Blob	Content-MD5	Always available
Azure File Storage	Content-MD5	Only if set on upload
GCS	MD5 (metadata)	Always available

info

• AWS S3: The ETag is not a reliable MD5 checksum for multipart uploads. Checksum verification is unavailable in these cases. UDMG does not download and recalculate the MD5 locally, as this would be prohibitively expensive for large remote files.
• Azure File Storage: MD5 is available in most cases. However, for large chunked uploads, MD5 is only included if the source file already has an MD5 checksum. This is always the case for Local Filesystem to Remote Cloud Storage Pipeline (Azure File Storage), but may not apply in all scenarios.
• See Rclone documentation for additional details.

Permissions

For cloud transfers, file operation permissions are enforced based on the Pipeline type.

• UDMG as a Gateway: When UDMG initiates a transfer as a Gateway, permissions are enforced at the Pipeline level before any operation reaches the underlying storage. UDMG checks the configured permissions first. If the operation is not permitted, it is rejected immediately and the request never reaches the cloud provider. If the operation is permitted by UDMG but restricted by the cloud provider's access controls (e.g., IAM policy, bucket ACL), the cloud provider will reject it and the partner will receive a Transfer error.

  The Pipeline permissions govern the following operations: Read, Write, Delete, Mkdir, Rmdir, Overwrite, Rename, List, and Share.

• UDMG as a Client: When UDMG initiates a transfer as a Client, Pipeline permissions do not apply. For this Pipeline Type, access is governed entirely by:

  • The operating system permissions of the directory UDMG uses on the local side.
  • The cloud provider's access controls (IAM roles, access keys, bucket policies, etc.) on the remote side.