File Caching Job

Overview

A File Caching Job enables fast access to data stored on a central Primary Storage device using locally deployed Caching Gateways.

Primary Storage devices store the golden copy of the data synchronized within this Job. These are centrally deployed on-premises servers, or cloud storage solutions. Caching Gateways are located closer to the edge (for example, at each branch office) and accessed locally over SMB or NFS. You can arrange your Caching Gateway Agents into Gateway Scaleout groups to improve data availability. If one of the Agents in a Gateway Scaleout group downloads a file, other Agents in that group will automatically fetch the file so that it's immediately available to users in other locations.

Prerequisites

Only Windows and Linux endpoints can participate in Jobs as Caching Gateways. In addition to requirements listed in the System requirements topic, running Caching Gateway on Windows requires:

Windows 10 20H2 (OS build 19042) or newer.
Windows Server 2022 (21H2) or newer with desktop experience (from release channel LTSC)
Tip : To determine if your Windows operating system is capable of serving as a Caching Gateway in a File Caching Job, check if the cldapi.dll file is present in the C:\Windows\System32\ folder.

For Linux-based Caching Gateways, make sure the system has been set up accordingly. For details, see Configuring Linux OS cache servers.

Create a File Caching Job

To create a File Caching Job:

In the Management Console , from the top menu, select Jobs.
Select Configure Jobs , then click + CREATE NEW JOB.
Select File Caching , then click Next.
On the DETAILS tab:
1. Enter the name.
2. Enter the optional description that will help understand the purpose of this Job, then click Next.
On the SETTINGS tab:
1. Select Job’s priority.
  Note : Job priority determines the order in which the Agent handles its data transfers queue. High priority Jobs' transfers are handled first. While the lower priority Jobs are waiting for their transfer window, the Agent is performing non-transfer related tasks such as indexing files or merging the folder tree. In case of multiple Jobs with the same priority, their data transfers are performed simultaneously. For more information, see Job priority.
2. (Optional) Enable File Locking and adjust its parameters:
  - File lock timeout - Locked file idle time after which the lock server releases a locked file.
  - Locked files update interval - Time interval between locked file's status check.
    Note : For optimal performance, we recommend leaving the default value unchanged.
  - When Agent is not connected to Lock server - Access level to shared files when the locking server is unavailable.
  - Ignore locks for these files and folders - A regular expressions (PCRE2 syntax) to match files and folders that you want to exclude from locking. Add each regular expression in a separate line.
    1. In the User groups section, click + ADD USER GROUP , then select a group. Click Add and choose access level to the Job:
  - View only - Permission to view the Job on the Jobs list and access its status information.
  - Run - Permission to initiate a Job run (applicable to Distribution, Consolidation and Script Jobs).
  - Edit & Run - Permission to edit the Job's configuration and initiate Job runs. Users with Edit & Run permission cannot delete a Job.
  - Full Access - Permission to edit and delete a Job as well as initiate Job runs.
3. Click Next.
On the PROFILE tab, select Job settings - a pre-defined set of parameters that fine-tunes the Job run, then click Next.
Tip : Select Unique for this Job to customize profile settings specifically for the Job being set up.
On the PRIMARY STORAGE tab, set up your Primary Storage devices:
1. Select individual Agents, or Agent groups, that will participate as Primary Storage devices.
  Note : macOS devices must run Resilio Agent 4.2 or newer to participate in the File Caching Job as Primary Storage devices.
  Tip : Click + CREATE NEW GROUP to define a new Agents group. For details, see Groups tab and adding new groups.
2. Select options:
  - Read-only - Agent will be only allowed to download the files. Any files copied or added to the synchronization folder manually won't be propagated to other Agents.
    Note : There must be at least one non-read-only Primary Storage Agent.
  - Priority Agent - Priority Agents store a full copy of files in the synchronized folder. This ensures that when there are Agents with the Transparent Selective Sync (TSS) enabled (in File Caching, TSS is enabled by default), there's a full copy of the data available. If a TSS-enabled Agent wants to offload some of the files, it will be allowed to do so only when the upload to one of the Priority Agents has been completed. While all Agents comprise a mesh network and exchange the data with each other, data transfers to Priority Agents are prioritized.
    Note : There must be at least one Priority Agent selected.
  - Lock server - Agent will be responsible for keeping track of locked files. For details, see File locking in Resilio Active Everywhere Jobs.
3. Click Specify path to select path or click the suggested path to modify it.
4. Select storage location option:
  - Direct path - Provide a direct, absolute path to the synchronization folder.
  - Storage connector - Select a previously defined cloud storage connector or add a new one. For details, see Setting up cloud storage and configuring Jobs.
  - %FOLDERS_STORAGE% - A special path macro pointing to agent's default storage folder location.
  - %DOWNLOADS% - A path macro pointing to downloads folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
  - %HOME% - A path macro pointing to the home folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
  - %USERPROFILE% - A path macro pointing to user profile folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
5. (Optional) Click Browse to browse the file system and navigate to location where you want to store the synchronization folder for this Job. Select an existing folder or click + CREATE FOLDER to create a new one, then click Select.
  Note : For Agents with the limited user option enabled, file system browsing is limited to the default storage folder path. For details, see Limiting admin’s access to the Agent.
6. Click Save, then click Next.
On the CACHING GATEWAY tab, configure local caching instances:
1. Select individual agents, or agent groups, that will serve as an on-site cache.
  Important : Before you select a Linux-based system, make sure it has been set up accordingly. For details, see Configuring Linux OS cache servers.
  Note: macOS devices cannot participate in File Caching Jobs as Caching Gateways.
  Tip : Click + CREATE NEW GROUP to arrange Agents in a Gateway Scaleout group. The files, requested and cached on one of the Agents in the Scaleout group, will be automatically downloaded by other Agents within such a group. If a requested file is not available on a cache server, it's downloaded from Primary Storage device. For details on combining Agents in groups, see Groups tab and adding new groups.
2. Select options:
  - Read-only - Agent will be only allowed to download the files. Any files copied or added to the synchronization folder manually won't be propagated to other Agents.
  - File policy - A set of rules that manage data hydration. For details, see File policy.
3. Click Specify path to select path or click the suggested path to modify it.
4. Select storage location option:
  - Direct path - Provide a direct, absolute path to the synchronization folder.
  - %FOLDERS_STORAGE% - A special path macro pointing to agent's default storage folder location.
  - %DOWNLOADS% - A path macro pointing to downloads folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
  - %HOME% - A path macro pointing to the home folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
  - %USERPROFILE% - A path macro pointing to user profile folder location. The path that the macro is resolved to depends on the operating system the Agent is running on. For details, see Path macros.
    Important : For Linux-based servers, you must provide Linux and Linux cache paths. The former is the virtual path over which the data is exposed for NFS and SMB access, while the latter is where the files are actually stored.
5. (Optional) Click Browse to browse the file system and navigate to location where you want to store the synchronization folder for this Job. Select an existing folder or click + CREATE FOLDER to create a new one, then click Select.
  Note : For Agents with the limited user option enabled, file system browsing is limited to the default storage folder path. For details, see Limiting admin’s access to the Agent.
6. Click Save , then click Next.
On the REFERENCE AGENT tab, select a Reference Agent, then click Next.
Note : Reference Agent is mandatory for Jobs configured to synchronize NTFS or POSIX permissions. In other cases, selecting a Reference Agent isn't necessary, but it can speed up the initial merging of contents of non-empty synchronization folders. Read-only Job participants cannot be Reference Agents, thus they're not available for selection. For more information, see Reference Agent.
On the NOTIFICATIONS tab:
1. Click + ADD NOTIFICATIONS.
2. From the Trigger drop-down list, select an event that will trigger notification.
3. (Optional) Click Skip notice if Jobs did not transfer files to disable notification in case there were no files transferred during the scheduled Job run.
4. Click + ADD RECIPIENT and select one of the options:
  - Add me - Add the currently logged in user.
  - Add user - Add another Resilio user.
  - Add e-mail - Add an email address of the recipient.
  - Add webhook - Add a predefined webhook URL.
5. Click Save , then click Next.
On the SUMMARY tab, verify your Job's configuration, then click Save.

Monitor Cache Utilization

To monitor your cache utilization in a File Caching Job:

In the Management Console , from the top menu, select Jobs.
On the JOB RUNS list, locate and click the Job run you want to monitor.
Select the SERVERS tab, then locate and click the Caching Gateways.
On the OVERVIEW tab, locate the CACHE USAGE section for information on cache utilization.

Max cache size - The maximum size of the cache allowed as configured in the File policy.
Size / Number of files in cache - Size and the number of cached files.
Note

The size includes:
- Files fetched in the cache (pre-hydrated, pinned), either by File policy or manually by users.
- Files that were added manually by users, regardless of what file policy rule - Pinning or Hydration - they match, if any at all.
- Files in the Archive folder, managed by the Agent.
The size does not include:
- Files excluded from cache, either originated on remote Caching Gateways or Primary Storage devices, or added manually by users to the cache folder.
- Partly downloaded files in the .sync folder.
Size / Number of hydrated files - Files that are pre-hydrated in cache by the Hydration rule in the File policy. Files that were hydrated in cache manually (i.e. not through a File policy rule) are not counted here.
Size / Number of pinned files - Files that are pre-hydrated in cache by the pinning rule in the File policy. Files that were hydrated in cache manually (i.e. not through a File policy rule) are not counted here.
Percent of pinned files out of total - Pinned files (by File policy) relative to the maximum file cache. Files that were hydrated in cache manually (i.e. not through a File policy rule) are not counted here.
Size / Number of excluded files - Files that were added to the cache locally by the users, but are excluded from cache by the Exclusion rule in the File policy or Job's Ignore list. This parameter does not calculate the excluded files, present on a Primary storage or other Cache servers and that don't show up in cache.
Size / Number of re-downloaded files - F iles that were cleared from cache but then re-downloaded manually during last 7 days. Seeing large numbers here may indicate that cache file policy is misconfigured and frequently accesses flies are cleared from cache unnecessarily.
Max day of files in cache - This is the time during which a file is kept in the cache before it's cleared from it. This time is tracked by the oldest access time of the file that is cleared from cache. Pinned and excluded files are not taken into consideration when calculating this parameter.
Last time cache was filled up - The number of days since the cache was filled up the last time, which triggered automatic files clearing procedure.
Note : This doesn't indicate whether the file clearing procedure was successful or not.

Info

Zero-sized files are calculated differently on Windows and Linux, resulting in discrepancy in the file size reported in cache server details.
Windows servers don't include the zero-sized files in file counters in the job run details, until they are populated in cache by opening their location in a file browser. Until then these zero-sized files will be pending in Files queue for this server.
Linux servers calculate them in the cache details right away.

Peculiarities and limitations

End-user Agent can also manage the files using the TSS context menu. However, they cannot dehydrate the files that are pinned by the File policy.
End-user Agents may report invalid files counter values, if there are files that have been dehydrated manually rather than by the File policy rule.
Files on the Primary Storage, which are ignored by the File and folder name filter custom parameter in the Job Profile, may appear as in progress on the caching server.