HomeResourcesField Guides › PowerScale Data Protection & SmartFail

Field Guide · Storage Operations

Dell PowerScale Data Protection: SmartPools Settings, Requested Protection, and What SmartFail Actually Does

Local data protection on PowerScale is decided by three things that are usually configured independently and almost never reconciled: the global SmartPools settings, the requested protection policy on the node pool, and the number of nodes you actually have. This guide walks each setting, gives the node-count arithmetic that governs which protection levels are even possible, and shows what happens when you smartfail a node on a cluster that cannot afford to lose one.

Who this is for: storage engineers · NAS administrators · infrastructure teams · Dell PowerScale administrators

Validated against OneFS 8.2.x and 9.x · field labels shift between releases · Reviewed by WUC Storage Infrastructure Engineering · Last validated: 2026-07-14

FocusLocal protection · FEC & mirroring · SmartFail
PlatformsPowerScale F/H/A series · Isilon X/NL/S
OneFS8.2.x and 9.x
InterfacesWebUI + CLI, both shown
Read time16 min

!!Command safety model

Protection changes are not configuration edits. They are data movements. Raising a node pool's requested protection queues a cluster-wide restripe that runs for hours, consumes back-end bandwidth, and increases space consumption. Lowering it reduces resilience the moment it takes effect. SmartFail evacuates an entire node and is irreversible once data movement is underway. Read the badge before you press Enter.
BadgeMeaning
Read-onlyInspects state. Safe in production, any time.
DisruptiveTriggers a restripe or interrupts I/O. Recoverable. Change window.
DestructiveRemoves a node or reduces resilience. Hard or impossible to reverse.

ENVReference cluster

Every output below comes from a 3-node PowerScale cluster (boston), all-flash F-series, replicating via SyncIQ to a 3-node DR cluster (phoenix). Three is not an arbitrary choice — it is the minimum supported cluster size in OneFS, and that single fact drives almost everything on this page. If your cluster is larger, the arithmetic changes but the method does not.

Why the node count is the first thing to establish. Operators reach for a protection level based on how important the data feels. OneFS does not work that way. The node count is a hard ceiling on which levels are achievable, and OneFS will accept a level it cannot deliver rather than refuse it. Establish the node count before you touch the protection policy — see family 02.
boston-1# isi status
Cluster Name: boston
Cluster Health:     [  OK ]
                   Health  Throughput (bps)   SSD Storage
ID |IP Address     |DASR |  In    Out  Total| Used / Size
---+---------------+-----+------+-----+-----+-----------------
  1|10.10.10.11    | OK  |  142M| 388M| 530M|   57T/  82T( 69%)
  2|10.10.10.12    | OK  |  138M| 401M| 539M|   57T/  82T( 69%)
  3|10.10.10.13    | OK  |  151M| 377M| 528M|   57T/  82T( 70%)
---+---------------+-----+------+-----+-----+-----------------
Cluster Totals:          |  431M|1.16G|1.59G|  171T/ 246T( 69%)

01SmartPools global settings, line by line

Where these live: File systemStorage pools → the SmartPools settings tab — the path is highlighted across the top of the panel below. On the CLI the same values sit behind isi storagepool settings view.

They are cluster-wide defaults. They do not override an explicit file pool policy, but they decide what happens to everything a policy does not explicitly claim — which, on most clusters, is the majority of the data.

Dashboard ▾ Cluster management ▾ File system ▾ Data protection ▾ Access ▾ Protocols ▾ Storage pools Summary File pool policies SmartPools CloudPools SmartPools settings CloudPools settings ① → ② → ③ File system → Storage pools → SmartPools settings Edit SmartPools settings Local storage settings Increase directory protection to a higher requested protection than its contents Enable global namespace acceleration greyed out — all-flash cluster, nothing to accelerate Use SSDs as L3 cache by default for new node pools Virtual hot spare (VHS) Subtract the space reserved for the virtual hot spare when calculating available free space Deny data writes to reserved disk space VHS space reserved At least 1 virtual drive(s) At least 0 % of total storage Enable global spillover Spillover data target anywhere Enable transfer limit Default transfer limit 90 % of pool capacity SmartPools action settings Manage protection settings Apply to files with manually-managed protection Manage I/O optimization settings Apply to files with manually-managed I/O optimization settings 1 · VHS is not stolen capacity It reserves the room FlexProtect needs to rebuild a failed drive. Both boxes on — otherwise the reservation is advisory and clients write into it. 2 · Spillover governs CLIENT WRITES When a pool is full, writes land elsewhere rather than failing — silently breaking your tiering intent. 3 · Transfer limit governs the JOB SmartPools stops moving files INTO a pool once it hits 90% full. Different guard, different actor from spillover. Do not confuse the two. OneFS 9.5+. 4 · Leave these two UNCHECKED Ticking them lets SmartPools overwrite every manual protection override on the cluster, on the next job run. Someone set those overrides deliberately. This erases the record of intent. Revert changes Save changes Reproduction of the OneFS SmartPools settings panel. Field labels vary by OneFS release; transfer limit is OneFS 9.5 and later.
The SmartPools settings panel, annotated. Four of these controls decide whether the cluster can rebuild after a drive failure, whether your tiering policy means anything, when SmartPools stops filling a tier, and whether someone’s deliberate protection overrides survive the next job run. Callouts 2 and 3 are the pair most often conflated — spillover governs what happens to client writes when a pool is full; the transfer limit governs when the SmartPools job itself stops adding to it. And the two unchecked boxes at the bottom remain the most consequential things on the screen.

isi storagepool settings view Read-only

Read this before changing anything. Every WebUI checkbox below maps to one line here.

boston-1# isi storagepool settings view
        Automatically Manage Protection: files_at_default
   Automatically Manage Io Optimization: files_at_default
   Protect Directories One Level Higher: Yes
          Global Namespace Acceleration: disabled
          Virtual Hot Spare Deny Writes: Yes
           Virtual Hot Spare Hide Spare: Yes
         Virtual Hot Spare Limit Drives: 1
        Virtual Hot Spare Limit Percent: 0
                      Spillover Enabled: Yes
                       Spillover Target: anywhere

Increase directory protection to a higher requested protection than its contents Read-only setting

Maps to Protect Directories One Level Higher. Leave it on. Directories are metadata; losing a directory does not lose one file, it orphans every file beneath it. Protecting the directory tree one level above its contents costs almost nothing — directories are tiny — and it is the difference between losing a file and losing the path to a million of them.

Enable global namespace acceleration (GNA) Read-only setting

Greyed out in the reference cluster, and that is expected. GNA stores metadata on SSDs across the cluster so that metadata reads do not have to touch spinning disk. It requires the cluster to hold a minimum ratio of SSD capacity, and it is meaningless on an all-flash cluster — every drive is already an SSD, so there is nothing to accelerate. If GNA is greyed out on an F-series cluster, that is not a fault. If it is greyed out on a hybrid H- or A-series cluster, you have not met the SSD ratio, and that is worth investigating.

Use SSDs as L3 cache by default for new node pools Read-only setting

On hybrid nodes, L3 turns the SSDs into a read cache in front of the HDDs rather than a usable storage tier. On all-flash nodes it is not applicable. The trap: if you enable L3 on a pool and later want those SSDs as a tier, you must disable L3 and let the pool re-form — you cannot have both.

Virtual hot spare (VHS) Affects usable capacity

VHS reserves free space so the cluster can re-protect data after a drive fails. This is the single most misunderstood setting on the page, because it looks like it is stealing capacity from you. It is not — it is reserving the room FlexProtect will need at the exact moment you can least afford not to have it.

  • Subtract the space reserved for the virtual hot spare when calculating available free space — makes the reservation honest. Free space reports drop by the VHS amount. Turn this on: you want your monitoring to see the capacity you can actually use, not the capacity that includes your safety margin.
  • Deny data writes to reserved disk space — enforces it. Without this, the reservation is advisory and clients will happily write into it. Turn this on. Reserved-but-writable is not reserved.
  • VHS space reserved: at least N virtual drives / at least N% of total storage — the reference cluster is set to 1 virtual drive and 0%. One virtual drive means "keep enough free space to rebuild one failed drive." On a large cluster, one drive's worth is thin; the percentage field exists for exactly that reason.
A cluster above 90% used cannot reliably re-protect itself. FlexProtect needs somewhere to put the rebuilt blocks. VHS is what guarantees that somewhere exists. Operators who turn VHS off to reclaim capacity on a full cluster have removed the safety margin at precisely the moment the cluster is most likely to need it — a full cluster is a stressed cluster, and a stressed cluster drops drives.

Enable global spillover · Spillover data target Affects placement

When a node pool fills up, spillover decides whether writes targeted at it may land anywhere else on the cluster instead of failing. Target anywhere means "any pool with room."

The trade-off is real and it is not obvious. Spillover on keeps writes succeeding but silently violates your tiering intent — data you carefully policied onto fast storage ends up wherever there was room, and you find out months later during a performance investigation. Spillover off makes the write fail loudly the moment the pool is full, which is operationally painful and diagnostically honest. On a single-pool cluster, as in the reference environment, the setting has nothing to spill to and is effectively inert — which is worth knowing before you spend an afternoon tuning it.

Enable transfer limit · default 90% Affects placement

Introduced in OneFS 9.5. The transfer limit is the point — expressed as a percentage of a pool's capacity — at which SmartPools stops moving more files into that pool. The default is 90%.

Before transfer limits existed, a file pool policy would keep packing data into its target tier until the tier was simply full, at which point spillover (or a failed write) took over. That is a bad way to discover you have run out of fast storage. The transfer limit turns a cliff into a guard rail: SmartPools declines to make the situation worse, and the pool retains the headroom it needs to restripe.

Transfer limit and spillover are not the same control, and conflating them is common. They govern different actors:
  • Transfer limit → constrains the SmartPools job. "Stop moving files into this pool once it is 90% full."
  • Spillover → constrains client writes. "When this pool is full, may a client's new write land somewhere else instead of failing?"
A pool can therefore be at 92% — above its transfer limit, so SmartPools has stopped feeding it — while clients continue writing directly into it, because spillover only engages at genuinely full. Leaving the transfer limit at the 90% default and understanding which of the two is firing is most of the diagnosis when a tier fills unexpectedly.
# The transfer limit surfaces per-pool, not just as a global default:
boston-1# isi storagepool nodepools view f600_82tb_384gb | grep -i transfer
   Transfer Limit: 90%
   Transfer Limit State: default

SmartPools action settings: Manage protection settings / Manage I/O optimization settings Affects existing files

Maps to Automatically Manage Protection and Automatically Manage Io Optimization, both set to files_at_default in the reference cluster.

The nested checkbox — "Apply to files with manually-managed protection" — is the one that matters, and it is unchecked for good reason. files_at_default means SmartPools manages only files that have never been given an explicit, individual protection setting. Ticking the nested box changes that to all: SmartPools will now overwrite every manual protection override anyone has ever set on any file, cluster-wide, on the next SmartPools job run.

Do not tick "Apply to files with manually-managed protection" to fix a protection problem. It does not fix a problem; it erases a record of intent. Somewhere on that cluster is a dataset that someone deliberately protected higher than the pool default — a compliance archive, a master image, an irreplaceable research set. This checkbox silently drops it to the pool default. If you genuinely need to reset manual overrides, find them first (isi get -DD on the paths in question), agree the change, and then act.

02Protection levels and the node-count rule

OneFS offers two families of local protection: FEC (forward error correction, erasure coding) and mirroring. FEC is space-efficient and computationally heavier. Mirroring is space-hungry and simple. Which ones you may use is not a matter of preference — it is arithmetic.

The rule that governs everything: an erasure-coded level of +Nn requires at least 2N+1 nodes. That is not a recommendation, it is a structural requirement — the cluster must retain quorum after losing N nodes. And OneFS will accept a protection level your node count cannot deliver without throwing an error. It records what you requested and then quietly delivers something lower as actual. A cluster can therefore believe it is protected at a level it has never once achieved.
ProtectionTypeMinimum nodesToleratesValid on 3-node boston?
+1nFEC31 drive or 1 nodeYes
+2d:1nFEC32 drives or 1 nodeYes
+3d:1nFEC33 drives or 1 nodeYes
+2nFEC5 (2×2+1)2 drives or 2 nodesNo — silently under-delivers
+3nFEC7 (2×3+1)3 drives or 3 nodesNo
+4nFEC9 (2×4+1)4 drives or 4 nodesNo
2xMirror21 nodeYes
3xmirrored over 3 nodesMirror32 nodesYes — exactly at the limit
4x8xMirror4 … 83 … 7 nodesNo

Note the last column on the 3x row. Three-way mirroring is achievable on a 3-node cluster — and it consumes the entire node count to do it. Every one of the three nodes holds one copy. There is no spare node. Hold that thought; it is the whole of section 04.

isi storagepool health -v Read-only

The command that exposes the requested-versus-actual gap. This is the single most important protection check on the cluster, and almost nobody runs it.

boston-1# isi storagepool health -v
Name              Health  Used    Size   Protection  Actual
--------------------------------------------------------------
f600_82tb_384gb   ATTN    69.4%   246T   +2n         +2d:1n   <-- MISMATCH
--------------------------------------------------------------
# +2n was requested. It needs 5 nodes. This cluster has 3.
# OneFS did not refuse the setting. It is delivering +2d:1n and
# reporting ATTN. Nothing in the WebUI shouts about this.

Always read the Actual column. Never trust Protection. The left column is what someone asked for. The right column is what your data has.

03Setting Requested Protection to 3x mirroring

There are two entirely different places to set protection, and choosing the wrong one is a common and expensive mistake.

ScopeWhereAffectsUse when
Node poolStorage pools → SmartPoolsEvery file in the pool with no explicit settingYou want a new cluster-wide default
Directory / fileFile system explorer → View/EditThat path only — a manual overrideOne dataset has an SLA the pool default does not meet

The task here is the second one. The SLA says the sales business unit's home directories must be mirrored across all three nodes. That is a requirement for one path, not for the whole cluster — so it is set on the directory, and it becomes a manual override.

Step 1 · Select the correct access zone Read-only

File system → File system explorer. Set Current access zone to sales before you go looking for the path.

The access zone selector is not cosmetic, and skipping it is the most common way to edit the wrong directory. Each access zone has its own base path and its own namespace view. If you leave the zone on System and navigate to something called home, you may well be looking at a completely different directory than the one the sales users actually mount. Set the zone first, confirm the path (/ifs/sales/home), and only then edit.
Dashboard ▾ Cluster management ▾ File system ▾ Data protection ▾ Access ▾ Protocols ▾ File system explorer Current access zone: sales Explorer File system explorer + Create directory Path: /ifs/sales Browse... Search Name Size Updated Actions sales 133 B 2026-07-13 09:46 View details sales_audit 0 B 2026-07-13 09:02 View / Edit sales_data 185 B 2026-07-13 10:34 View / Edit home 97 B 2026-07-13 09:00 View / Edit sales_doc 0 B 2026-07-13 15:39 View / Edit sales_gen 45 B 2026-07-13 09:05 View / Edit ① Set the access zone FIRST. Leave it on System and the “home” you edit may not be the one sales users mount. ② Then View / Edit on home → View Directory Property Details → Edit Properties → Protection settings. Reproduction of the OneFS File system explorer. Paths and timestamps are synthetic.
File system explorer, scoped to the sales access zone. The zone selector at the top is doing real work: it decides which namespace the path below is resolved against. Every directory in this listing exists within that zone. Confirm the path reads /ifs/sales before you touch View / Edit on home — a directory called home exists in almost every zone, and they are not the same directory.

Step 2 · Open the directory properties

Against the home directory, choose View / Edit, then Edit Properties in the View Directory Property Details window. Scroll to Protection settings.

Protection settings ! Manual configuration of protection settings prevents SmartPools from managing these settings in the future. Managed by SmartPools Data storage target anywhere SSD strategy Use SSDs for metadata read acceleration (Recommended) Requested protection [3x] Mirrored over 3 nodes I/O optimization settings Managed by SmartPools — left ON. Only protection is being overridden. Cancel Save changes Unchecking this CREATES a manual override OneFS says so in the banner above. Remember it — see the warning below. 3x = one copy per node Valid on a 3-node cluster — and it consumes all three. Lose one and 3x can no longer be met. See section 04. Reproduction of the OneFS directory Protection settings dialog (File system explorer → View/Edit → Edit Properties). Field labels vary by OneFS release.
The directory Protection settings dialog. Unchecking Managed by SmartPools is what turns this path into a manual override — OneFS tells you so in the orange banner, and most operators click straight past it. I/O optimization settings stays managed: only protection is being overridden, and overriding more than you need to is how clusters accumulate settings nobody can explain two years later.

Step 3 · Uncheck "Managed by SmartPools", set 3x, save

Set Requested protection to [3x] Mirrored over 3 nodes. Leave Data storage target at anywhere and SSD strategy at the recommended metadata-read-acceleration option unless you have a specific reason to change them. Leave I/O optimization settings → Managed by SmartPools checked — you are overriding protection, not I/O behaviour. Save changes, then Close.

isi set -p 3x -R /ifs/sales/home Disruptive

The CLI equivalent, and the one to use if you are doing this to more than one path. -R applies it recursively; without it you have protected the directory inode and nothing inside it.

# Baseline first — what is this path protected at TODAY, and is it already manual?
boston-1# isi get -D /ifs/sales/home | head -20

# Apply the override recursively
boston-1# isi set -p 3x -R /ifs/sales/home

# Verify. The protection policy should now read 3x, set manually.
boston-1# isi get -D /ifs/sales/home
POLICY   W   LEVEL   PERFORMANCE  COAL  FILE
3x       M   3x/3    concurrency  on    /ifs/sales/home
#        ^
#        M = Manually managed. This is the flag that matters.

# A restripe job will queue to lay down the third copy:
boston-1# isi job jobs list
You have just created exactly the thing that Setting 4 in section 01 destroys. This directory is now manually managed. If anyone later ticks “Apply to files with manually-managed protection” in SmartPools settings — a box that looks tidy, harmless, and like it makes SmartPools work “properly” — the next SmartPools job will overwrite this 3x mirroring back to the pool default. There is no alert. There is no confirmation naming the paths it is about to change. The sales home directories silently drop below the SLA you were told to enforce, and the first anyone will know is an audit or a data-loss event.

If you set manual overrides, record them. A manual override is an undocumented decision unless you write it down. Keep a list of the paths, the level, and the SLA that mandated it — and check it against isi get -D periodically, because the cluster will not tell you when it stops being true.

Contrast · changing the whole node pool instead Disruptive

If the requirement were cluster-wide rather than one directory, you would change the node pool default instead — no manual override, nothing for SmartPools to later overwrite.

boston-1# isi storagepool health -v                                    # baseline
boston-1# isi storagepool nodepools modify --name=f600_82tb_384gb --protection-policy=3x -v
boston-1# isi job jobs list                                            # SetProtectPlus queues
boston-1# isi storagepool health -v                                    # actual must equal requested
Understand the capacity cost before you commit, at either scope. 3x mirroring stores three complete copies — usable capacity collapses to roughly one third of raw. Applied to one home directory that is usually affordable. Applied to a node pool already at 69% used, it is a request to store three times the data the cluster currently holds, and it will not fit. A protection change that runs a pool to 100% mid-restripe is a self-inflicted outage.

04SmartFail, quorum, and why access survives

SmartFail is the graceful removal of a node or drive. It is not a failure — it is the opposite. OneFS quarantines the device, re-protects every block it holds onto the remaining devices, and only then releases the hardware. Clients keep working throughout. Understanding why they keep working is the whole point of this section, because the mechanism is quorum, and quorum is what tells you when they will stop.

The quorum rule Read-only concept

A cluster of N nodes needs ⌊N/2⌋ + 1 nodes online to accept writes. That is a simple majority. On the 3-node boston cluster, quorum is 2. Lose one node — to failure or to smartfail — and two remain. Two is a majority of three. Quorum holds, and the cluster stays read-write.

Lose a second node and you are down to one. One is not a majority of three. Quorum is lost, and OneFS places the file system into a protected read-only state: reads continue to be served from the data that is still available, but every write is denied. The cluster does not go dark — it goes read-only. That distinction matters enormously when you are on the phone explaining the impact.
Cluster sizeQuorum neededNodes you can lose and still WRITEMax concurrent smartfails
3 nodes211
4 nodes311
5 nodes322
7 nodes433
N nodes⌊N/2⌋+1N − (⌊N/2⌋+1)⌈N/2 − 1⌉

Note the last column. OneFS permits smartfailing up to ⌈N/2 − 1⌉ nodes concurrently, provided quorum is not broken. On a 3-node cluster that is exactly one node — so smartfailing node 3 is permitted, and it works. It is not blocked, and it should not be. This is the design, not a loophole.

Step 1 · Initiate the SmartFail Destructive

Cluster management → Hardware configuration → Nodes. On the target node, open More and choose Smartfail node.

Hardware configuration Nodes Drives Cluster Manage nodes + Add a node Bulk actions Node Serial number Status OneFS version Configuration Chassis Actions 1 FCH2401A001 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A001 View details More ▾ 2 FCH2401A002 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A002 View details More ▾ 3 FCH2401A003 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A003 View details More ▾ Smartfail node Stop smartfail on node Shut down node Reboot node View node partitions View node sensors “Stop smartfail on node” is right here. Your undo, while the evacuation is still running. Once the node is released, there is no undo. Reproduction of the OneFS Hardware configuration → Nodes tab. Serials and versions are synthetic.
Smartfailing node 3 on a 3-node cluster is permitted. OneFS allows up to ⌈N/2 − 1⌉ concurrent smartfails — one, at this cluster size — because removing one node still leaves quorum. Note Stop smartfail on node sitting directly beneath it in the same menu: that is your undo, and it only works while the evacuation is in flight.

Step 2 · FlexProtect runs Read-only

Cluster management → Job operations → Job summary. A FlexProtect job appears within about two minutes.

Job operations Job summary Job types Job reports Job events Impact policies Active jobs Status ID Type Priority Impact policy Elapsed Phase Progress Actions Running 121 FlexProtect 1 MEDIUM (Medium) 2 s 1 of 6 Started View / Edit More ▾ Why does FlexProtect run, and what does it do? OneFS quarantines the smartfailing device and redistributes every block it holds onto the remaining nodes, restoring the requested protection level wherever the surviving node count still allows it. It runs at job-engine priority 1 — the highest — because until it finishes, your data is under-protected. Reproduction of the OneFS Job operations → Job summary tab.
FlexProtect at priority 1, medium impact, six phases. It is not optional and it is not background housekeeping — it is the cluster racing to restore protection while the data is exposed. Let it finish. The job disappearing from Active jobs is how you know the smartfail is complete.

isi devices node smartfail --node-lnn 3 Destructive

# Establish quorum headroom BEFORE you start
boston-1# isi status                       # 3 nodes -> quorum is 2 -> you can afford to lose 1

boston-1# isi devices node smartfail --node-lnn 3
Are you sure you want to smartfail node 3? (yes/[no]): yes

boston-1# isi status
ID |IP Address     |DASR |
  1|10.10.10.11    | OK  |
  2|10.10.10.12    | OK  |
  3|10.10.10.13    |--S- |   <-- S = Smartfailed. Quorum still held (2 of 3).

boston-1# isi job jobs list
ID   Type         State    Impact  Pri  Phase  Running Time
-----------------------------------------------------------
121  FlexProtect  Running  MEDIUM  1    1/6    00:00:02

isi devices node stopfail --node-lnn 3 Disruptive

The undo — and it has a deadline. Aborts an in-flight smartfail and returns the node to service. Usable only while the evacuation is still running. Once the node is fully smartfailed and released, there is no stopfail; you are rejoining a node from scratch. In the WebUI it sits directly beneath Smartfail node in the same More menu — know it is there before you need it.

boston-1# isi devices node stopfail --node-lnn 3
boston-1# isi status                       # node should return to OK
What smartfail on a 3-node cluster does NOT give you. It is permitted, and access survives — but be clear about what you are left with:
  • Any path requested at 3x cannot be satisfied by two nodes. The sales home directory you mirrored in section 03 needs three nodes to hold three copies. FlexProtect will re-protect it as best it can, and isi storagepool health -v will show actual below requested. You are no longer meeting the SLA.
  • Three nodes is the minimum supported cluster size. A completed smartfail leaves two. It will run; it is not a configuration Dell supports.
  • You now have zero fault tolerance. Quorum on the original 3-node cluster was 2. With one node gone, losing one more takes you below quorum and the file system goes read-only. You survived this failure. You will not survive the next one.
Smartfail on a minimum-size cluster is a valid procedure and a poor plan. If you are retiring a node, add its replacement first.

05Verifying read-write access

"The cluster is up" and "clients can read and write" are different claims. Verify the second one directly, from a client, over the protocol your users actually use — and verify it before the smartfail so you have a baseline to compare against.

Before: establish the baseline Read-only

Map the share (\\sales\sales_gen), copy a file into it, and confirm the write lands. If you cannot write before the smartfail, nothing you observe afterwards means anything.

During and after: read and write again Read-only

Network ▸ sales ▸ sales_gen Search sales_gen Name Date modified Type Size docs 2026-07-13 13:05 File folder media 2026-07-13 13:06 File folder report_q3_baseline.pdf 2026-07-13 17:28 Adobe Acrobat D... 389 KB isilon-data-availability-protection.pdf 2026-07-13 17:25 Adobe Acrobat D... 1,533 KB New Text Document.txt 2026-07-14 02:21 Text Document 0 KB READ verified Existing documents open normally while node 3 is smartfailing and FlexProtect is running. WRITE verified — because quorum holds A new file is created successfully. 2 of 3 nodes is a majority. Lose quorum and this pool goes READ-ONLY. Reproduction of a mapped SMB share during a node smartfail. Filenames and timestamps are synthetic.
The share stays readable and writable while node 3 is smartfailing. This is the whole answer to “is my data still available?” — and the reason is quorum, not luck. Two of three nodes is a majority, so the node pool accepts writes. Take a second node out and the majority is gone: OneFS drops the file system to read-only, and that new file would fail to create while the existing PDFs would still open.

Cluster-side confirmation Read-only

# 1. Quorum and node state — is the smartfailed node still counted?
boston-1# isi status

# 2. Group membership — has the group re-formed cleanly?
boston-1# isi_group_info

# 3. Is protection still being satisfied? (3x on 2 nodes will NOT be)
boston-1# isi storagepool health -v

# 4. Is FlexProtect advancing, or stalled?
boston-1# isi job jobs list
boston-1# isi job jobs view 121
Healthy: quorum held, writes succeed, FlexProtect advancing through its phases.
Warning: reads and writes work, but actual protection is below requested — you are serving data with less resilience than the SLA promises.
Critical: writes failing while reads still succeed. That is the read-only signature of lost quorum. Do not "try a restart" — you are one node away from the file system being unavailable.
Next: if reads work but writes fail, check quorum first, then capacity and the VHS deny-writes reservation. It is almost never the protocol.

06Putting the node back

There are two ways back, and which one is available to you depends entirely on how fast you moved. They are not equivalent, and the difference between them is hours of data movement.

Path A — Stop the smartfailPath B — Add the node back
WhenFlexProtect still running; node still listedFlexProtect finished; node already removed
ActionMoreStop smartfail on node+ Add a node
CostNear zero. The node keeps its data.Hours. The node rejoins empty.
ResultNode returns to service as it wasNew member; AutoBalance refills it from scratch
This is why stopfail is worth knowing before you need it. The undo is free while the evacuation is in flight. Once FlexProtect completes and the node is released, that menu item is gone, and the recovery is a full re-join followed by a cluster-wide rebalance — moving data back onto a node that already had it an hour ago. Same end state, vastly different bill.

Path A · Stop smartfail on node Disruptive

Cluster management → Hardware configuration → Nodes. If the node is still listed, open More and select Stop smartfail on node. If this succeeds, you are done — skip Path B entirely.

boston-1# isi devices node stopfail --node-lnn 3
boston-1# isi status                 # node 3 returns to OK
boston-1# isi job jobs list          # FlexProtect winds down

Path B · Add the node back Disruptive

If FlexProtect completed, the node is gone from the cluster and the Nodes tab shows only the survivors. Rejoin it with + Add a node.

Dashboard ▾ Cluster management ▾ File system ▾ Data protection ▾ Access ▾ Protocols ▾ Hardware configuration Nodes Drives Cluster Manage nodes + Add a node Bulk actions Node Serial number Status OneFS version Configuration Chassis Actions 1 FCH2401A001 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A001 View details More ▾ 2 FCH2401A002 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A002 View details More ▾ 3 — removed. FlexProtect completed and the node was released. Too late for “Stop smartfail”. That menu item is gone. The node rejoins EMPTY and AutoBalance must refill it — hours of data movement. The cluster is running on 2 nodes — and that is the problem. Quorum on a 2-node cluster is 2. You now need BOTH nodes to accept writes: lose either one and the file system goes read-only. Zero fault tolerance. Rejoin the third node promptly — this is not a state to leave running. Reproduction of the OneFS Hardware configuration → Nodes tab after a completed smartfail. Serials are synthetic.
After FlexProtect completes, node 3 is gone and the cluster runs on two. It runs — but quorum on a 2-node cluster is 2, so you now need both survivors to accept writes. Lose either and the file system drops to read-only. This is a state to exit quickly, not one to sit in.

The Add node to cluster dialog

OneFS discovers unconfigured nodes on the internal (back-end) network and lists them as available. Select the node and click Add node. It takes a couple of minutes.

Add node to cluster Help ? Available nodes Product Serial # OneFS version Status PowerScale-F600-1U-384GB-2x25GE... FCH2401A003 B_9_5_0_002R available Try to add node if it is down Skip hardware version checks Cancel Add node Status must read “available” If the node is missing from this list, it still holds its old cluster config. Reset it before it can be discovered — nothing here will find a node that thinks it is busy. LEAVE BOTH UNCHECKED “Skip hardware version checks” disables the compatibility gate that stops you joining a mismatched node to the cluster. It exists for Support to use, on a case. Not for you, to make an error go away. Reproduction of the OneFS Add node to cluster dialog. Serials and versions are synthetic.
The Add node to cluster dialog. The node must appear with status available — a node still holding its old cluster configuration will not be discovered at all. And note the two unchecked boxes: Skip hardware version checks disables the compatibility gate that prevents a mismatched node joining the cluster. It exists so Dell Support can use it deliberately on an open case. Ticking it to make an error message disappear is how a cluster acquires a node it should never have accepted.

Confirm the node is Healthy

Back on Cluster management → Hardware configuration → Nodes, node 3 should return to the list with status Healthy.

Hardware configuration Nodes Drives Cluster Manage nodes + Add a node Node Serial number Status OneFS version Configuration Chassis Actions 1 FCH2401A001 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A001 View details More ▾ 2 FCH2401A002 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A002 View details More ▾ 3 FCH2401A003 Healthy 9.5.0.0 PowerScale-F600-1U-384GB-2x25GE-123TB FCH2401A003 View details More ▾ Healthy is not the same as protected. The node has rejoined the group and holds no data yet. AutoBalance must finish before isi storagepool health -v shows actual == requested again. Reproduction of the OneFS Hardware configuration → Nodes tab after the rejoin. Serials are synthetic.
Node 3 rejoined and Healthy — quorum headroom restored. With three nodes back, quorum returns to 2 and you can once again lose a node without going read-only. But Healthy describes group membership, not protection: the node is empty until AutoBalance refills it, and the 3x sales SLA is not met again until isi storagepool health -v says so.

What actually happens when the node rejoins Read-only concept

The node does not come back with its data. It comes back empty. This is the single most misunderstood thing about re-joining a smartfailed node, and it changes your time estimate by an order of magnitude.
  • FlexProtect already moved every block off that node during the smartfail. That was the entire point of the operation.
  • The rejoining node is therefore a new, empty member of the cluster. It gets a fresh LNN and holds nothing.
  • AutoBalance (or MultiScan) then runs to redistribute data across the restored node count — putting data back onto a node that had it an hour ago.
  • If the node still carries its old cluster configuration, it will not be discoverable as an available node. It must be reset first before Add a node can see it.
You will move the same data twice: once off the node during FlexProtect, once back on during AutoBalance. That round trip is the price of letting the smartfail complete when you meant to stop it.

Verify the rejoin Read-only

# 1. Three nodes back in the group?
boston-1# isi status
ID |IP Address     |DASR |
  1|10.10.10.11    | OK  |
  2|10.10.10.12    | OK  |
  3|10.10.10.13    | OK  |   <-- rejoined; quorum headroom restored

# 2. AutoBalance / MultiScan redistributing onto the new node
boston-1# isi job jobs list
ID   Type          State    Impact  Pri  Phase  Running Time
------------------------------------------------------------
122  AutoBalance   Running  LOW     4    1/3    00:12:41

# 3. THE check that closes the loop: is the sales SLA met again?
boston-1# isi storagepool health -v
Name              Health  Used    Size   Protection  Actual
--------------------------------------------------------------
f600_82tb_384gb   OK      69.4%   246T   3x          3x
#                                        ^^^^^^^^^^^^^^^^^^
# With 3 nodes back, 3x mirroring is satisfiable again and actual
# has caught up with requested. Until AutoBalance finishes, it will not have.

boston-1# isi get -D /ifs/sales/home     # confirm the manual 3x override survived
Do not declare victory at isi status. Three nodes showing OK means the node has joined the group. It does not mean your data is protected at the requested level again — AutoBalance has to finish first, and isi storagepool health -v is the only thing that will tell you when actual has caught up with requested. Between the rejoin and that moment, the sales SLA is still unmet.

07Runbook · node failure vs. node removal

Runbook A · You want to REMOVE a node (planned)

  1. isi status — how many nodes remain, and does that still meet quorum ⌊N/2⌋+1? On 3 nodes, removing one leaves 2 — quorum holds, writes continue. Removing two does not.
  2. isi storagepool health -v — can the requested protection still be satisfied by the remaining node count? 3x needs 3 nodes. After removal you have 2, so actual will fall below requested and the SLA is breached — even though the smartfail itself succeeds.
  3. Free capacity — can the survivors absorb the evacuated data? Above ~70% used on a small cluster, assume no.
  4. Only if all three checks pass → isi devices node smartfail --node-lnn <n>.
  5. Watch FlexProtect: isi job jobs list. If it stalls, isi devices node stopfail while you still can.

Runbook B · A node has FAILED (unplanned)

  1. isi status — is quorum held? On 3 nodes, losing 1 keeps quorum (2 of 3). Losing 2 does not — the cluster stops serving.
  2. Verify client read-write from an actual client. This is the question the business is asking. Answer it first.
  3. isi storagepool health -v — actual protection is now below requested. Expected. You are degraded, not broken.
  4. Do not smartfail the failed node reflexively. If the node is coming back, smartfailing it evacuates data you did not need to move and cannot re-protect anyway. Repair the node.
  5. Capture evidence before remediating: isi_gather_info.

Protection policy set by someone who has left?

WUC engineers audit PowerScale protection posture — requested-versus-actual gaps, node-count feasibility, VHS and spillover configuration, and DR failover readiness — on Isilon and PowerScale estates inside and outside OEM support.

Talk to engineering →

FAQFrequently asked questions

Q01Why can't I set +2n on my small cluster?

You can set it. That is the problem. An erasure-coded +Nn level needs 2N+1 nodes, so +2n needs 5 and +3n needs 7. Below that, OneFS accepts the setting and silently delivers a lower actual protection. Run isi storagepool health -v and read the Actual column.

Q02Can I smartfail a node on a 3-node cluster?

Yes — OneFS permits up to ⌈N/2 − 1⌉ concurrent smartfails, which is one node at this size, and access survives because quorum (2 of 3) holds. But understand what you are left with: any path requested at 3x can no longer be satisfied by two nodes, two nodes is below the supported minimum cluster size, and you now have zero remaining fault tolerance. It is a valid procedure and a poor plan. If you are retiring a node, add its replacement first.

Q03I started a smartfail by mistake. Can I stop it?

Yes — isi devices node stopfail --node-lnn <n>, but only while the evacuation is still running. Once the node is fully smartfailed and released, there is no undo; you are rejoining a node from scratch.

Q04I let the smartfail finish. How do I get the node back?

Cluster management → Hardware configuration → Nodes → + Add a node. But understand what you are getting: the node rejoins empty — FlexProtect already moved its data onto the survivors — so AutoBalance must then redistribute data back onto it. You move the same data twice. That round trip is the cost of missing the stopfail window. If the node still holds its old cluster config, reset it first or Add a node will not discover it.

Q05What actually happens to my data if a node dies?

On a 3-node cluster you keep quorum (2 of 3) and data stays readable and writable, because surviving mirror copies or FEC stripes cover the loss. You run degraded: actual protection drops below requested and cannot be restored until the node returns. You have survived this failure, not the next one.

Q06Is 3x mirroring "safer" than +2d:1n?

It tolerates more node loss, and it costs roughly three times the raw capacity to do it. On a 3-node cluster 3x consumes every node to hold the three copies, leaving no headroom for a planned removal. "Safer" depends on which failure you are buying insurance against — and on a small cluster, higher mirroring can make you less able to perform maintenance safely.

Q07Should I turn off Virtual Hot Spare to reclaim capacity?

No. VHS reserves the free space FlexProtect needs to rebuild after a drive failure. Turning it off on a nearly-full cluster removes the safety margin at exactly the moment the cluster is most likely to need it. If you need the capacity that badly, that is a capacity-planning finding, not a VHS setting.

Q08What does "Apply to files with manually-managed protection" actually do?

It changes SmartPools from managing only default-protected files to managing all files — overwriting every individual protection override anyone has ever set, cluster-wide, on the next job run. Someone set those overrides deliberately. Find them with isi get -DD and agree the change before you tick that box.

RFReferences

  1. Dell Technologies Info Hub — PowerScale OneFS Technical Overview: Data Protection
  2. Dell — High Availability and Data Protection with PowerScale Scale-Out NAS (H10588)
  3. Dell PowerScale OneFS Documentation — CLI Command Reference and Administration Guide

From the same practice

WE

About WUC Engineering

Storage engineers at WUC Technologies operate Dell PowerScale and Isilon clusters — from legacy X- and NL-series through F-, H- and A-series — under post-OEM storage maintenance engagements across enterprise data centers.

Is your requested protection the protection you actually have?

Most clusters we assess have at least one node pool where Actual does not match Protection — and nobody knew. WUC storage engineers deliver protection-posture audits, node-count feasibility reviews, capacity and restripe modelling, and DR failover validation for PowerScale and Isilon estates inside and outside OEM support.

  • Protection posture audit — requested vs. actual, every pool
  • Node-count and capacity feasibility before you change a policy
  • SmartFail / node-refresh planning
  • SyncIQ and DR failover validation
e.g. Cisco, Dell, NetApp - and when your next contract renews.

Prefer to talk it through first? Book a technical consultation → · View managed services

All hostnames, domains, users, paths, serial numbers and IP addresses in this guide are synthetic reference values. No customer environment is depicted.

Get a Custom Solution