Home › Resources › Field Guides › PowerScale (Isilon) OneFS Command Reference

Dell PowerScale (Isilon) OneFS Command Reference for Production Storage Operators

This guide consolidates the OneFS commands storage engineers actually reach for to validate cluster health, inspect node hardware, untangle hung SMB sessions, read SmartPools and protection policy, chase performance, drive SyncIQ failover, and package evidence for Dell support. Every command states what it shows, when to run it, whether it is safe to run, and a representative output.

Who this is for: storage engineers · NAS administrators · infrastructure teams · operations teams · Dell PowerScale administrators

Command syntax validated against OneFS 8.2.x and 9.x · outputs shown from 9.5.x · field labels and counter names shift between releases · Reviewed by WUC Storage Infrastructure Engineering · Last validated: 2026-07-13

!!Command safety model — read this first

ENVReference environment

Every command and output on this page is shown against a single consistent reference estate, so the examples hold together across families. It is a routed two-site design: a production cluster in Boston serving home directories, departmental shares and research data, replicating over a WAN link to a DR cluster in Phoenix. The two sites sit in separate Class A subnets and do not share a broadcast domain — which is the whole point of a DR site.

QSMost-used commands

IXCommands by outcome

• Need to prove the cluster is healthy → isi status, isi devices list, isi storagepool health -v
• Need to fix hung SMB clients → isi smb sessions list, then lwsm restart lsass on the affected node only
• Need to find the noisy neighbour → isi statistics client --sort=Ops --long, isi statistics heat list
• Need to know why a login fails → isi auth status -v, isi auth mapping token, isi zone zones list -v
• Need to prove DR is current → isi sync reports list, isi sync jobs list
• Need to fail over to DR → Runbook 4 — and read it before, not during, the disaster
• Need to know if a file is tiered or stubbed → isi get -DD <path>, isi cloud recall
• Need to collect evidence for Dell → isi_gather_info, isi upgrade patches list, isi_hw_status

01Cluster health & events

This is where every investigation starts. Before you theorise about a protocol, a client, or a switch, establish whether the cluster believes it is healthy, whether every node is in the group, and whether anything is already alarming. OneFS is unusual in that a cluster can be degraded — a node split out of the group, a drive smartfailing — while still serving data perfectly well, so "users aren't complaining" is not evidence of health.

isi status Read-only

The single highest-value command on the cluster. Health, capacity, per-node state and running jobs in one screen. On an all-flash F-series cluster the HDD column reads zero — all capacity is SSD.

isi status

Cluster Name: boston
Cluster Health:     [ ATTN]
Cluster Storage:  HDD                 SSD Storage
Size:             0 (0 Raw)           246T (369T Raw)
VHS Size:         18T
Used:             0 (0%)              171T (69%)
Avail:            0 (0%)              75T (31%)

                   Health  Throughput (bps)   SSD Storage
ID |IP Address     |DASR |  In    Out  Total| Used / Size
---+---------------+-----+------+-----+-----+-----------------
  1|10.10.10.11    | OK  |  142M| 388M| 530M|   57T/  82T( 69%)
  2|10.10.10.12    | OK  |  138M| 401M| 539M|   57T/  82T( 69%)
  3|10.10.10.13    |-A-- |    0 |   0 |   0 |   57T/  82T( 70%)
---+---------------+-----+------+-----+-----+-----------------
Cluster Totals:          |  280M| 789M|1.06G|  171T/ 246T( 69%)

     Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

isi status -q Read-only

Quick variant — suppresses the per-node table. Use it when you only want the cluster verdict and capacity.

isi status -q

Cluster Name: boston
Cluster Health:     [  OK ]
Size:  246T (369T Raw)
Used:  171T (69%)
Avail: 75T (31%)

isi status -n 3 Read-only

Scope to a single node by LNN. The fastest way to confirm the state of the one node an alert named — here, the node showing A (Attention) above.

isi status -n 3

Cluster Name: boston
Node LNN:    3
Node Health: [  OK ]
Node Storage: SSD 57T / 82T (70%)
Throughput:  In 151M  Out 377M

isi devices list Read-only

Every drive and node device with its state. Each F600 carries eight NVMe drives, so a 3-node cluster has 24 — small enough to read, but on larger estates the healthy rows are noise.

isi devices list

Lnn  Location  Device     Lnum  State      Serial
--------------------------------------------------------------
1     Bay  1   /dev/nvd0  7     HEALTHY    S5NXNG0R100001
1     Bay  2   /dev/nvd1  6     HEALTHY    S5NXNG0R100002
3     Bay  5   /dev/nvd4  3     SMARTFAIL  S5NXNG0R100021
3     Bay  6   /dev/nvd5  2     HEALTHY    S5NXNG0R100022
--------------------------------------------------------------
Total: 24

isi devices list | egrep -vi "healthy|l3" Read-only

The version you will actually type. Strips healthy drives and any L3-cache SSDs, leaving only what deserves attention. An empty result is the goal.

isi devices list | egrep -vi "healthy|l3"

Lnn  Location  Device     Lnum  State      Serial
3     Bay  5   /dev/nvd4  3     SMARTFAIL  S5NXNG0R100021

isi_for_array -n 3 'isi devices list | egrep -v HEALTHY' Read-only

Same filter, executed on node 3 specifically. isi_for_array is the cluster-wide fan-out wrapper; -n pins it to one node.

isi_for_array -n 3 'isi devices list | egrep -v HEALTHY'

boston-3: Lnn  Location  Device     Lnum  State      Serial
boston-3: 3     Bay  5   /dev/nvd4  3     SMARTFAIL  S5NXNG0R100021

isi_for_array -s 'isi_hw_status | grep SerNo; isi devices list | egrep -vi "healthy|l3"' Read-only

A composite audit sweep: for every node, print its serial then any unhealthy device. This is the one-liner to run before opening a hardware case, because it pairs the fault to the chassis serial Dell will ask for. -s sorts output by node number.

isi_for_array -s 'isi_hw_status | grep SerNo; isi devices list | egrep -vi "healthy|l3"'

boston-1:   SerNo: FCH2401A001
boston-2:   SerNo: FCH2401A002
boston-3:   SerNo: FCH2401A003
boston-3: 3     Bay  5   /dev/nvd4  3     SMARTFAIL  S5NXNG0R100021

isi_group_info Read-only

Group membership and the history of group changes. When a node "disappears", this tells you when it split and whether it rejoined. A cluster that is quietly forming and re-forming groups has an interconnect problem, not a protocol problem — and on a 3-node cluster, losing one node to a group split leaves you with no protection headroom at all.

isi_group_info

Cluster is in a fully functional state.
  1: 10.10.10.11  up
  2: 10.10.10.12  up
  3: 10.10.10.13  up
Last group change: 2026-07-13T04:22:11 (drive smartfail, node 3)

isi event events list Read-only

Current unresolved events. Read this before you clear anything.

isi event events list

ID       Occurred     Sev  Lnn  Event                     Message
---------------------------------------------------------------------------
7.12841  07/13 04:22  crit  3   400160002 (DRIVE_FAIL)    Drive in bay 5 failed
7.12839  07/12 23:07  warn  0   200010001 (CAPACITY)      Pool f600_82tb at 87%
---------------------------------------------------------------------------
Total: 2

isi event bulk --resolved=true Disruptive

Mass-resolves every event on the cluster. It is disruptive not to data but to your visibility — it clears genuine unresolved faults alongside stale noise. Never run this as a reflex to make a dashboard green; list the events first and fix what is real.

# Capture what you are about to clear:
boston-1# isi event events list > /ifs/data/events-20260713.txt
boston-1# isi event bulk --resolved=true

isi event bulk --ignore=true Disruptive

Same caution, stronger effect — suppresses events rather than resolving them, so they will not re-alert. Use only after you have captured the event list to a file.

isi event bulk --ignore=true

Are you sure? (yes/[no]): yes
14 events ignored.

02Node & hardware

Hardware questions arrive in two forms: an alert naming a component, or an audit demanding serials. Both are answered by isi_hw_status fanned across the array. The commands here are read-only except the last two, which take a node out of service — and on a minimum-size cluster, that is a far bigger decision than it looks.

isi_hw_status Read-only

Full hardware inventory and sensor state for the node you are on.

isi_hw_status

  SerNo: FCH2401A001
  Config: PowerScale F600
  ChsSerN: FCH2401A001
  FamCode: F
  ChsCode: 1U
  GenCode: 60
  Tier: 0
  Class: storage
  Product: F600-1U-Single-384GB-2x25GE SFP28-123TB
  HWGen: PSI
  Chassis: PowerScale
  CPU: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
  RAM: 412316860416 Bytes
  Power Supplies OK
  Power Supply Slot1-PS0 good
  Power Supply Slot2-PS1 good

isi_for_array -s isi_hw_status | egrep 'SerNo|Product' Read-only

The audit one-liner. Serial and model for every node in the cluster, sorted. Paste the output straight into an RMA request or an asset register.

isi_for_array -s isi_hw_status | egrep 'SerNo|Product'

boston-1:   SerNo: FCH2401A001
boston-1:   Product: F600-1U-Single-384GB-2x25GE SFP28-123TB
boston-2:   SerNo: FCH2401A002
boston-2:   Product: F600-1U-Single-384GB-2x25GE SFP28-123TB
boston-3:   SerNo: FCH2401A003
boston-3:   Product: F600-1U-Single-384GB-2x25GE SFP28-123TB

isi_for_array -s isi_hw_status | grep -i power Read-only

Power supply state across every node. Run it after any PDU work, any data-centre power event, and before any planned power maintenance.

isi_for_array -s isi_hw_status | grep -i power

boston-1:  Power Supplies OK
boston-2:  Power Supplies OK
boston-3:  Power Supplies ATTN

isi_for_array -s isi_hw_status | grep -i -A2 "Power Supplies" Read-only

The better form — -A2 pulls the two PSU slot lines that follow the summary, so you see which supply is unhappy rather than just that one is.

isi_for_array -s isi_hw_status | grep -i -A2 "Power Supplies"

boston-1:  Power Supplies OK
boston-1:  Power Supply Slot1-PS0 good
boston-1:  Power Supply Slot2-PS1 good
--
boston-2:  Power Supplies OK
boston-2:  Power Supply Slot1-PS0 good
boston-2:  Power Supply Slot2-PS1 good
--
boston-3:  Power Supplies ATTN
boston-3:  Power Supply Slot1-PS0 good
boston-3:  Power Supply Slot2-PS1 absent

isi_for_array -n 3 isi_hw_status | grep -i power Read-only

Scoped to node 3. Once the sweep above names a node, drop to it directly rather than re-running across the array.

isi_for_array -n 3 isi_hw_status | grep -i power

boston-3:  Power Supplies ATTN
boston-3:  Power Supply Slot1-PS0 good
boston-3:  Power Supply Slot2-PS1 absent

isi_for_array -s isi statistics drive list Read-only

Per-drive I/O and latency on every node. A drive that is technically HEALTHY but showing latency an order of magnitude above its peers is a drive about to fail — this is how you find it before the alert does.

isi_for_array -s isi statistics drive list

boston-1: 1:nvd0 NVMe 14.2 398.1 0.4ms 69.0%
boston-2: 2:nvd0 NVMe 13.9 401.7 0.4ms 69.1%
boston-3: 3:nvd4 NVMe 12.1 410.2 184ms  69.1%   <-- outlier

isi devices node smartfail --node-lnn 3 Destructive

Begins evacuating a node from the cluster — every block it holds is re-protected onto the remaining nodes, then the node is removed. It is the correct way to retire a node, it is not reversible once data movement is underway, and it runs for hours.

# Verify BOTH node count and free capacity before even considering this:
boston-1# isi status                      # how many nodes remain after removal?
boston-1# isi storagepool health -v       # can they absorb the evacuated data?
boston-1# isi devices node smartfail --node-lnn 3
boston-1# isi job jobs list               # watch the FlexProtect job that follows

isi config → reboot 3 Disruptive

Reboots node 3 from the isi config subshell. Clients connected to that node's IPs are disconnected; a dynamic IP pool will fail the addresses over to the surviving nodes, a static pool will not. On a 3-node cluster you are removing a third of your capacity and front-end bandwidth for the duration — do it in a window.

boston-1# isi config
boston-1>>> status
boston-1>>> reboot 3
boston-1>>> exit

03Networking & SmartConnect

OneFS networking is a four-layer hierarchy — groupnet, subnet, pool, interface — and almost every "the share is unreachable" ticket resolves to a misunderstanding of which layer owns the problem. Walk it top-down: does the groupnet resolve DNS, does the subnet have the right gateway and VLAN, does the pool have IPs and member interfaces, is the interface up.

isi network external view Read-only

The top-level external network configuration — the starting point for any connectivity question.

isi network external view

    Groupnets: 1
      Subnets: 1
        Pools: 2
  IP Alloc:  dynamic (SmartConnect)

isi network groupnets list -v Read-only

Groupnets carry DNS settings and the AD/LDAP binding context. A groupnet with the wrong DNS servers breaks name resolution for an entire access zone while every other layer looks perfectly healthy.

isi network groupnets list -v

       ID: groupnet0
     Name: groupnet0
  DNS Servers: 10.10.10.2, 10.10.10.2
DNS Search: nexora.local
  Subnets: subnet0

isi network subnets list -v Read-only

Subnets, netmasks, gateways, VLAN tags, and the SmartConnect service IP — the address the DNS delegation points at.

boston-1# isi network subnets list -v
             ID: groupnet0.subnet0
           Name: subnet0
       Groupnet: groupnet0
    Addr Family: ipv4
      Base Addr: 10.10.10.0
           CIDR: 10.10.10.0/24
        Gateway: 10.10.10.254
Gateway Priority: 10
            MTU: 9000
          Pools: pool-smb, pool-nfs
SC Service Addrs: 10.10.10.20
   VLAN Enabled: True
        VLAN ID: 110

# For contrast, the same command on the DR cluster — a different site,
# a different subnet, a different VLAN, a different gateway:
phoenix-1# isi network subnets list -v
             ID: groupnet0.subnet0
      Base Addr: 10.30.30.0
           CIDR: 10.30.30.0/24
        Gateway: 10.30.30.254
SC Service Addrs: 10.30.30.20
        VLAN ID: 130

The MTU is worth a second look. Jumbo frames (9000) inside a site are routine; across the WAN to Phoenix they are usually not, and a path-MTU mismatch is a classic cause of SyncIQ transfers that stall or crawl while ping and small transfers work perfectly. If replication is inexplicably slow, test the path at full frame size before you blame the policy.

isi network pools view groupnet0.subnet0.pool-smb Read-only

The pool is where SmartConnect lives — the zone name, the balancing policy, the IP range, the access zone, and the member interfaces. If a client resolves the SmartConnect name to an IP that nothing is listening on, the answer is here.

isi network pools view groupnet0.subnet0.pool-smb

              ID: groupnet0.subnet0.pool-smb
        Groupnet: groupnet0
          Subnet: subnet0
            Name: pool-smb
          Ranges: 10.10.10.11-10.10.10.13
    Alloc Method: dynamic
          Ifaces: 1:25gige-1, 2:25gige-1, 3:25gige-1
     Access Zone: zone-clinical
SC Connect Policy: round_robin
         SC Zone: nas.nexora.local
SC Suspended Nodes: -

isi network pools list Read-only

Every pool at a glance. Run it first, then view the one you care about.

isi network pools list

ID                            SC Zone                Alloc    Access Zone
--------------------------------------------------------------------------
groupnet0.subnet0.pool-smb    nas.nexora.local       dynamic  zone-clinical
groupnet0.subnet0.pool-nfs    research.nexora.local  dynamic  zone-research
--------------------------------------------------------------------------

isi network interfaces list -v Read-only

Physical and logical interfaces per node, with their assigned IPs and link state. This is the bottom of the network stack — if the NIC is down here, nothing above it matters.

isi network interfaces list -v

LNN  Name       Status      Owners                        IP Addresses
------------------------------------------------------------------------------
1    25gige-1   Up          groupnet0.subnet0.pool-smb    10.10.10.11
1    25gige-2   Up          groupnet0.subnet0.pool-nfs    10.10.10.40
2    25gige-1   Up          groupnet0.subnet0.pool-smb    10.10.10.12
3    25gige-1   No Carrier  -                             -
------------------------------------------------------------------------------

isi network rules list Read-only

Provisioning rules that bind interfaces into pools automatically as nodes join. A node that joined the cluster but is serving no traffic often has no rule matching it.

isi network rules list

ID          Node Type  Iface     Groupnet.Subnet.Pool
----------------------------------------------------------------
rule-smb    any        25gige-1  groupnet0.subnet0.pool-smb
rule-nfs    any        25gige-2  groupnet0.subnet0.pool-nfs

isi_for_array -s '/usr/bin/netstat -sn | grep "listen queue overflows"' Read-only

Listen queue overflows mean the node accepted connections faster than the protocol daemon could service them — a strong signal of daemon saturation, not network loss. A climbing counter here alongside hung SMB clients points at lwio/lsass, not at the switch. This is the command that names the bad node.

isi_for_array -s '/usr/bin/netstat -sn | grep "listen queue overflows"'

boston-1: 0 listen queue overflows
boston-2: 0 listen queue overflows
boston-3: 4412 listen queue overflows

isi_for_array -s 'netstat -an | grep 5667' Read-only

SyncIQ's worker port. Use it to confirm replication to Phoenix is actually establishing connections, and from which nodes. Because the sites are routed, every one of these sessions crosses the WAN — so this command doubles as proof that the inter-site path and its firewall rules are open. A SyncIQ policy that "runs slowly" is frequently only using a subset of nodes.

isi_for_array -s 'netstat -an | grep 5667'

boston-1: tcp4  0  0  10.10.10.11.42188   10.30.30.17.5667   ESTABLISHED
boston-2: tcp4  0  0  10.10.10.12.51022   10.30.30.18.5667   ESTABLISHED
boston-3: tcp4  0  0  10.10.10.13.39471   10.30.30.19.5667   ESTABLISHED

All three prod nodes connected to all three DR nodes is what healthy looks like. If only boston-1 appears, you are replicating at one-third of the available parallelism — usually a firewall permitting the WAN path from one source address rather than the whole prod subnet. Nothing errors; the policy just takes three times as long, and eventually misses its RPO.

isi_for_array -s 'netstat -an | grep ".445" | grep CLOSED | wc -l' Read-only

Counts SMB sockets stuck in CLOSED per node. A large and growing number is a fingerprint of the hung-session condition covered in family 06.

04Storage pools & protection

SmartPools decides where data lives and how hard it is protected. Two questions dominate: is anything under-protected relative to policy, and where did this particular file actually land. On a small cluster the protection question is sharper than most operators expect, because node count is a hard ceiling on the protection levels available to you.

isi_for_array -s 'netstat -an | grep ".445" | grep CLOSED | wc -l'

boston-1: 0
boston-2: 0
boston-3: 4412     <-- buildup of CLOSED SMB sockets on node 3

isi storagepool health -v Read-only

The verdict command for this family — requested vs actual protection, per pool. The mismatch below is exactly the failure mode described above.

isi storagepool health -v

Name              Health  Used    Size   Protection  Actual
--------------------------------------------------------------
f600_82tb_384gb   ATTN    69.4%   246T   +2n         +2d:1n   <-- requested != actual
--------------------------------------------------------------
# Requested +2n needs 5 nodes. This cluster has 3. It is quietly
# delivering +2d:1n and reporting ATTN rather than refusing the setting.

isi storagepool list -v Read-only

All pools — nodepools and tiers together — with capacity and membership.

isi storagepool list -v

Name              Type      Nodes  Protection  Used    Size
-----------------------------------------------------------------
f600_82tb_384gb   nodepool  1,2,3  3x          69.4%   246T

isi storagepool nodepools list -v Read-only

Nodepools specifically: which nodes are in them, what protection they carry, which L3 settings apply. On an all-flash cluster L3 cache is not applicable — the drives are the fast tier.

isi storagepool nodepools list -v

                ID: 1
              Name: f600_82tb_384gb
             Nodes: 1, 2, 3
      Node Type IDs: 1
 Protection Policy: +2d:1n
 Manual Protection: False
                L3: False
              Tier: -
             Usage: 69.4% (171T of 246T)

isi storagepool tiers list -v Read-only

Tiers group nodepools for file-pool policy targeting. If a file-pool policy is "not working", confirm the tier it targets actually contains the nodepool you assume it does. A single-nodepool cluster has nothing to tier between — a common source of misconfigured policies that silently do nothing.

isi storagepool tiers list -v

Name   Node Pools   Notes
----------------------------------------
(no tiers configured — single node pool cluster)

isi storagepool settings view Read-only

Global SmartPools behaviour — spillover, virtual hot spare reservation, global namespace acceleration, and the default protection applied to data with no matching policy.

isi storagepool settings view

       Automatically Manage Protection: files_at_default
   Automatically Manage Io Optimization: files_at_default
   Protect Directories One Level Higher: Yes
          Global Namespace Acceleration: disabled
           Virtual Hot Spare Deny Writes: Yes
            Virtual Hot Spare Hide Spare: Yes
          Virtual Hot Spare Limit Drives: 2
                      Spillover Enabled: Yes
                       Spillover Target: anywhere

isi get -DD /ifs/research/genomics/cohort-a.bam Read-only

The forensic command for a single file — its actual protection, its pool, its layout, and whether it is a SmartLink (CloudPools stub). Verbose to the point of overwhelming, so filter it.

isi get -DD /ifs/research/genomics/cohort-a.bam

isi get -DD <path> | grep -i smart Read-only

The practical form. Answers one question fast: has this file been tiered out to the cloud, or is it still on disk? A "missing" research file that is really a stub whose CloudPools account is disabled will show as SMARTLINKED here — see the warning in family 08.

isi get -DD /ifs/research/genomics/cohort-a.bam | grep -i smart

*  SMARTLINKED:       True
*  SMARTLINK STATE:   Stubbed
*  SMARTLINK ACCOUNT: cp-archive-01

isi_test_cpool_cbm --show-object-paths /ifs/research/genomics/cohort-a.bam Support-directed

Maps a SmartLink stub to the underlying cloud object paths. This is a Dell diagnostic binary, not a documented admin command — run it when Support asks you to prove where a stub's data physically lives during a CloudPools recovery.

isi_test_cpool_cbm --show-object-paths /ifs/research/genomics/cohort-a.bam

SmartLink: /ifs/research/genomics/cohort-a.bam
  Account:  cp-archive-01
  Objects:  3
    0: container=nha-archive  key=a1f3.../0000
    1: container=nha-archive  key=a1f3.../0001

isi storagepool nodepools modify --name=f600_82tb_384gb --protection-policy=+2d:1n Disruptive

Changes the protection policy for a nodepool. Innocuous-looking, heavy in effect: raising protection immediately queues a cluster-wide restripe that will consume back-end bandwidth for hours and will increase space consumption. Lowering protection reduces resilience instantly. On this cluster, +2d:1n is the correct target — +2n would be accepted and silently under-delivered.

boston-1# isi storagepool health -v            # confirm headroom AND node count first
boston-1# isi storagepool nodepools modify --name=f600_82tb_384gb --protection-policy=+2d:1n -v
boston-1# isi job jobs list                    # a restripe job will appear
boston-1# isi storagepool health -v            # verify actual == requested afterwards

isi_classic job history -j FSAnalyze -v Read-only

Historical run data for a named job. FSAnalyze feeds InsightIQ's file-system reporting; if the InsightIQ server is showing stale capacity or quota data, this tells you whether the job has been failing or was simply never scheduled.

05Authentication, zones & identity

Most "permission denied" tickets are not permission problems — they are identity problems. The user is being mapped to a different token than you assume, or the access zone is not the one you think the client hit, or a provider on that particular node has silently gone offline. Establish identity before you touch a single ACL.

isi_classic job history -j FSAnalyze -v

Job       ID  Start                End                  State
------------------------------------------------------------------
FSAnalyze 88  2026-07-12T22:00:02  2026-07-12T22:41:18  Succeeded

isi auth status -v Read-only

The health of every authentication provider, per zone. Run this before anything else on a login complaint. Note both providers below — AD for the Windows estate, LDAP for the Linux clients.

isi auth status -v

Zone: zone-clinical
  Provider: lsa-activedirectory-provider:NEXORA.LOCAL
    Status: online
    Connection: dc01.nexora.local (10.10.10.2)
  Provider: lsa-local-provider:zone-clinical
    Status: online

Zone: zone-research
  Provider: lsa-activedirectory-provider:NEXORA.LOCAL
    Status: online
  Provider: lsa-ldap-provider:nexora-ldap
    Status: online
    Connection: ldap://10.10.10.3
  Provider: lsa-file-provider:System
    Status: online

isi auth ads list -v Read-only

Active Directory provider detail — the joined domain, the machine account, the domain controllers in use, and the AD site. A cluster talking to a DC across a WAN link because site discovery failed will authenticate correctly but slowly, and users will call it "the storage being slow".

isi auth ads list -v

Name:               NEXORA.LOCAL
NetBIOS Domain:     NEXORA
Joined:             Yes
Machine Account:    BOSTON$
Site:               Default-First-Site-Name
Domain Controllers: dc01.nexora.local

isi auth ldap list -v Read-only

LDAP provider detail — the server URI, base DN, bind DN, and the attribute maps that turn an LDAP entry into a POSIX identity. When NFS clients get the wrong UID or land as nobody, the answer is almost always in the attribute mapping here, not in the export.

isi auth ldap list -v

             Name: nexora-ldap
      Server URIs: ldap://10.10.10.3
          Base DN: dc=nexora,dc=local
          Bind DN: cn=readonly,dc=nexora,dc=local
           Status: online
   User Filter: (objectClass=posixAccount)
  Group Filter: (objectClass=posixGroup)
       UID Attr: uidNumber
       GID Attr: gidNumber

isi auth mapping view --user=NEXORA\\jdoe --zone=zone-research Read-only

Shows how a single human is mapped across providers — the Windows SID on one side, the POSIX UID on the other, and whether OneFS has united them into one on-disk identity. In a dual-provider estate this is the command that explains why the same user sees different permissions over SMB than over NFS. If the mapping is absent or points at an unexpected UID, the ACL is a red herring.

isi auth mapping view --user=NEXORA\\jdoe --zone=zone-research

isi auth refresh Read-only

Flushes the authentication provider cache. Effectively read-only in impact — it forces the next lookup to go to the source rather than serving a stale cached answer. The right first move after a group membership change has not taken effect.

isi auth refresh

Authentication provider cache flushed.

isi auth mapping token --user=NEXORA\\acarter Read-only

The command that ends arguments. It prints the exact access token OneFS builds for a user — their UID, primary GID, and every SID in their group membership. If the group you granted access to is not in this list, the ACL was never the problem.

isi auth mapping token --user=NEXORA\\acarter

         User
                Name: NEXORA\acarter
                 UID: 1000042
                 SID: S-1-5-21-1004336348-1177238915-682003330-1109
  Primary Group
                Name: NEXORA\domain users
                 GID: 1000000
   Supplemental Identities
                Name: NEXORA\storage-admins
                 GID: 1000210
                Name: NEXORA\clinical-ops
                 GID: 1000315

isi auth mapping token --user=NEXORA\\jdoe --zone=zone-research Read-only

The same lookup scoped to an access zone. Critical in a multi-zone estate: the identity a research user resolves to in zone-research is not necessarily the identity they resolve to in zone-clinical. Always pass --zone when the zones differ, or you will debug the wrong token.

isi auth mapping token --user=NEXORA\\jdoe --zone=zone-research

isi auth users view --user=NEXORA\\jdoe Read-only

The provider's view of the user object. Confirms the account exists and is enabled from the cluster's perspective, which is not always what the AD console shows.

isi auth users view --user=NEXORA\\jdoe

isi zone zones list -v Read-only

Access zones, their auth providers, and their base paths. Zones are the most common source of "it works for me but not for them" — two users hitting different SmartConnect names land in different zones, with different providers and different namespaces.

isi zone zones list -v

        Name: System
        Path: /ifs
Auth Providers: lsa-local-provider:System, lsa-file-provider:System

        Name: zone-clinical
        Path: /ifs/clinical
Auth Providers: lsa-activedirectory-provider:NEXORA.LOCAL
 NetBIOS Name: NHA-CLIN

        Name: zone-research
        Path: /ifs/research
Auth Providers: lsa-activedirectory-provider:NEXORA.LOCAL
 NetBIOS Name: NHA-RSCH

isi_run -z 2 <command> Read-only

Executes a command in the context of a specific access zone by ID. Necessary because many isi commands operate against the System zone by default and will silently return nothing useful when the object you are asking about lives in another zone. "The share doesn't exist" is frequently "you queried the wrong zone".

isi_run -z 2 <command>

Auditing secure-channel failures in lsassd.log Read-only

When AD authentication degrades intermittently, secure-channel open failures in lsassd.log are the evidence. This counts them per domain controller for a given date. In a production estate with several DCs it isolates the one unhealthy controller the cluster keeps retrying; against the single DC here, a high count simply means the cluster is struggling to hold a secure channel to it at all — which is a domain-side or time-skew problem, not a storage one.

boston-1# isi_for_array -s "grep AD_NetrlogonOpenSchannel /var/log/lsassd.log \
  | grep 2026-07-13 \
  | awk -F'(' '{print \$2}' | awk -F')' '{print \$1}' \
  | sort -rn | uniq -c"
boston-1:  214 dc01.nexora.local
boston-2:  198 dc01.nexora.local
boston-3:  203 dc01.nexora.local

Counts this high on every node point away from a single wedged node and toward the domain itself. Check clock skew first — Kerberos fails hard past a five-minute drift, and it presents to users as random permission denials rather than as an authentication error.

06Protocol sessions & hung-session recovery

This is the most dangerous family on the page and the one most often copy-pasted without thought. The symptom is familiar: SMB clients hang, new connections stall, but the cluster reports itself healthy. The cause is usually a wedged lsass (authentication) or lwio (SMB I/O) daemon on a specific node. The cure is to restart that daemon — on that node.

isi smb sessions list Read-only

Active SMB sessions with user, client IP, and node. The first thing to check — and to capture — before you restart anything.

isi smb sessions list

Computer      User          Node  Openfiles
--------------------------------------------------
10.10.10.41   NEXORA\jdoe   1     3
10.10.10.44   NEXORA\svc-x  2     1

isi smb openfiles list Read-only

Files currently held open over SMB. A single file held open by a stale session is a far smaller problem than a wedged daemon, and it has a far smaller fix — close the specific file rather than bouncing a service.

isi smb openfiles list

ID    File                            User          Node
-----------------------------------------------------------------
1042  /ifs/finance/shared/q3.docx     NEXORA\jdoe   1
1051  /ifs/research/projects/run.log  NEXORA\svc-x  2

isi statistics query current list --keys=node.clientstats.connected.smb,node.clientstats.active.smb2 -n all -d Read-only

Connected vs active SMB clients per node. The distinction matters enormously: a node with many connected and zero active clients is a node whose daemon has stopped servicing work. That is your wedged node, and this is how you name it.

isi statistics query current list \

  --keys=node.clientstats.connected.smb,node.clientstats.active.smb2 -n all -d
Node  node.clientstats.connected.smb  node.clientstats.active.smb2
------------------------------------------------------------------
1     1412                            138
2     1398                            141
3     1455                            0        <-- wedged
------------------------------------------------------------------

isi statistics query current list --keys=node.clientstats.connected.nfs,node.clientstats.active.nfs -n all -d Read-only

The NFS equivalent, for the Linux research clients. Same interpretation — connected without active is the tell.

isi statistics query current list --keys=node.clientstats.connected.nfs,node.clientstats.active.nfs -n all -d

Continuous session monitor Read-only

A watch loop for a live incident: every 90 seconds, print connected/active counts and the CLOSED socket count per node. Run it in a second terminal while you work, so you can see the moment a node wedges — and the moment a restart fixes it.

while true; do
  echo "***** START *****"; date; echo
  isi statistics query current list \
    --keys=node.clientstats.connected.smb,node.clientstats.active.smb2 -n all -d \
    | grep -v "0 0"
  echo
  echo "***** Closed connections *****"
  isi_for_array -s 'netstat -an | grep ".445" | grep CLOSED | wc -l'
  echo "***** END *****"
  sleep 90
done

isi_for_array -n 3 '/usr/likewise/bin/lwsm restart lsass' Disruptive

The preferred recovery. A managed, graceful restart of the authentication daemon on node 3 only. Prefer this over killall in every case — it lets the service manager stop and start the daemon cleanly rather than aborting it. Clients on node 3 reconnect; with a dynamic IP pool most fail over transparently.

boston-1# isi_for_array -n 3 '/usr/likewise/bin/lwsm restart lsass'
boston-3: Stopping service: lsass
boston-3: Starting service: lsass
boston-1# isi auth status -v          # confirm the provider came back online

isi_for_array -n 3 'killall -6 lsass' Disruptive

Aborts the authentication daemon on node 3 with SIGABRT, forcing a core dump before the service manager restarts it. Use this instead of lwsm restart only when you need the core file for Dell to analyse — the crash dump is the entire point. Authentication on node 3 is interrupted while it restarts.

isi_for_array -n 3 'killall -6 lsass'

isi_for_array -n 3 'killall -6 lwio' && sleep 45 Disruptive

Same treatment for the SMB I/O daemon. The sleep 45 is not decoration — lwio takes time to come back and tear down its sockets, and hammering the node before it settles will make the diagnosis worse. Wait, then re-check session counts.

isi_for_array -n 3 'killall -6 lwio' && sleep 45

isi_for_array 'killall -6 lsass' Destructive

Cluster-wide SIGABRT to the authentication daemon on every node at once. Every SMB session on the cluster drops. Rated destructive not because data is lost but because the blast radius is total, and because it is routinely run by accident when someone omits -n. Reach for it only when node-scoped recovery has already failed, and only inside an accepted outage window.

isi_for_array 'killall -6 lsass'

isi_for_array 'killall -6 lwio' && sleep 45 Destructive

The cluster-wide SMB equivalent. Same warning, same blast radius.

isi_for_array 'killall -6 lwio' && sleep 45

isi_for_array 'killall -HUP isi_hangdump' Support-directed

Triggers the hang-dump collector. Do not run this speculatively — it exists to capture state for Dell engineering during an active hang investigation, and it is meaningful only when they have asked for it and told you when to fire it.

isi_for_array 'killall -HUP isi_hangdump'

07Performance & statistics

"The NAS is slow" is a claim, not a diagnosis. This family turns it into a name — a client, a directory, a drive, or a node. Work it in two passes: first a fast health check to rule out a degraded cluster, then the statistics to find the offender. Skipping the health check is how people spend an hour tuning a client when the real problem was a node out of the group.

A · Health check — start hereThirty seconds to confirm the cluster itself is healthy before you blame a client. Fuller cluster-health coverage is family 01.

isi status Read-only

The one-screen verdict: cluster health, per-node state, capacity, and running jobs. On a large, busy cluster it can take a moment to gather throughput — if you want it instantly, use the quick variant below.

isi status

Cluster Name: boston
Cluster Health:     [  OK ]
                   Health  Throughput (bps)   SSD Storage
ID |IP Address     |DASR |  In    Out  Total| Used / Size
---+---------------+-----+------+-----+-----+-----------------
  1|10.10.10.11    | OK  |  142M| 388M| 530M|   57T/  82T( 69%)
  2|10.10.10.12    | OK  |  138M| 401M| 539M|   57T/  82T( 69%)
  3|10.10.10.13    | OK  |  151M| 377M| 528M|   57T/  82T( 70%)
---+---------------+-----+------+-----+-----+-----------------
Cluster Totals:          |  431M|1.16G|1.59G|  171T/ 246T( 69%)
     Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

isi status -pqv  ·  isi status -p -q -v Read-only

The pool-oriented health view. -p breaks the report out by node pool / tier, -q is quick — it skips the slower throughput gather so the command returns immediately on a large cluster — and -v is verbose. The two forms below are identical: OneFS accepts flags bundled (-pqv) or written separately (-p -q -v). Use whichever you will remember; the bundled form is faster to type, the separated form is easier to read in a runbook.

isi status -pqv

isi status -p -q -v

Cluster Name: boston
Cluster Health:     [  OK ]

Node Pool: f600_82tb_384gb
     Health:            [  OK ]
     Requested Protection: 3x
     Actual Protection:    3x
     HDD Total:         0
     SSD Total:         246T
     SSD Used:          171T (69%)
     Nodes:             1, 2, 3
--------------------------------------------------------------
     Health Fields: D = Down, A = Attention, S = Smartfailed, R = Read-Only

isi status -q Read-only

Just quick, no pool breakdown. The fastest possible "is the cluster OK right now?" — it returns the health verdict and capacity without waiting to sample throughput. This is the one to reach for the instant a slowness ticket lands, because it costs the cluster almost nothing and answers the first question.

isi status -q

isi status -n 3 Read-only

Scope the health view to a single node by LNN — when a ticket or an earlier command has already named the node you care about.

isi status -n 3

B · Performance statisticsOnce the cluster is confirmed healthy, name the offender — a client, a directory, a drive, or a node.

isi statistics client --sort=Ops --long Read-only

The workhorse. Every client sorted by operations per second. Nine times out of ten the top row is your answer — here, a batch job on node 3 dwarfing everything else.

isi statistics client --sort=Ops --long

 Ops    In     Out    TimeAvg  Node  Proto  Class  UserName            RemoteName
--------------------------------------------------------------------------------
14.2k  92.1M  410.3M  18.4ms   3     nfs3   read   root                10.10.10.3
 1.1k   8.4M   31.2M   2.1ms   1     smb2   write  NEXORA\jdoe         10.10.10.1
  842   2.1M   19.8M   1.8ms   2     smb2   read   NEXORA\svc-imaging  10.10.10.101
--------------------------------------------------------------------------------

isi statistics client --nodes=all --sort=ops --format=top Read-only

A live, auto-refreshing top-style view. Leave it running during an incident. The username column truncates, so cross-reference by IP when a name looks wrong or empty.

isi statistics client --nodes=all --sort=ops --format=top

isi statistics client list --sort=Ops --long --format=csv Read-only

The same data as CSV, for when you need to hand evidence to someone or chart it. Redirect it to a file under /ifs/data/ and pull it off the cluster.

isi statistics client list --sort=Ops --long --format=csv

isi statistics heat list --nodes=all --limit=30 Read-only

The thirty hottest files and directories on the cluster. This is how you find the contended object — a single lock-heavy directory that every client is queuing behind, which no client-level view will ever reveal.

isi statistics heat list --nodes=all --limit=30

 Ops    Node  Event      Class      Path
------------------------------------------------------------------
 8.4k   3     lock       write      /ifs/research/genomics/scratch/
 6.1k   3     getattr    namespace  /ifs/research/genomics/scratch/
 1.2k   1     read       read       /ifs/clinical/imaging/study-4471/
------------------------------------------------------------------

isi statistics drive -nall --long --sort=OpsOut Read-only

Per-drive throughput and latency across every node, sorted by outbound operations. Look for the outlier: a drive whose latency is far above its siblings is failing, whatever its HEALTHY status claims.

isi statistics drive -nall --long --sort=OpsOut

Drive    Type  OpsIn  OpsOut  TimeAvg  Slow  Used
-----------------------------------------------------
3:nvd4   NVMe   12.1   410.2  184.2ms   41   69.1%   <-- outlier
1:nvd2   NVMe   14.8   398.7    0.4ms    0   69.0%
2:nvd3   NVMe   13.2   402.1    0.4ms    0   69.2%
-----------------------------------------------------

isi statistics protocol list --nodes=all --long Read-only

Latency broken out by protocol operation. Tells you whether the pain is in metadata (getattr, lookup) or in data (read, write) — a distinction that points at completely different fixes.

isi statistics protocol list --nodes=all --long

isi statistics system --nodes=all Read-only

CPU, network, and disk throughput per node. Answers the blunt question: is the cluster actually busy, or idle and still slow? An idle-but-slow cluster is a locking or metadata problem, not a capacity one.

isi statistics system --nodes=all

isi_cache_stats Read-only

L1/L2 cache hit rates. Collapsing hit rates alongside rising latency usually means the working set has outgrown cache — a capacity-planning finding, not a fault.

isi_cache_stats

isiperf_v3.sh — Dell performance capture Support-directed

Dell's performance capture harness — samples at an interval, for a number of rounds, and packages the result. Run it when Support asks for a capture window, with the parameters they specify: -i interval in seconds, -e iterations per round, -r rounds.

/bin/bash /ifs/data/Isilon_Support/isiperf_v3.sh -i 10 -e 5 -r 12

08Jobs, SyncIQ & CloudPools

The job engine is the cluster's background metabolism — restripes, protection repair, dedupe, tree deletes — and the most common invisible cause of "unexplained" slowness. SyncIQ and CloudPools sit alongside it, and both contain commands that can break a DR relationship or make data unavailable in a single keystroke.

isi job jobs list Read-only

What the job engine is doing right now, and at what impact policy. Always check this before investigating a performance complaint.

isi job jobs list

ID    Type          State    Impact  Pri  Phase  Running Time
-------------------------------------------------------------
1247  FlexProtect   Running  HIGH    1    2/4    14h 22m
1249  SmartPools    Paused   LOW     6    1/2    -
-------------------------------------------------------------

isi job jobs view 1247 Read-only

Detail on a single job — phase, progress, estimated completion. A FlexProtect that has not advanced a phase in a day is stuck and needs a case.

isi job jobs view 1247

ID:            1247
Type:          FlexProtect
State:         Running
Impact Policy: HIGH
Priority:      1
Phase:         2 of 4
Progress:      lin-based scan (est. 3h 40m remaining)

isi job events list Read-only

Job engine history — what ran, what failed, when. The audit trail for "did something run last night that explains this?"

isi job events list

Time                 Job          Event
------------------------------------------------------------
2026-07-13T04:22:14  FlexProtect  Started (drive smartfail, node 3)
2026-07-12T22:00:02  FSAnalyze    Succeeded

isi sync policies list Read-only

SyncIQ replication policies, their targets, and their schedules. This is the inventory of your DR coverage — and the place to notice that a critical dataset has no policy at all.

isi sync policies list

Name                  Path              Target       Action  Enabled  Schedule
-------------------------------------------------------------------------------
sync-clinical-phx     /ifs/clinical     10.30.30.20  sync    Yes      every 4 hours
sync-home-phx         /ifs/home         10.30.30.20  sync    Yes      daily 22:00
sync-research-phx     /ifs/research     10.30.30.20  sync    Yes      daily 01:00
-------------------------------------------------------------------------------

isi sync jobs list Read-only

Currently running replication jobs with throughput. A policy whose runtime creeps up each night is heading for a missed RPO — catch it before the auditor does.

isi sync jobs list

Policy Name        ID   State    Action  Elapsed   Transferred
--------------------------------------------------------------------
sync-clinical-phx  412  running  sync    00:14:02  184.2 GB

isi sync reports list --policy-name=sync-clinical-phx Read-only

The evidence that DR actually works. Historical results per policy — this is what you show an auditor to prove the Phoenix copy is current, and what tells you a policy has been silently failing for a fortnight.

isi sync reports list --policy-name=sync-clinical-phx

Policy Name        Job ID  Start            End              Action  State
----------------------------------------------------------------------------
sync-clinical-phx  412     07/13 04:00:02   07/13 04:18:44   sync    finished
sync-clinical-phx  411     07/13 00:00:01   07/13 00:21:07   sync    finished
sync-clinical-phx  410     07/12 20:00:02   07/12 20:19:33   sync    finished
----------------------------------------------------------------------------

isi sync policies view sync-clinical-phx Read-only

Full policy definition — source, target, snapshot settings, and whether accelerated_failback is enabled. Check that last one before a disaster, not during: without it, failing back to Boston requires a full differential rebuild.

isi sync policies view sync-clinical-phx

Name:                 sync-clinical-phx
Source:               /ifs/clinical
Target Host:          10.30.30.20
Action:               sync
Schedule:             every 4 hours
Accelerated Failback: Yes

isi sync recovery allow-write sync-clinical-phx Destructive

Makes the target of a replication policy writable — the failover action, run on the DR cluster. It breaks the sync relationship: Phoenix is no longer a faithful copy of Boston, and returning to the original direction requires a resync that can move enormous volumes of data across the WAN.

# Run on the DR cluster (phoenix), during a declared failover only:
phoenix-1# isi sync policies list                      # confirm the policy name
phoenix-1# isi sync recovery allow-write sync-clinical-phx
phoenix-1# isi sync recovery resync-prep sync-clinical-phx   # later, to prepare failback

isi cloud accounts list Read-only

CloudPools accounts and whether each is enabled. Check this before assuming a stub file is corrupt.

isi cloud accounts list

Name           Type   Enabled  URI
------------------------------------------------------
cp-archive-01  ECS    Yes      https://ecs.nexora.local:9021

isi cloud accounts modify cp-archive-01 --enabled=no Destructive

The most quietly dangerous command in this guide. Disabling a CloudPools account deletes nothing — but every SmartLink stub pointing at that account becomes unreadable immediately. To users, files that were there a moment ago now throw I/O errors. There is no data loss and the fix is to re-enable, but the user-visible effect is indistinguishable from mass data loss, and on an archive tier it can hit millions of files at once. Know what is stubbed against the account (isi get -DD) before you disable it.

boston-1# isi cloud accounts list
boston-1# isi cloud accounts modify cp-archive-01 --enabled=no
# Undo — restores access immediately, nothing was deleted:
boston-1# isi cloud accounts modify cp-archive-01 --enabled=yes

isi cloud recall /ifs/research/genomics/cohort-a.bam -v Disruptive

Rehydrates a stub — pulls the file's data back from the cloud onto the cluster. Disruptive by capacity: recalling a large stubbed dataset can consume far more space than anyone estimated, and there is no dry-run. On a cluster already at 69%, recalling an archived genomics cohort is a capacity event. Check the size of what you are recalling, and the free space you have, before you start.

isi cloud recall /ifs/research/genomics/cohort-a.bam -v

Recalling /ifs/research/genomics/cohort-a.bam ...
  3 cloud objects retrieved (2.1 GB)
  File is now fully local; SmartLink stub removed.

isi job jobs start treedelete --paths=/ifs/research/genomics/scratch --priority=10 --policy=low Destructive

This deletes a directory tree and everything under it. It is the efficient way to remove millions of files — the job engine does it far faster than rm -rf — and it is irreversible. The --policy=low keeps it from starving client I/O; --priority=10 puts it at the back of the queue. Neither flag makes it safer. Verify the path twice, confirm a snapshot exists, and never let a trailing-slash typo choose the path for you.

# VERIFY the path resolves to what you think it does, FIRST:
boston-1# ls -ld /ifs/research/genomics/scratch
boston-1# isi snapshot snapshots list | grep genomics   # is there a rollback?
boston-1# isi job jobs start treedelete --paths=/ifs/research/genomics/scratch --priority=10 --policy=low

09Permissions & ACLs

OneFS runs a hybrid permission model — POSIX mode bits and Windows ACLs over the same namespace — and the friction between them causes a disproportionate share of escalations. It is worst where a directory is reached by Windows staff over SMB and by Linux research clients over NFS. Before changing anything here, prove the user's identity with isi auth mapping token. Most ACL "fixes" are applied to problems that were never ACL problems.

ls -led /ifs/clinical/shared Read-only

The command to run before and after any ACL change. Prints the effective ACL, owner, group, and the inheritance control flags. Capture the "before" output somewhere you can get back to.

ls -led /ifs/clinical/shared

drwxrwx---  +  12 NEXORA\svc-owner  NEXORA\domain users  496 Jul 13 08:35 .
 OWNER: user:NEXORA\svc-owner
 GROUP: group:NEXORA\domain users
 CONTROL:dacl_auto_inherited,dacl_protected
 0: group:NEXORA\clinical-ops allow dir_gen_read,dir_gen_write,dir_gen_execute,std_delete,object_inherit,container_inherit
 1: group:NEXORA\research-faculty allow dir_gen_read,dir_gen_execute,object_inherit,container_inherit
 2: SYSTEM allow dir_gen_all,object_inherit,container_inherit

chmod -R +a group NEXORA\\storage-admins allow dir_gen_all,object_inherit,container_inherit,inherited_ace /ifs/clinical/shared Disruptive

Adds an inheritable full-control ACE for a group across a tree. The three inheritance flags matter: object_inherit propagates to files, container_inherit to directories, inherited_ace marks the ACE as inherited rather than explicit. Omit them and the grant applies only to the top directory — the single most common cause of "I gave them access and it still doesn't work".

# Capture the before-state first:
boston-1# ls -led /ifs/clinical/shared > /ifs/data/acl-before-20260713.txt
boston-1# chmod -R +a group NEXORA\\storage-admins allow \
  dir_gen_all,object_inherit,container_inherit,inherited_ace \
  /ifs/clinical/shared
boston-1# ls -led /ifs/clinical/shared

chmod -R +a user NEXORA\\svc-backup allow dir_gen_all,object_inherit,container_inherit,inherited_ace /ifs/clinical/shared Disruptive

The same grant for a service account rather than a group. Prefer groups — a per-user ACE on a large tree is a maintenance liability you will inherit later. Note the escaped backslash in the domain-qualified name.

chmod -R +a user NEXORA\\svc-backup allow dir_gen_all,object_inherit,container_inherit,inherited_ace /ifs/clinical/shared

chmod -R -b 770 /ifs/research/projects Destructive

Strips every ACL from the tree and reverts it to pure POSIX mode bits. This is the legitimate fix when a directory has accumulated so much conflicting ACL cruft that Windows and Unix clients disagree about who can do what — a classic outcome on a shared research tree — and it permanently discards every existing ACE. Everyone who had access via an ACL loses it the instant this completes. The mode (770) is your choice; the ACL destruction is not optional.

# There is NO undo. Capture the existing ACLs first — that file is your only record:
boston-1# ls -leRd /ifs/research/projects > /ifs/data/acl-backup-20260713.txt
boston-1# chmod -R -b 770 /ifs/research/projects

10SMB share management

Everything up to this point inspects the cluster. This family changes who can reach it. Creating a share, granting a permission, or toggling oplocks are all live access-control operations — read the badges, and treat a permission grant with the same care you would a firewall rule, because that is what it is.

isi smb shares list --zone=zone-research Read-only

Every SMB share in a given access zone. Shares are per-zone objects, so a share that "doesn't exist" almost always exists in a zone you did not query.

boston-1# isi zone zones list                    # what zones exist?
boston-1# isi smb shares list --zone=zone-research
Share Name        Path
------------------------------------------------
Finance_Shared$   /ifs/finance/shared
projects          /ifs/research/projects
------------------------------------------------

isi smb shares view Finance_Shared$ --zone=zone-research Read-only

Full definition of one share — its path, its permissions, and its per-share settings. Read this before you change anything about the share.

isi smb shares view Finance_Shared$ --zone=zone-research

Share Name:  Finance_Shared$
Path:        /ifs/finance/shared
Permissions:
  NEXORA\finance-team   allow  full
Oplocks:     Yes
Access Zone: zone-research

isi smb shares create --zone= Disruptive

Creates a share. Disruptive because it exposes a filesystem path over SMB the moment it succeeds — the access-control decision is made here, not later.

isi smb shares create Finance_Shared$ /ifs/finance/shared --zone=zone-research

# The trailing $ hides the share from network browsing. It does NOT restrict
# access — a hidden share with open permissions is still open. Hiding is not
# securing; set the permissions.

isi smb shares permission create — grant access Disruptive

Grants a permission on a share. Prefer a named AD group over a well-known identity every time.

isi smb shares permission create Finance_Shared$ \

  --group "NEXORA\finance-team" --permission-type allow --permission full \
  --zone=zone-research

# Do NOT reach for this reflexively:
boston-1# isi smb shares permission create Finance_Shared$ \
  --wellknown "NT AUTHORITY\Authenticated Users" \
  --permission-type allow --permission full --zone=zone-research

isi smb shares permission delete Disruptive

Removes a permission. The classic use is stripping the default Everyone ACE that some share-creation paths add. Confirm what you are removing before you force it — --force skips the confirmation, not the consequences.

boston-1# isi smb shares permission view Finance_Shared$ --zone=zone-research   # look first
boston-1# isi smb shares permission delete Finance_Shared$ \
  --wellknown Everyone --force --zone=zone-research

isi smb shares modify --oplocks=false — a latency fix with a cost Disruptive

Opportunistic locks let clients cache aggressively. On a share with heavy concurrent multi-writer access — a shared finance workbook, a database file over SMB — oplock break storms can present as latency, and disabling oplocks can resolve it.

isi smb shares modify --share=Finance_Shared$ --zone=zone-research --oplocks=false

isi smb openfiles close --id=<id> — clear a stuck lock Disruptive

When a single file is wedged open by a stale session — the "someone has this file locked and they went home" ticket — this closes that specific handle without touching any other session. Find the handle first, then close it by ID. This is a scalpel; the daemon restarts in family 06 are the hammer, and you reach for the scalpel first.

boston-1# isi_for_array "isi smb openfiles list" | grep -i "quarterly-report.docx"
ID    File                                    User
-----------------------------------------------------------
1042  /ifs/finance/shared/quarterly-report.docx  NEXORA\jdoe
boston-1# isi smb openfiles close --id=1042

Finding a file across the namespace Performance impact

When you need the full path of a file by name, find works — but scope it. A find rooted at / on a multi-petabyte cluster is a metadata-read storm that will show up in someone's latency graph. Root it at the narrowest path you can, and background it.

find /ifs/finance -type f -name "quarterly-report.docx" >> /ifs/data/found.txt &

# For a metadata-only search at scale, isi statistics heat (family 07) is lighter.

11Time, NTP & Kerberos alignment

Time is an authentication dependency, and almost nobody treats it as one until it breaks. Kerberos rejects a ticket whose clock is skewed more than five minutes from the KDC. When a node's clock drifts, AD authentication on that node fails — and it presents to users not as "authentication is down" but as random, intermittent permission denials. This family is short, and it is the first thing to check when AD works on some nodes and not others.

isi_for_array -s /usr/likewise/bin/lw-get-dc-time <domain> Read-only

The domain controller's clock, as each node sees it. This is your reference — the number every node must agree with, because it is the clock Kerberos is measured against.

isi_for_array -s /usr/likewise/bin/lw-get-dc-time nexora.local

isi_for_array -s date Read-only

Each node's own clock, side by side. Compare against the DC time above; any node more than a few seconds out is drifting, and any node near five minutes out is already failing Kerberos.

isi_for_array -s date

boston-1: Mon Jul 13 14:02:11 EDT 2026
boston-2: Mon Jul 13 14:02:11 EDT 2026
boston-3: Mon Jul 13 14:06:44 EDT 2026     <-- 4m33s ahead. Kerberos is about to fail here.

isi_for_array -s "ntpdate -u -b <ntp-server-ip>" Disruptive

Forces an immediate time correction on every node. Disruptive because stepping a clock is not free — a large jump can disturb time-sensitive services and, briefly, invalidate in-flight Kerberos tickets as the clock crosses the skew boundary. Correct a genuine skew, but do it in a maintenance window if the jump is large, and fix the underlying NTP configuration so it does not recur.

# Prefer pointing NTP at the same source the domain uses (often the DC itself).
boston-1# isi_for_array -s "ntpdate -u -b 10.10.10.2"
boston-1# isi_for_array -s date        # re-verify every node now agrees

12SPN, domain controllers & node renumbering

The operations here are infrequent, high-consequence, and easy to get wrong because they touch the cluster's identity in the domain and its identity to itself. None of them are daily commands; all of them are worth knowing before the day you need them.

isi auth ads spn list <domain> Read-only

The Service Principal Names registered for the cluster's machine account. When Kerberos authentication to a share fails but the provider is online, a missing or duplicate SPN is a prime suspect — the client asks for a ticket to a name the KDC has no SPN for, and the request fails before it ever reaches the cluster.

isi auth ads spn list NEXORA.LOCAL

isi auth ads spn check / fix <domain> Disruptive

check is read-only and reports missing SPNs; fix registers them, which writes to the machine account in AD. Run check first, understand what is missing and why, then fix deliberately.

boston-1# isi auth ads spn check NEXORA.LOCAL      # read-only report
boston-1# isi auth ads spn fix   NEXORA.LOCAL      # writes to AD — deliberate only

isi auth ads modify --domain-controller=<dc> Disruptive

Pins the cluster to a specific domain controller instead of letting AD site discovery choose. This is a legitimate emergency lever when discovery keeps selecting an unhealthy or distant DC — but it is also a trap.

boston-1# isi auth ads modify nexora.local --domain-controller=dc01.nexora.local
# Undo — return to automatic site-based discovery:
boston-1# isi auth ads modify nexora.local --domain-controller=""

isi devices node smartfail --node-lnn <n> Destructive

Covered in depth — including the quorum arithmetic and why a small cluster survives it — in the dedicated PowerScale Data Protection & SmartFail field guide. Note the correct spelling: devices, smartfail — the command is frequently mistyped, and a mistyped destructive command that happens to match a different valid command is its own hazard.

lnnset modify — renumber nodes after a smartfail Disruptive

Logical Node Numbers (LNNs) do not automatically close up after a node is removed — smartfail node 2 of three and you are left with LNNs 1 and 3, not 1 and 2. That gap is cosmetic, but it confuses scripts, monitoring, and humans reading isi status. lnnset modify in the isi config subshell renumbers them.

boston-1# isi config
boston-1>>> lnnset list              # show current LNN -> device mapping
boston-1>>> lnnset modify 3 2        # renumber LNN 3 to LNN 2, closing the gap
boston-1>>> commit
boston-1>>> exit

13Upgrade, patches & firmware

Patching and OS upgrades are the highest-consequence routine work you do on a cluster — they touch every node and, done wrong, take the whole cluster offline. OneFS upgrades are rolling by default (one node at a time, quorum preserved), which is why access survives them; but "rolling" is not "risk-free." Know your current version before you touch anything, read what a patch changes before you install it, and never start an OS upgrade without a tested rollback plan.

isi version Read-only

The current OneFS release, every time. This is the first thing Support asks and the first thing a known-issue search needs — behaviour, bugs, and even command syntax shift between releases, so establish it before you reason about anything else.

isi version

Isilon OneFS v9.5.0.0 B_9_5_0_002(RELEASE)
     Build:  B_9_5_0_002(RELEASE)
     Cluster: boston  (3 nodes, all at 9.5.0.0)

isi upgrade patches list Read-only

Which patches are installed. The first thing to check when a known bug matches your symptom, and the list Support expects attached to any case. Compare it against Dell's Security Advisories and patch bulletins for your release.

isi upgrade patches list

Patch Name         Description                              Status
-------------------------------------------------------------------
patch-321045       SMB session cleanup fix                  Installed
patch-320981       NFS lock reclaim on failover             Installed
-------------------------------------------------------------------

isi upgrade patches list --format=table Read-only

The same data in a clean, fixed-column table — easier to read at a glance and easier to parse if you are feeding it into a report or a scripted inventory across many clusters.

isi upgrade patches list --format=table

isi upgrade patches view <patch> Read-only

What a specific patch actually changes — the services it touches, whether it needs a reboot, and what it supersedes or depends on. Read this before you install, so you know whether the change is a quiet daemon restart or something with client impact.

isi upgrade patches view patch-321045

Patch Name:         patch-321045
Description:        SMB session cleanup fix
Status:             Installed
Services affected:  lwio  (rolling restart, no reboot)
Supersedes:         none
Required by:        none

isi upgrade patches install <path> Disruptive

Installs a patch, rolling across the nodes. Most patches restart a service rather than reboot a node, but "restart a service" still means a brief interruption for clients on that node. Stage the patch under /ifs, read its view first, and install in a change window unless Support has told you it is non-disruptive.

isi upgrade patches install /ifs/data/patches/patch-321102.tgz

Patch patch-321102 staged.
Installing across 3 nodes (rolling) ...
  node 1: done   node 2: done   node 3: done
Patch patch-321102 installed.

isi upgrade patches uninstall <patch> Disruptive

Removes a patch, rolling. The same client-impact caution as install applies. Confirm nothing else depends on the patch (its view shows this) before removing it.

isi upgrade patches uninstall patch-321102

isi upgrade view Read-only

The status of an in-progress OS upgrade — which nodes are done, which phase the cluster is in, and whether it is waiting on you to commit. When nothing is running it says so; run it any time you are unsure what state an upgrade left the cluster in.

isi upgrade view

Upgrade:  OneFS 9.5.0.0  ->  9.7.0.0
  Status:          committed pending
  Process:         rolling
  Nodes upgraded:  3 of 3
  Action needed:   run "isi upgrade cluster commit" to finalise

isi upgrade cluster start <install-image> Destructive

Begins a rolling OneFS OS upgrade. This is the highest-stakes command in the guide: it changes the operating system on every node in the cluster.

isi upgrade cluster start /ifs/data/install/OneFS_v9.7.0.0.isi

isi upgrade cluster rollback Destructive

Rolls a cluster back to the prior release — but only while the upgrade is still in the pre-commit state. Once you have committed, this command is no longer available; the way back is a full downgrade, which is itself an upgrade operation with its own risk. This is why the pre-commit window matters so much: it is the entire span in which "undo" exists.

isi upgrade cluster rollback

isi upgrade cluster commit Disruptive · irreversible

Finalises the upgrade and closes the rollback window for good. Run it only once every node is upgraded, the cluster is healthy, and you have verified applications against the new release. After commit, the prior version is gone.

isi upgrade cluster commit

13Incident runbooks

Four sequences covering the majority of PowerScale escalations. Each step names the command and the branch condition — the point is to stop early when the evidence says stop, rather than working the whole list out of habit.

Runbook 1 · SMB clients hung, cluster reports healthy

1. isi status — is the cluster actually OK, or is a node out of the group? Node down → hardware/group problem, not protocol. Stop here.
2. isi auth status -v — is the AD provider online on every node? Offline cluster-wide → domain-side problem. Stop here.
3. listen queue overflows + CLOSED socket count — which single node is saturating? This names the node.
4. connected vs active SMB clients — the wedged node shows connections but zero activity. Confirms the node.
5. isi_for_array -n <node> '/usr/likewise/bin/lwsm restart lsass' — graceful restart, that node only. Re-check step 4.
6. Still wedged after a settle period → restart lwio on the same node, wait 45s, re-check. Escalate to cluster-wide only inside an accepted outage.

Runbook 2 · Storage is slow, nothing is down

1. isi job jobs list — is a FlexProtect or SmartPools job running at HIGH impact? Yes → very likely your answer. Lower the impact policy.
2. isi statistics client --sort=Ops --long — is one client dominating? Yes → you have a name. Go talk to its owner.
3. isi statistics heat list --nodes=all --limit=30 — is one directory contended? Yes → a locking hot-spot; no client-level view would have shown this.
4. isi statistics drive -nall --long --sort=OpsOut — is one drive's latency an outlier? Yes → a failing drive, regardless of its HEALTHY state.
5. isi statistics system --nodes=all — is the cluster busy at all? Idle but slow → metadata/locking, not capacity.

Runbook 3 · Users report files have vanished

1. ls -led on the parent — do the files exist with an ACL the user cannot traverse? A permission problem masquerading as absence.
2. isi auth mapping token --user=NEXORA\\<user> --zone=<zone> — is the user in the group you think grants access? Frequently not. Stop here if so.
3. isi get -DD <path> | grep -i smart — is the file a CloudPools stub? Stubbed → check the account.
4. isi cloud accounts list — is the account backing that stub disabled? Disabled → this is the cause. Re-enable it; the data was never gone.
5. Only after all of the above → check snapshots and SyncIQ for an actual deletion.

Runbook 4 · DR failover to the secondary site

1. Confirm the primary is genuinely lost. A cluster that is unreachable from your desk is not necessarily down. If Boston is recoverable, recover it — failover is more expensive than almost any repair.
2. isi sync reports list on the DR cluster — when did each policy last complete? This tells you exactly how much data you are about to lose. Write the number down; it is your actual RPO, not the one in the SLA.
3. isi sync policies view — is accelerated_failback enabled? No → budget for a full differential rebuild on the way back.
4. isi sync recovery allow-write <policy> on the DR cluster — makes the target writable. The relationship is now broken. There is no undo that does not involve a resync.
5. Repoint clients: update the SmartConnect DNS delegation to the DR cluster's service IP. DNS TTL now governs how fast users come back — check it in advance, not now.
6. isi auth status -v run on the DR cluster — are AD and LDAP online from Phoenix? This is the step that catches the single-DC gap. If the only domain controller lives at the failed site, Phoenix has current data and can authenticate nobody — the failover produces an outage with extra steps. Establish this before the disaster; there is no fixing it during one.
7. Confirm DNS: does anything outside Site A still resolve the SmartConnect delegation? A delegation served only from the failed site's DNS makes the DR name unresolvable no matter how healthy the cluster is.
8. When the primary returns: isi sync recovery resync-prep <policy> to stage the failback, and re-verify before repointing DNS again.

14Escalation collection

When a case goes to Dell, the quality of your evidence sets the pace of the resolution. Collect before you remediate — the moment you restart a daemon, the state that would have explained the fault is gone.

isi_gather_info Read-only

The full diagnostic bundle — the direct analogue of a switch's show tech-support. Logs, configuration, hardware state and statistics from every node, packaged and optionally uploaded to Dell. This is what Support expects attached to a case. It is large and it takes time; start it early.

boston-1# isi_gather_info
# ... writes to /ifs/data/Isilon_Support/pkg/
boston-1# ls -lh /ifs/data/Isilon_Support/pkg/
-rw-r--r--  1 root  wheel   412M Jul 13 09:14 boston-20260713-091402.tar.gz

isi_gather_info --local-only Read-only

Restricts collection to the node you are on. Much faster, and appropriate when the fault is unambiguously node-local — but confirm with Support that a local gather is acceptable before you rely on it.

isi_gather_info --local-only

Gathering local node only ...
  Package: /ifs/data/Isilon_Support/pkg/boston-1-20260713-091402.tar.gz (86M)

isi_gather_info --nologs Read-only

Configuration and state without the log payload. Useful when you need a fast configuration snapshot for a design review rather than a fault investigation.

isi_gather_info --nologs

Gathering configuration and state (no logs) ...
  Package: /ifs/data/Isilon_Support/pkg/boston-config-20260713.tar.gz (12M)

isi upgrade patches list Read-only

Attach this to every case. Support's first act is to check whether your symptom matches a known issue already fixed in a patch you have not applied.

isi upgrade patches list

IOCA — on-cluster analysis Support-directed

Dell's on-cluster health analyser. It reports known-issue exposure for your specific OneFS version. Fetch it, stage it in the support directory, and run it against your running version.

# Fetch and stage:
boston-1# curl --disable-epsv -O ftp.emc.com/pub/rcm/Isilon/tools/IOCA
boston-1# scp IOCA root@<cluster-mgmt-ip>:/ifs/data/Isilon_Support/
# Run against the running OneFS version:
boston-1# perl IOCA -u 9.5.0.0

Log locations worth knowing Read-only

When you need to grep rather than gather.

/var/log/messages          # general system
/var/log/lsassd.log        # authentication / AD secure channel
/var/log/lwiod.log         # SMB I/O daemon
/var/log/isi_job_d.log     # job engine
/var/log/isi_migrate.log   # SyncIQ

Operating notes

A healthy cluster and a healthy service are different claims. isi status can report OK while a wedged lwio on one node hangs every client SmartConnect happened to land there. Cluster health is a statement about the cluster's opinion of itself, not about whether users can open a file.

Omitting -n on isi_for_array is the single most expensive typo on this platform. It converts a node-scoped recovery into a cluster-wide outage. Build the habit of typing the -n flag first, before you type the command it will run.

Prefer lwsm restart over killall. killall -6 aborts a daemon and forces a core dump; the service-manager restart stops and starts it cleanly. Use the abort only when Dell wants the core file. Most operators reach for killall because it is what they found in a forum thread, not because they need the dump.

Capture evidence before you remediate. Restarting the daemon fixes the symptom and destroys the diagnosis. If you want the fault to stop recurring, gather first — isi_gather_info, session counts, listen queue overflows — and restart second.

Read the actual protection column, never the requested one. OneFS will accept a protection level your node count cannot deliver and quietly under-protect the data. A 3-node cluster asking for +2n is the textbook case: no error, no refusal, just a gap between what you asked for and what you have.

Disabling a CloudPools account is indistinguishable from data loss, from where the user sits. Stubs go unreadable instantly and en masse. Nothing is deleted and re-enabling restores access, but expect a P1 in the intervening minutes. Know what is stubbed against an account before you disable it.

allow-write is a disaster declaration, not a DR test. It breaks the replication relationship and commits you to a failback cycle. If you need to prove DR works — and you should, regularly — snapshot the target and mount it read-only. Do not satisfy an audit checkbox by breaking your own DR.

Verify identity before you touch an ACL, and pass --zone. A large share of ACL changes are made to solve problems that were never permission problems — the user simply was not in the group everyone assumed, or was resolving to a different token in a different access zone.

chmod -R -b has no undo, and ls -leRd is the only record you will get. Redirect it to a file before you strip ACLs. That file is your rollback plan, and it is the entire rollback plan.

FAQFrequently asked questions

RFReferences

1. Dell PowerScale OneFS Documentation — CLI Command Reference and Administration Guide
2. Dell Technologies Info Hub — PowerScale technical white papers

→From the same practice

• Cisco MDS NX-OS Command Reference
• Post-OEM storage maintenance
• Managed infrastructure services

Running PowerScale or Isilon in production?

Bring us the outputs this page taught you to collect. WUC storage engineers deliver cluster health assessments, protection and SmartPools reviews, SyncIQ and DR validation, performance investigations, and migration planning — for PowerScale and Isilon estates inside and outside OEM support, under post-OEM storage maintenance.

• Cluster health assessment
• Protection & SmartPools review
• SyncIQ / DR validation
• Migration & refresh planning

Please enable JavaScript in your browser to complete this form.

Full Name *

Company

Email *

Phone Email Company

Phone

OEMs in your estate + next renewal window

e.g. Cisco, Dell, NetApp - and when your next contract renews.

Request my assessment

Prefer to talk it through first? Book a technical consultation → · View managed services

No prep slides required · References available under NDA · Multi-OEM coverage across Dell EMC, HPE, IBM, NetApp

All hostnames, domains, users, groups, paths, serial numbers and IP addresses in this guide are synthetic reference values. No customer environment is depicted.