Hi all,
Whenever I mount a remote share via NFS I can no longer access the web interface, once I log in an error pops up which if I hit retry just loops. This had worked in the past.
Session Expired
Server session has expired.
Please login again.
Internal Server Error
My NFS mount looks like this:
10.1.1.90:/pure_backup on /remotebackups type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.1.10,local_lock=none,addr=10.1.1.90)
The NFS share is sat on a dedicated 10 gig link, I can browse the share and create/delete files as the pure user from the cli just fine.
root@pure:~# ls /remotebackups/
PureLog.db
pure.reserved
SpProps.db
VM-03811701-bc01-43d1-7763-06cbc3e3c93f
VM-0673b726-1dd2-4f5c-5ec5-24886912936e
VM-0b8cfa42-c24c-48e5-4b33-caac40ba80ae
...etc...
root@pure:~# ifconfig
ens160: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.1.106 netmask 255.255.255.0 broadcast 172.16.1.255
ether 00:50:56:94:15:52 txqueuelen 1000 (Ethernet)
RX packets 37722 bytes 54241988 (54.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 24030 bytes 35322298 (35.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0ens193: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.1.1.10 netmask 255.0.0.0 broadcast 10.255.255.255
ether 00:50:56:94:18:14 txqueuelen 1000 (Ethernet)
RX packets 69021 bytes 179326388 (179.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 76449 bytes 19931870 (19.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 154 bytes 12157 (12.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 154 bytes 12157 (12.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0root@pure:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 100 0 0 ens160
10.0.0.0 0.0.0.0 255.0.0.0 U 101 0 0 ens193
172.16.1.0 0.0.0.0 255.255.255.0 U 100 0 0 ens160root@pure:~# systemctl status archiware-pure.service
● archiware-pure.service - Archiware Pure
Loaded: loaded ( removed link ; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-03-09 12:00:47 CET; 1h 10min ago
Process: 2852 ExecStop= removed link (code=exited, status=0/SUCCESS)
Process: 2959 ExecStart= removed link (code=exited, status=0/SUCCESS)
Main PID: 2991 (nsd)
Tasks: 17 (limit: 2315)
CGroup: removed link
├─2989 /usr/local/pure/vmsn/bin/nsd -w -u root -t etc removed link
└─2991 /usr/local/pure/vmsn/bin/nsd -w -u root -t etc removed linkMar 09 12:00:41 pure systemd[1]: archiware-pure.service: Killing process 2726 (n/a) with signal SIGKILL.
Mar 09 12:00:41 pure systemd[1]: Stopped Archiware Pure.
Mar 09 12:00:41 pure systemd[1]: Starting Archiware Pure...
Mar 09 12:00:41 pure start.sh[2959]: Starting Archiware Pure application server, be patient...
Mar 09 12:00:41 pure nsd[2989]: watchdog: started.
Mar 09 12:00:41 pure nsd[2991]: server: started.
Mar 09 12:00:47 pure start.sh[2959]: Archiware Pure pid: 2991 - running.
Mar 09 12:00:47 pure start.sh[2959]: Started Archiware Pure application server.
Mar 09 12:00:47 pure systemd[1]: archiware-pure.service: Supervising process 2991 which is not our child. We'll most likely not notice when it exits.
Mar 09 12:00:47 pure systemd[1]: Started Archiware Pure.root@pure:~# qperf 10.1.1.90 tcp_bw tcp_lat
tcp_bw:
bw = 1.24 GB/sec
tcp_lat:
latency = 51.7 usroot@pure:~# iperf3 -c 10.1.1.90
Connecting to host 10.1.1.90, port 5201
[ 4] local 10.1.1.10 port 52226 connected to 10.1.1.90 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.13 GBytes 9.73 Gbits/sec 7 805 KBytes
[ 4] 1.00-2.00 sec 1.15 GBytes 9.88 Gbits/sec 0 919 KBytes
[ 4] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 0 971 KBytes
[ 4] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 998 KBytes
[ 4] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1006 KBytes
[ 4] 5.00-6.00 sec 1.15 GBytes 9.91 Gbits/sec 0 1006 KBytes
[ 4] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1006 KBytes
[ 4] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 1006 KBytes
[ 4] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.11 MBytes
[ 4] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.11 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 11.5 GBytes 9.88 Gbits/sec 7 sender
[ 4] 0.00-10.00 sec 11.5 GBytes 9.88 Gbits/sec receiveriperf Done.
If I unmount the share the issue goes away and I can access the web interface, but obviously my backups/repository is missing.
Any suggestions where I can start looking for the cause?
Cheers,
Mike.
Is there anything of interest in Pure logs at '/usr/local/pure/vmsn/log'?
Session expiration could be due to some error causing timeouts or outright breaking something that GUI does not expect to get broken.
Is there anything of interest in Pure logs at '/usr/local/pure/vmsn/log'?
Session expiration could be due to some error causing timeouts or outright breaking something that GUI does not expect to get broken.
Here's what I found in vmsn. log
[09 removed link ][-conn:vmsn:default:19:87-] Warning: page/procedure removed link ran with error
[09 removed link ][-conn:vmsn:default:19:87-] Warning: server session restarted ...
[09 removed link ][-conn:vmsn:default:19:87-] Warning: invalid value "" for number
: while executing
: "$j map_open map_key id string $nextCell map_key xname string $dataArray(xname) map_key host string $dataArray(host) ..."
: ("foreach" body line 3)
: invoked from within
: "foreach nextCell [array names tblRow] {
: array set dataArray $tblRow(${nextCell})
: $j map_open map_key id ..."
: invoked from within
: "ns_tdp_return 200 text/html {##############################################################################
: #
: # vm_board.tdp
: #
: # page to respond to..."
: (procedure "::vmsn:: removed link " line 2)
: invoked from within
: "$cmd"
: ("eval" body line 1)
: invoked from within
: "eval {$cmd} $args"
[09 removed link ][-conn:vmsn:default:19:88-] Notice: Executed from production environment
[09 removed link ][-conn:vmsn:default:17:102-] Error: ::vmsn:: removed link : invalid value "" for number
[09 removed link ][-conn:vmsn:default:17:102-] Warning: page/procedure removed link ran with error
[09 removed link ][-conn:vmsn:default:17:102-] Warning: server session restarted ...
[09 removed link ][-conn:vmsn:default:17:102-] Warning: invalid value "" for number
: while executing
: "$j map_open map_key id string $nextCell map_key xname string $dataArray(xname) map_key host string $dataArray(host) ..."
: ("foreach" body line 3)
: invoked from within
: "foreach nextCell [array names tblRow] {
: array set dataArray $tblRow(${nextCell})
: $j map_open map_key id ..."
: invoked from within
: "ns_tdp_return 200 text/html {##############################################################################
: #
: # vm_board.tdp
: #
: # page to respond to..."
: (procedure "::vmsn:: removed link " line 2)
: invoked from within
: "$cmd"
: ("eval" body line 1)
: invoked from within
: "eval {$cmd} $args"
One of the configuration settings has an invalid value - it looks as if it is "" (empty) whereas Pure expects a number. The tricky part is going to be detecting and fixing this issue if you are not able to access GUI in the first place.
If TeamViewer is ok for you, I'll send you a private message so we can connect and fix the offending value.
Yeah that would be great, please feel free to drop me a PM.
Cheers.
