Migrating Pure to bare metal  

  RSS

YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
04/07/2019 11:09 am  

We thought it would be a good idea to move our Pure installation to bare metal. We have a Xeon box running Ubuntu 18.04 with a rather large RAIDz3 pool. Currently we have a dataset that is exported via NFS to our Pure VM (so all our backups are already local to this machine):

riesenpool/pure 3.97T 4.03T 3.97T /mnt/pure

As I understood we can now use the very same dataset with our future local Pure installation (which is great of course). Are there best practices for a Pure installation on Ubuntu Bionic? We have two 10 GbE NICs in this machine with one currently in use. Do you recommend to use a dedicated 10GbE NIC for backup traffic between Pure and vCenter (is it vCenter? Which kind of data is this? HTTP chunks? Streams?)

This is our top priority right now (moving the Pure installation to the physical Ubuntu Bionic host). But related (and with way lower priority) I wonder about the capabilities we have to access the VM’s filesystems. When trying to access VM filesystems in the Pure VM I managed to somewhat ‚crash‘ the Pure installation and needed to restart nsd manually. But I guess it’s not worth trying to debug this since it’s 2 weeks ago and I didn’t kept logs. So most probably once we got Pure up and running on the Ubuntu Bionic machine I’ll give it another shot and then get back to you with details (and some questions about the layout of the images and how your nbd ‚plugin‘ works to access the contents).

Thanks in advance and best regards


Quote
marijan
(@marijan)
Member Admin
Joined: 5 months ago
Posts: 18
04/07/2019 2:30 pm  

Installing Pure on a bare metal Linux should be super simple - just create a folder somewhere, extract the content of vmsn.2.0.0.tgz and run start.sh to configure auto startup and run the software. During the initial configuration wizard, select the existing backup repository and you are done.

As for the network performance best practices, there is very little we can actually tweak. Pure uses the official VMware transport libraries as described here:
https://pubs.vmware.com/vsphere-6-5/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkDataStruct.5.5.html

It is a closed source modification of the NBD protocol. A potential way of increasing performance can be found in this (very old) KB article but you won't know the actual results unless you test.
https://kb.vmware.com/s/article/2052302

If you have a dedicated 10GbE NIC you can spare just for Pure backups, that will surely increase performance but the actual amount will again depend on your actual system (maybe the network load is low enough that backups wouldn't be impacted even if they shared the same NIC with other tasks). There are some built in limitations to the network transfer protocol on VMware's side but this can be mitigated by executing several backups in parallel.

As for our own File restore and its use of NBD, we can discuss this further when you reach that point. Some related bugs were corrected there recently so it might be that you were simply using an older version at that time. If the problems occur again, we will investigate.


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
04/07/2019 3:10 pm  

Just some quick (positive) feedback. 

As you described installing Pure on the bare metal machine worked flawlessly. I first deployed the beta version 1.9.50 and after pointing it to the already existing dataset with the backups everything was there (and ‚Pure security best practices‘ went on my low priority TODO list since there was also no need to re-enter logon credentials and so on. But that’s another topic for later).

Unfortunatenly all backups failed (see below) so I decided to move the 1.9.50 install from /usr/local/pure to /usr/local/pure-debug and moved over to 2.0.0 in /usr/local/pure, this time using a different repository (/mnt/pure-backups instead of the former /mnt/pure where all the old backups made with your Pure VM resided).

Still same situation: I started 3 backups manually but they all failed with same error message about needing to consolidate disks.


ReplyQuote
marijan
(@marijan)
Member Admin
Joined: 5 months ago
Posts: 18
04/07/2019 3:40 pm  

Is it possible that you are still running Pure VA in parallel with your new Pure installation?
All the backup failures were caused by virtual disks being locked. Most likely they are attached somewhere else - e.g. Pure VA during backup. This is further confirmed by the fact that Pure is getting already snapshotted disks paths (e.g. [ds1] nix-file5/nix-file5-000002.vmdk) instead of flat disks (without the last -0000002 part).
If that is the case, please make sure that two Pure instances are not trampling over each other.

As for the VMware logon credentials, they are indeed saved in Pure configuration (not in plain text). When your new Pure installation was pointed to the existing backup repository, it read the existing configuration along with access credentials.


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
04/07/2019 3:50 pm  

Is it possible that you are still running Pure VA in parallel with your new Pure installation?

No, I powered off the Pure VM before. But maybe there was a short timeframe where both the Pure VM was still running and I had already pointed the bare metal installation to the same repository.

All the backup failures were caused by virtual disks being locked. Most likely they are attached somewhere else - e.g. Pure VA during backup. This is further confirmed by the fact that Pure is getting already snapshotted disks paths (e.g. [ds1] nix-file5/nix-file5-000002.vmdk) instead of flat disks (without the last -0000002 part).
If that is the case, please make sure that two Pure instances are not trampling over each other.

Well, currently there is only one active but your remarks lead me to thinking about prior failures inside the Pure VM. I will check up on this.

As for the VMware logon credentials, they are indeed saved in Pure configuration (not in plain text). When your new Pure installation was pointed to the existing backup repository, it read the existing configuration along with access credentials.

Ok, this explains it. And then after deleting the former primary repository (/mnt/pure) and then creating the other at /mnt/pure-backups the configuration has been saved there, I would assume. Great for disaster recovery and stuff but I guess we need to look a bit closer how this is stored — postponed, now it’s about getting everything up and running 🙂


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
08/07/2019 2:00 pm  

Out of curiosity we shutdown Pure on ‚datengrab‘ (the bare metal install) again, fired up the Pure VM and let one VM being backed up. Worked flawlessly.

What I fail to understand is why there are no open network connections to both the vcenter host and the ESX server in question (dell-2 in this case).

What’s different here when Pure is running as appliance and bare metal?


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
08/07/2019 2:21 pm  

Ok, forget about my previous post since now I understand how Pure deals with this. The Pure VM got the respective .vmdk assigned locally.

That’s now:

root@pure:~# cat /proc/partitions 
major minor #blocks name

7 0 91524 loop0
7 1 91388 loop1
7 2 90560 loop2
8 0 6291456 sda
8 1 1024 sda1
8 2 6288384 sda2

And this is how it looked before while the backup was running:

root@pure:~# cat /proc/partitions 
major minor #blocks name

7 0 91524 loop0
7 1 91388 loop1
7 2 90560 loop2
8 0 6291456 sda
8 1 1024 sda1
8 2 6288384 sda2
8 16 104857600 sdb
8 17 104847360 sdb1
8 25 8192 sdb9

So locally attached as /dev/sdb. Now I understand what’s different 🙂


ReplyQuote
marijan
(@marijan)
Member Admin
Joined: 5 months ago
Posts: 18
08/07/2019 2:23 pm  

One quick thing you can try is to use the actual 'root' credentials for your ESXi (that is, if you are using 'Administrator' or some other similar custom role).
If this is the problem, then we just need to investigate what permission is missing.
According to a VMware forum post I found, the 'Nfcsvc' plugin is not started on ESXi server when non-root user logs in.


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
08/07/2019 4:41 pm  

I removed the vCenter but had already set up one ESXi with root credentials in the server infrastructure before.

A job was terminated after 5 minutes (instead of 15) with the same error 14009.

The VMs that want to back up Pure all get in vCenter the status "Consolidation needed". Sometimes I can consolidate with no error, sometimes I get the message, that the files are locked and can't be deleted.

I've restarted the management agents on dell-1 where VMs with the error resides. At least it helped to consolidate the VMs.


ReplyQuote
YannB
(@yannb)
Active Member
Joined: 5 months ago
Posts: 10
09/07/2019 9:45 am  

Our Pure runs as virtual machine again. We changed the default route in vCenter for the ESXis to match the networks.

First manual backups ran without errors. In the next period (Saturday 00:00) all VMs were successfully backed up. A manual backup of a new VM (nix-mail) ran without problems.
The backups from Sunday (starting at 00:45 a.m.) were also error-free (including verify).

This morning all VMs except two were backed up. The error message is new (there are none) says that none of the disks of nix-mail2 and nix-office2 could be proceeded. We have to grey marked vms, mac-office and mac-external who also are green.

Investigating the network connections between Pure and vSphere, we discovered that our vSphere network settings are not correct. Everything works on VMware side but the settings are not configured as they should be. Seeing how we are going on vacation soon, we will postpone network reconfiguration for later.

We rely on Pure for our VM backups and this needs to run flawlessly so we will keep the Virtual appliance version for the time being.


ReplyQuote
marijan
(@marijan)
Member Admin
Joined: 5 months ago
Posts: 18
09/07/2019 12:07 pm  

I'm glad that Pure is working for you and I'm sure you will manage to fix the bare metal installation. Just the other day a customer installed Pure on a bare metal Debian 9 with local NFS storage used as backup repository (they used to run several instances of Pure 1 sending data to that same NFS), 3 separate ESXi servers and a vCenter VM running on one of them, all the ESXi hosts only use their local storage for now. Quite similar setup to what you have and it works out of the box for a week now.

As for the error you encountered in your last backup, that is a known bug that was being fixed these days - there was a memory leak so Pure would use excessive amounts of RAM if GUI was left open for a long time. This is now fixed and I want to thank you guys for helping us detect it in the first place.

Please update your Pure installation to version 2.0.1

 


ReplyQuote
Share:

Please Login or Register