I hope someone can help me on my problem with an aggregate, as I am a Solaris noob. I tried doing this according to the official documentation from Oracle (unfortunately, as a new user to the forum, I may not post URLs...) and also googled a lot around, but have not found any solution yet.
I am in the process of building a Solaris NAS system. I have installed Solaris 11 (not express, as it is not available anymore). I have 3 NIC's installed in the system. I want to use one (net3) for the management of the box, and the other two (net0 and net1) as an aggregate for the data traffic to the NAS.
My problem is that the aggregate always shows up as "disabled" after a reboot, and I have to manually enable it (which works without problems, by the way).
So here is what I did:
Disabled nwam: text svcadm disable nwam svcadm enable network/physical:default
removed any previous config from the adapters text ipadm delete-if net0 ipadm delete-if net1
created the aggregate text dladm create-aggr -L active -l net0 -l net1 aggr0
Then, according to the doc, it should be possible to create the address using "ipadm create-addr -T static...", but in my case, this did not work. Instead, I had to plumb the aggregate: text ifconfig aggr0 plumb 192.168.1.224/24
then I was able to bring it up - manually. I am aware that this command is only temporary and not persistent.
I then tried to find a solution here and there, and after a lot of googling and trying around, I found the ipadm to work all of a sudden - honestly, I don't know why :o text ipadm create-ip aggr0 ipadm create-addr -T static -a 192.168.1.224/24 aggr0/v4
That's as far as I got. Oh yes, my hosts file reads:
After each reboot, I see:
I can then bring the interface up with simply ipadm enable-if -t aggr0, however, as stated, it's still not persistent.
When googling, I learnt that in earlier versions of Solaris, there was a file for each interface like hostname.interface . This seems not to be the case anymore, according to the documentation from Oracle. Nevertheless, I have tried it just to be sure, and so I created a file /etc/nas1.aggr0, however, this did not work as well.
Yes I have tried this, but rather by accident - I simply forgot "-t" several times. I then always get a message stating that "this operation is not allowed on the adapter" or something like that... I have to look up the exact wording tonight when I get home from work. I'll post it here afterwards.
Thanks,
Cap
My co-worker just just did this on a Solaris 10 server.
He did it like this
dladm create-aggr -L active -l net0 -l net1 0
Also the hosts file like this.
::1 nas1 localhost
127.0.0.1 nas1 localhost loghost
192.168.1.222 nas1
192.168.1.224 nas1aggr <--is this the right hostname?
My co worker made 4 ports on 2 different NIC that the same IP.
I've setup several aggregates in the past. Just to make sure, are you working with your network team on this? The aggregate has to be setup on the switch end as well or it won't work. Both connections have to be going to the same switch. I've had trouble in the past with network guys who didn't know how to set it up properly on their end.
Sometimes the network guys also won't know what you mean by "aggregate" since it can sometimes go different names depending on vendors. I had to actually send our network guy the Wikipedia link so that he could go research how to do it on their Cisco switch -- Link aggregation - Wikipedia, the free encyclopedia
Depending on the network infrastructure in the data center sometimes network teams will freak out on you about this as well. I tried to set it up one data center, and the network team rejected it because they only had 2 GB links going between their switches in the data center and they were concerned I would flood their links.
Good luck! Once it's setup it works like a charm, and is persistant across reboots. I did this for a backup server in one data center that was getting bogged down at night due to so much traffic from various servers. After I set that up it knocked the backup times almost in half. It was awesome watching how fast the data was transferring.
Example syntax would be similar to this:
dladm create-aggr -d bge0 -d bge1 2
dladm show-aggr ### (to verify you set it up properly)
#### (Then do your ifconfig commands as necessary, you shouldn't have to reboot to make sure it's working.
# Once you know it's working update your /etc/hostname.* files in Solaris 10 or ipadm in Solaris 11 and
# reboot to make sure it continues to work across reboots.)
### Example:
ifconfig aggr2 plumb
ifconfig aggr2 ................
Hi christr
Thanks for your reply.
As posted from the beginning, I don't have a problem creating the aggregate - this works fine! The aggregate comes up and responds to ping, no problem here at all. I have also stated all steps that led me there, I am pretty sure they're following the Oracle documentation. I do have Solaris 11, so I have to work with ipadm and dladm.
Also, this is going to be a home NAS, no datacenter or networking guys included, but as I said, the aggregate works!
...it's just not persistent. After each reboot, it comes up like this:
I can then enable it like this:
Of course, the "-t" is for temporary, so this won't stay. It won't accept the command without "-t", stating "ipadm: persistent operation not supported for enable-if"...
This is driving me nuts. :wall:I tried creating /etc/nas1.aggr1 and put the IP in there, but it doesn't help - after a reboot, it's disabled again. This is especially frustrating, as I am able to create the aggregate and bring it up - manually.
If you can help on this.. please do, I'm really desperate by now...
That's interesting. I haven't had a chance to really work with Solaris 11 yet. Doing an init script is a good work around, but it's strange you have to do that. Maybe it's a bug with ipadm being so new.
Just curious, but for a home server is it even capable of those aggregate speeds? I run Solaris in my home server inside ESXi along with several flavors of Linux virtual machines. It's an Intel Q6600 with 6 gigs of RAM. The PCI bus can't reach over about 45 MB/sec for the I/O, so it never even reaches the full potential 100 MB/sec a gigabit provides. And is the home NAS cabable of going over 100 MB/sec also? I personally have never seen those kinds of speeds outside of SPARC or maybe Xeon hardware. Just curious, as it would be interesting to know for my future upgrades. I can see how you would want to set that up at the house just because you can too, as I certainly have overkill stuff like that too.
This also reminds me I need to upgrade my home Solaris machine to Solaris 11. We'll still be on Solaris 10 for the next few years at my work.
I assume it is a wee bit overkill, as you said... However, I didn't want to take a risk and since my hardware supports aggregating NIC's, I wanted to do it from the beginning instead of running short one day and having to add it later... I guessed this would be the bigger effort.
I have a separate VMWare ESXi Server with currently 5 guests on local disks. I want to run the VM's on this NAS, and also need around 2 TB of CIFS space for my movies. Today, if I am using the VM's heavily, the performance on the CIFS shares sometimes drops significantly, as they are now stored in the vmdk's. Also, I don't know how much impact has deduplication, compression, snapshotting etc.
For comparison, I use the following hardware:
Supermicro X8SIE-F mainboard
12 GB ECC RAM
Xeon L3426 CPU
LSI 9211-8i Controller flashed with the IT firmware
This card has a PCI-Express x8 Interface.
2 x 320 GB HDD's in a ZFS mirror for OS
7 x 1 TB Samsung F3's for data, one being spare
The PCI Express 2.0 x8 Interface has a theoretical throughput of 16Gbit/s or 2 GB/s, according to Wikipedia, so I don't expect a bottleneck here. Of course, the performance of the card being able (or not) to use this bandwidth is a totally different story. After all, with 6 active disks, there should be plenty of performance for my needs, assuming the card can hold up on this workloads. I'll see it going on with my NAS configuration, and will let you know about my experiences here.
Regards,
Cap'
I would do it too if the hardware supported aggregation. After all, why not if the hardware can do it.
That sounds like a cool setup. I run everything inside my ESXi with 4 guests. Solaris 10, Ubuntu 11, CentOS 5, and a Windows VM. Solaris 10 acts as the router, DHCP server, firewall, etc... for the entire network. Ubuntu is my media and file server with two 600 GB disks for storage of video, family photos, music, documents, data, backups of home PCs, etc.... It has PS3 Media Server, qTorrent, Samba, NFS server, and MiniDLNA always running on it too. I have it rsync the data once a week for backup purposes. The CentOS just sits there running for when I feal like doing something in a Red Hat distro, and Windows runs a few tools like DynDNS and OpenDNS that don't have Unix versions of their tools. I also use the Windows VM for a few other things form time to time like being able to pull up a file on Dropbox remotely that I need a Windows PC for from my phone for stuff like Quicken, etc...
I plan to build a new home server sometime in 2012, as this current machine is now 4 years old and 600 gigs isn't cutting it anymore for storage. I'll probably do MicroATX with two 2 or 3 TB disks and setup ZFS mirrors on seperate vmdks on each physical disk, and then to the same with LVM for Linux VMs. Won't bother mirroring Windows, as who cares about Windows anyway.... :). I'll probably wait to do Solaris 11 when I build the new machine. It's all so time consuming.
Interesting.
I mirrored Solaris, btw, not Windows.
If you want to set up a Solaris 11 home NAS, I definitely recommend you this article OpenSolaris derived ZFS NAS/ SAN (Nexenta*, OpenIndiana, Solaris Express) - [H]ard|Forum
This is were I got the idea from, and it's what I intend to build (with napp-it, that is). However, you still need to invest a LOT of time...
Good luck,
Cap'
Installed it last night, went absolutely smoothly, no prob at all.
Setting up SMB and NFS for my VMWare was a breeze, performance looks pretty good. My pool consists of 6 x 1 TB Samsung Disks in a simple RAIDz (will add another one as hotspare as soon as I have migrated the data from it).
Definitely recommend napp-it to anyone who isn't 100% experienced with Solaris Commandline! Napp-it rocks!