Sun 7110 port aggregation to Cisco Catalyst 2960
We've been running our Sun 7110 with a 2-port aggregation, LACP configured with L3 policy, mode "off" (because the Ciscos weren't configured to do LACP). The 2 links were connected 1-each to a pair of Cisco 2960s. Based on this config, I had assumed we would get load balancing when both links are up, and "dumb" redundancy based on the Sun detecting one of the links being down. Not optimal, but better than a single link. However, we discovered that when one of the links was mistakenly disconnected, we lost connectivity completely. We had to reconnect the link to restore connectivity.
Based on this behavior, it seems like there is not much value to allowing an aggregation that specifies LACP mode "off". So, we're looking at the Cisco 2960 docs for enabling LACP, and from what I read, the Cisco wants to establish an LACP group of which one port will be active and the others will be hot spares. If this is the only option for LACP configuration, then how would we achieve load balancing and higher throughput using LACP?
Can anyone advise what a reasonable config would be to achieve load balancing and redundancy, while making use of multiple interfaces to achieve higher throughput (with or without LACP)? Ideally, we would like to only advertise a single IP address for NFS connections, but this is not a show stopper.
I'm not a network admin, so if the above has left out any critical data, please let me know. Thanks!