There are several ways of doing this. One of the easiest and straightforward ways is 'TEQL' - "True" (or "trivial") link equalizer. Like most things having to do with queueing, loadsharing goes both ways. Both ends of a link may need to participate for full effect.
Imagine this situation:
+-------+ eth1 +-------+
| |==========| |
'network 1' ----| A | | B |---- 'network 2'
| |==========| |
+-------+ eth2 +-------+
A and B are routers, and for the moment we'll assume both run Linux. If traffic is going from network 1 to network 2, router A needs to distribute the packets over both links to B. Router B needs to be configured to accept this. Same goes the other way around, when packets go from network 2 to network 1, router B needs to send the packets over both eth1 and eth2.
The distributing part is done by a 'TEQL' device, like this (it couldn't be easier):
# tc qdisc add dev eth1 root teql0 # tc qdisc add dev eth2 root teql0 # ip link set dev teql0 up
Don't forget the 'ip link set up' command!
This needs to be done on both hosts. The device teql0 is basically a roundrobbin distributor over eth1 and eth2, for sending packets. No data ever comes in over an teql device, that just appears on the 'raw' eth1 and eth2.
But now we just have devices, we also need proper routing. One way to do this is to assign a /31 network to both links, and a /31 to the teql0 device as well:
FIXME: does this need something like 'nobroadcast'? A /31 is too small to house a network address and a broadcast address - if this doesn't work as planned, try a /30, and adjust the ip adresses accordingly. You might even try to make eth1 and eth2 do without an IP address!
On router A:
# ip addr add dev eth1 10.0.0.0/31 # ip addr add dev eth2 10.0.0.2/31 # ip addr add dev teql0 10.0.0.4/31
On router B:
# ip addr add dev eth1 10.0.0.1/31 # ip addr add dev eth2 10.0.0.3/31 # ip addr add dev teql0 10.0.0.5/31
Router A should now be able to ping 10.0.0.1, 10.0.0.3 and 10.0.0.5 over the 2 real links and the 1 equalized device. Router B should be able to ping 10.0.0.0, 10.0.0.2 and 10.0.0.4 over the links.
If this works, Router A should make 10.0.0.5 its route for reaching network 2, and Router B should make 10.0.0.4 its route for reaching network 1. For the special case where network 1 is your network at home, and network 2 is the Internet, Router A should make 10.0.0.5 its default gateway.
Nothing is as easy as it seems. eth1 and eth2 on both router A and B need to have return path filtering turned off, because they will otherwise drop packets destined for ip addresses other than their own:
# echo 0 > /proc/net/ipv4/conf/eth1/rp_filter # echo 0 > /proc/net/ipv4/conf/eth2/rp_filter
Then there is the nasty problem of packet reordering. Let's say 6 packets need to be sent from A to B - eth1 might get 1, 3 and 5. eth2 would then do 2, 4 and 6. In an ideal world, router B would receive this in order, 1, 2, 3, 4, 5, 6. But the possibility is very real that the kernel gets it like this: 2, 1, 4, 3, 6, 5. The problem is that this confuses TCP/IP. While not a problem for links carrying many different TCP/IP sessions, you won't be able to to a bundle multiple links and get to ftp a single file lots faster, except when your receiving or sending OS is Linux, which is not easily shaken by some simple reordering.
However, for lots of applications, link loadbalancing is a great idea.