I have a very strange problem here, been working on it for 4 months now,
hopefully someone can give me some advice.
Anyways, I have a client with 200 users running Citrix. The network
configuration is:
all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
2 domain controllers - win'2000
4 Citrix servers
1 SQL server
1 Print Server
1 EMC Clarian SAN
1 Lotus
1 HP-UX running BPCS
Switches:
4 Intel Express Hubs 12 Port
2 3COM hubs
2 HP 2524 Switches
2 Cisco Catalyst 2900 Switches
1 Cisco Catalyst 3500 Switch
4 Old SMC Hubs for the phone system
Anyways, the client was running old servers, so they upgraded to HP DL380
servers, and Citrix ran great, then a few days later it went to crap. The
logins were slow, trying to open office documents through office was slow
(but opening through explorer was not), and when typing, it would lag real
bad. (This is Citrix Specific)
I came in to try to fix it, and we tried everything until we finally rebuilt
the domain controllers and Citrix farm, since the problem really seemed like
a server issue. Well, after rebuilding from scratch, we stil had the same
problem, so we tested everything, and it looked good. I then got an idea, it
felt like it might be network traffic.
I took a catalyst 2900 switch that had never been connected to the network,
and ran the servers to it, and 6 thin clients. We turned it on, and the
thing cruised. It rocked, logins were instant, it was the first time it
worked in 4 months. So, I figured I would track down the problem, I started
plugging in switches one at a time until the problem surfaced. I got all the
Intel hubs connected, nothing, then I connected the 3coms, still nothing,
and then the HP switches, so far so good.
By now, its 1:00 AM, and I decide to go home, I quickly connect in the rest
of the switches at once, and BOOM, the problem shows up, so I figure I have
it down to only a few switches, lets go home and try tomorrow. I move the
network cables over to the old servers, then I Connect this switch into the
network, and leave. (It was easier, than moving cables everywhere, since the
switch room is seperate from the server room, and I was tired)
Next Day, I'm excited, so I decide to show it off. WELL, I move my network
cables to the old servers, and unplug the switch from the rest of the
network. The problem is back!!! I reboot the switch, I reboot the servers,
nothing. I even swap network cables. This is the exact same configuration as
last night when it was running great. I dont get it. I guess maybe something
from one cisco switch has infected this one, who knows.
I do know that the HP switches didnt cause a problem, so I empty them out,
and plug in only my servers and thin clients, nothing else, completely
seperate from the network. STILL THE SAME PROBLEM!!
This is weird, and I need to fix it. Its like whenever you add a switch, it
acquires this problem, but a brand new switch runs great until it is
connected to the network for any length of time. It ran great on the other
switch for a good 5 hours, so I know its not the servers, gotta be the
switch. Does anyone have any ideas?
I am resetting my catslyst to factory default then going to try it. Someone
please help, I need this up by monday.
Here are a few things I caught using ethereal, if anyone can tell me what it
means. Everything looks like normal traffic, except from one PC, and the
citrix boxes.I can export the file to text and e-mail it if anyone wants
some more detail, just email me at cnash@plego.com.
This is only half of it, but you get the idea, out of hundreds of pc's amd
about 15 random servers, these are the only ones doing this particulaar
thing.
Source Destination
Info
Colleen-pc.company.int 192,168.102.6 TCP
3480 > 5321 [PSH, ACK] Seq=1 Ack=0 Win=54512 Len=120.
Colleen-pc.company.int 192,168.102.6 TCP
3480 > 5321 [ACK] Seq=1 Ack=0 Win=54512 Len=120
Colleen-pc.company.int 192,168.102.6 TCP
3480 > 5321 [SYN] Seq=0 Ack=0 Win=54512 Len=120 MSS=1460
Colleen-pc.company.int 192.168.102.14 TCP
3479 > 1352 [ACK] Seq=1 Ack=0 Win=64512 Len=0
Colleen-pc.company.int 192.168.102.14 TCP
3479 > 1352 [SYN] Seq=0 Ack=0 Win=64512 Len=0 MSS=1460
---------------
Then on Citrix, I have a bunch of these, on all the servers:
Cxp03.company.int 192.168.102.150 TCP
1494 > 1041 [ACK] Seq=0 Ack=0 Win=63412 Len=0
-----------------------------------
Along with a LOT of these:
Cxp03.company.int 192.168.102.150 TCP
[TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=121622 Ack=4049
Win=63783 Len=1459
Cxp03.company.int 192.168.102.150 TCP
[TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=2045131 Ack=22451
Win=63783 Len=1459
