FPS Drop Issues

Specs:
OS: Windows 7 64-bit
Graphics Card:ATI 4890 1gb
Memory: 4gb Ram
CPU: Amd Phenom II X4 955 Processor 3,1ghz orsmth
Harddrive: 7400rpm 1gb

I do have 170fps in town, and around 100 fps when playing alone, and 40 fps when playing with others. But the problem is the drops... It drops to 5 fps, even lower,and it stays there alot. And it is very variable from who i play with and not..

The specs isnt the best but..
This comp played Skyrim, Fallout 3 etc on full graphics

Its mostly the same no matter graphics i chose, the highest or the lowest..

I probably get a little less lagg with lower graphics and resolution, but its almost not noticeable

Please help me

Also have to say i just format the computerto see if it got better, it didnt..
Downloaded ATI drivers alredy, and directx. also tried to download the newest drivers on the motherboard

I'm having the same here