Don't use the task manager to do memory profiling. It doesn't mean what you think it means, especially for C# or other managed applications. ANTS Memory Profiler is a wonderful managed memory profiler that I've used repeatedly. It is not free, but it does have a trial. As Jonathan Dickinson points out in the comments, the CLR itself also exposes several performance counters you can view in perfmon.exe, including several related to .NET memory and garbage collection.
There is a very good chance there's nothing actually rooting your bullet objects once they are removed from that list and that, upon a subsequent collection of the appropriate garbage generation, the memory consumed by the objects will be reclaimed. Without seeing more of your code I can't be sure, but I think it's a safe bet.
However, you are doing something that is, if nothing else, inefficient and you could improve it:
You're removing items from a list, which implies you are allocating them and putting them in the list in the first place -- in other words you are allowing allocation to occur during your game loop, and this means you'll eventually cause garbage collection (and more to the point, possibly cause more frequent collection of higher GC generations). Further, you're removing items from the middle of the list (most of the time), which means every time you remove something, the following elements of the list must be copied.
Consider instead partitioning your list into two halves, an "active" and a "disabled" half. You will need an additional integer variable somewhere to track the index of the first disabled bullet in the list.
When your game starts, you preallocate some reasonable number of bullets in the list and set the "first disabled" index to 0 (because all bullets in the list are currently disabled). Whenever you need to "create" a new bullet you just set the properties of bullets[firstDisabled] to whatever values the new bullet should have:
bullets[firstDisabled].X = desiredX; bullets[firstDisabled].Y = desiredY; bullets[firstDisabled].Speed = 100; // The bullet is now active: ++firstDisabled;
In your main update loop, you only process bullets from 0 to firstDisabled:
for (var bulletIndex = 0; bulletIndex < firstDisabled; ++bulletIndex) { // Update the bullet here. }
To handle the case where a bullet has left the screen during the update loop, you don't remove it from the list. Instead you swap that bullet's data with the last enabled bullet's data (which is at firstDisabled - 1 and decrement firstDisabled.
Obviously this technique implies two things: you have a fixed upper limit of live bullets and that the order of bullets in the list doesn't matter. The latter should not be a problem, however the former might be. Fortunately you can deal with this if you have to by adding a new bullet at the end of the list (not in the middle) and swapping that bullet's properties with the bullet at index firstDisabled. Then you increment firstDisabled -- much like how you handle "removing" a disabled bullet, really.
The end result is that you have a system that allows bullets to be activated and deactivated in linear time (your original method was more intensive) and has a fixed memory budget which can't leak or induce garbage collection during gameplay and is easily cleaned up post-gameplay by simply clearing the entire list (if need be).
