The recommendation in Junit in Action 2e is actually not to create too many datasets (like one per test class), but just enough that is considered maintainable. Except for a few exceptional cases, I found it possible to use a master dataset for most unit tests, and individual datasets for integration tests. Limiting the usage of ExpectedDataSets is also an option.
Also, I used Unitils in combination with dbunit to simplify some of the setup and loading of test data, so you might want to consider it where appropriate.