How I can retrive 40 million records from Database?

Hello all,

please I am trying to retrieve 40 million records from Oracle Database, but my JSF page still loading without retrieve anything.

FYI, I use SQL statement to retrieve these data.

So is there any tools help. (hadoop Apache software not supported for Windows Production, only for development)

Thank you in advance.

Sami.

Because you are using JSF, I assume you want to display the information to a user on a web page. I'll skip asking why a user would want to see 40 million records. Does your app work when you try to display one record? If you post your code, maybe someone can comment on it.

Tom Reilly wrote:I'll skip asking why a user would want to see 40 million records.

I won't :-)

As Tom said, so many records should not be handled by the web layer because there's no way for a user to view them. They most likely should not be handled in one batch no matter what: if a record takes just 100 bytes of space then ... well, you do the math :-)

Thank you all, every day my customer sending 300000 messages to his customers, So he want to know what he send before 3 month.
let's say the point not see 40 million records, bcz I can make pagging in my JSF page that will retrieve 500 every search.

but when I am trying to make search with some conditions like From (Date), To (Date), Status, etc.... the page still loading without retrieve anything.

FYI.. the page retrieve 2 million records in 1 min .

Thank you again.

Does the search query work if you run it directly against the DB (using the DB's tools)?

Thank you,
I am using Toad, it retrieve 500 row when I make search, I want to know if there tools or software store huge number of data dealing with java language, but NOT hadoop.

Sami .

I think you should rethink your design. There is no point for the search query to return more records than will be displayed, and IMO 500 records is too many to put on a page meant for human consumption; consider displaying 50 records, or 100 at the most.

I think for fun you should create a dummy page that just create 40 million rows, repeated without going to the database, and then show this to your customer. If they can load it without their browser crashing and be able to look through it easily, then continue.

This is more of a case of a customer not understanding what they are asking for than a technical issue.

BTW here's an article on Pagination that may be useful. Do your really think when Google says "About 10,200,000 results for your query" they have a system designed, or ever expect, users to iterate through 10+ million results? What it sounds like you need, besides pagination, is a search engine built on top of your data. Granted, in this instance, simple database indexes may suffice.

I agree that you should be using paging. In addition to Scott's link, see our PaginationOrPaging FAQ page.

Thank you guys for your help