1

I have a two tables.

TABLE 1 has id(PRIMARY)|RANK|WEBSITE |ADDRESS This is filled with a million websites. TABLE 2 has id(PRIMARY)|tag1|tag2|tag3 ......|tag30

I am generating the tags by scraping for the meta tags and other attributes.

I have no problems populating the database for the first few thousand websites. After that, I am not able to populate it. I guess it timed out.

How should I proceed to populate the entire database?

EDIT:

I am getting the data by scraping the urls of the websites dynamically

Improve this question
2
  • Where is the data coming from? From a file or you are generating the data dynamically. Can you please elaborate? Commented Oct 31, 2010 at 6:46
  • stackoverflow.com/questions/891930/… Commented Oct 31, 2010 at 7:39

1 Answer 1

Reset to default
1

It would be good if you specify error message in your question, output of mysql_error etc (if any).

If your script execution is interrupted because of PHP timeout, you may try to set corresponding configuration parameters (see here for php.ini configuration for details, and here for configuration in runtime).

If reconfiguration is not possible, you may store last record ID you updated in file / database table, and continue execution starting from next ID next time you run your PHP script.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I am storing the last record ID. This is a better option for me.