Skip to main content
added 845 characters in body
Source Link
Ewan
  • 84.5k
  • 5
  • 91
  • 189

It's hard to debug without seeing the code, but...

created time stamp greater than or equal to the last successful application completion datetime

This is incorrect, you need to keep track of the max Created/Updated date that you have pulled, rather than when the app ran.

Also, you should set a period end date which is less than the current time, as well as a start date, to avoid inconsistencies at the end of the data set. maybe now - 1 min, so you avoid any problems with millisecond accuracy/rounding.

Also, because you store this info on a database, if you run the watchdog service twice at the same time, it's going to corrupt the date.

Get the remaining batches using the exact same filter but with a "skip" clause to separate them.

This assumes the order won't change, and the set stays the same. Say for example there are 10 records but five have the exact same date, if you pull 10 take 5 and then 10 skip 5 take 5 you might get duplicates and skip some rows if the order is different on the second pull. Follow @J_H's advice and get record with a date between two dates instead

Generate an SQL statement to insert the records from the temp table into the target table and run the statement.

Your whole mucking around with temp tables suggests there is some issue with just updating the main table directly? Unless the data has a nice primary key I can see this going wrong.

Do the same process, but this time filter by the modified time stamp. On completion, update the matching records in the target table.

This is going to bring back duplicates if you have a row that's been created and modified. Why not just query all rows with Created OR Modified >= lastMaxDate

It's hard to debug without seeing the code, but...

created time stamp greater than or equal to the last successful application completion datetime

This is incorrect, you need to keep track of the max Created/Updated date that you have pulled, rather than when the app ran.

Get the remaining batches using the exact same filter but with a "skip" clause to separate them.

This assumes the order won't change, and the set stays the same. Say for example there are 10 records but five have the exact same date, if you pull 10 take 5 and then 10 skip 5 take 5 you might get duplicates and skip some rows if the order is different on the second pull. Follow @J_H's advice and get record with a date between two dates instead

It's hard to debug without seeing the code, but...

created time stamp greater than or equal to the last successful application completion datetime

This is incorrect, you need to keep track of the max Created/Updated date that you have pulled, rather than when the app ran.

Also, you should set a period end date which is less than the current time, as well as a start date, to avoid inconsistencies at the end of the data set. maybe now - 1 min, so you avoid any problems with millisecond accuracy/rounding.

Also, because you store this info on a database, if you run the watchdog service twice at the same time, it's going to corrupt the date.

Get the remaining batches using the exact same filter but with a "skip" clause to separate them.

This assumes the order won't change, and the set stays the same. Say for example there are 10 records but five have the exact same date, if you pull 10 take 5 and then 10 skip 5 take 5 you might get duplicates and skip some rows if the order is different on the second pull. Follow @J_H's advice and get record with a date between two dates instead

Generate an SQL statement to insert the records from the temp table into the target table and run the statement.

Your whole mucking around with temp tables suggests there is some issue with just updating the main table directly? Unless the data has a nice primary key I can see this going wrong.

Do the same process, but this time filter by the modified time stamp. On completion, update the matching records in the target table.

This is going to bring back duplicates if you have a row that's been created and modified. Why not just query all rows with Created OR Modified >= lastMaxDate

Source Link
Ewan
  • 84.5k
  • 5
  • 91
  • 189

It's hard to debug without seeing the code, but...

created time stamp greater than or equal to the last successful application completion datetime

This is incorrect, you need to keep track of the max Created/Updated date that you have pulled, rather than when the app ran.

Get the remaining batches using the exact same filter but with a "skip" clause to separate them.

This assumes the order won't change, and the set stays the same. Say for example there are 10 records but five have the exact same date, if you pull 10 take 5 and then 10 skip 5 take 5 you might get duplicates and skip some rows if the order is different on the second pull. Follow @J_H's advice and get record with a date between two dates instead