1

I have a task to query a million records from one org and then upsert them in another org. I use a Bulk API query job to get all records.

HttpRequest request = new HttpRequest(); request.setEndpoint('callout:OrgIntegration/services/data/v58.0/jobs/query/id/results?locator=MTAwMDA&maxRecords=1000000'); request.setMethod('GET'); request.setHeader('Accept', 'text/csv'); request.setHeader('Accept-Encoding', 'gzip'); Http http = new Http(); HttpResponse response = http.send(request);

But I have the error: "Apex heap size is too large". I need that CSV to pass it to another Bulk API job - to insert records (also in Apex). So can you please give me any ideas for how I can deal with that large response?

Improve this question

2 Answers 2

Reset to default
1

SFDX to the rescue !

bulk export:

sf data query -o source.org -q "your query" -b -r csv -w 10 > records.csv

bulk import:

sf data upsert bulk -o target.org --external-id [your external id field api name] -f records.csv -s [your sobject name] -w 10
Improve this answer
0

Apex is not the right tool for this, as it is not designed to handle such a large amount of CSV data (hence why you are getting an error).

I would suggest using Salesforce Data Loader if this job can be done manually. Otherwise something like a function in AWS Lambda could handle this use case.

Alternatively, you could use Salesforce Connect to view the records in one org from another org, if that works with your use-cases.

Improve this answer
8
  • Initially this insertion of records should occur by clicking on the quick action. so i decided to use apex. how can i do this if not manually? Commented Jun 19, 2023 at 15:43
  • How many times do you need to do this migration? If just once, you can query all records in one org using Salesforce Data Loader and then use that result to insert the records into the second org. Commented Jun 19, 2023 at 15:45
  • not just once. i need to query that million of records from second org then compare with records on the firs org and upsert them. Commented Jun 19, 2023 at 15:49
  • does this need to happen automatically? how often will this comparison and upsert need to happen? Commented Jun 19, 2023 at 16:24
  • not automatically. when the user clicks on quick action like once a day Commented Jun 19, 2023 at 16:58

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.