Questions tagged [etl]
'Extract, Transform and Load': Data load processing, normally in the context of data warehouse systems.
260 questions
0 votes
0 answers
38 views
AzureSQL - JDBC - Threads hung over socket read
I have a Java application with an ETL process running with Azure SQL DB and some times the processing just freezes but the app still runs. I did some thread dump analysis and found that during this ...
- 91
0 votes
0 answers
48 views
SSIS Data Quality Services Issue - Object reference not set to an instance of an object
I have tried to load data using my SSIS pipelines and there is no issue when the source table have around 90k+ data. This Data Quality Services component works well. But once the data has reached 100k+...
- 11
-1 votes
1 answer
77 views
How can I make the real-time synchronization between main server and target server?
I have occurred in a serious problem, I had inserted some data into table from main server to local server through SSIS ETL technique. All is perfect till scheduling. But the main problem is, if in ...
0 votes
0 answers
136 views
Best practice for applying complex business logic during transformation in ETL from Aurora db to Redshift data warehouse
I'll be pushing data from an Amazon Aurora db to a Redshift data warehouse. The painful part is that the transformation portion of the ETL workflow relies heavily on business logic, that lives within ...
- 143
1 vote
1 answer
163 views
What are the idiomatic approaches to cross-server upserts?
This is a very common problem, faced by virtually every DBA who has to responsibilities to both application and BI teams. Consider the following: You have two T-SQL servers, Production and Reporting. ...
- 1,322
0 votes
2 answers
220 views
Growing SQL Server Data Warehouse - How to reorganize efficiently?
we've been using SQL Server for quite a while now in our company to host all application data. We use a typical set-up with data being loaded from various data sources into a Staging Area, which ...
- 3
0 votes
1 answer
135 views
Does it makes sense to have the operational DB and analytics DB on the same SQLServer?
Let's say I have a single table that stores tracing information from mobile devices (each record has deviceID, transaction info, a json column with full details about the operation, time of operation, ...
- 111
0 votes
0 answers
862 views
ETL staging area incremental load design considerations
We are currently working on a design for our new data warehouse. In our current data warehouse we have a 'persistent staging area' or PSA. We load new records incrementally into a staging table and ...
- 549