-1

I've been working on this project where my responsibilities are to code for validating, processing and then dumping into database excel file.

Me and my colleague tried multiple approaches to make it generic as much as possible but it seems to be not achievable (that's what I and he think).

Our last generic approach is that, we moved our business validation into query means no more code, we were validating, processing and dumping (insertion/updation) with query.

The real problem is that whenever there any new columns added into we need to modify whole query all the time and it is so much time-consuming even for one column.

When we were trying "Query Method" we came to know about that processing csv file is much easier and more efficient than excel so we convert our excel to csv and then we use "COPY" method of PostgreSQL to copy from csv to temporary table.

Is there any way I can make this process fully generic?

Improve this question
1
  • 1
    note that each excel have different business validation and requirements. Commented Jul 23, 2024 at 8:04

1 Answer 1

Reset to default
2

Most extract-transform-load operations cannot be made generic. Generalizing something means finding an abstraction that works in all use cases, and as you've found, when the business rules are different you cannot generalize.

The "generic" part about programmatically interacting with an Excel sheet is the code library you use to interact with it. At some point you need to do something real and specific, which means you need to write code. The Excel library doesn't know the semantics of you data. You do, and you codify those rules in the mapping and validation code.

The semantics of the data being specific to your application, and the logic required to map the input data to the structures that make sense for your application are the main reasons you cannot identify a generic solution.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.