0

I have an azure data factory pipeline that reads a 100 mb CSV from blob storage and inserts it into an Azure SQL DB that has tons of space (200 GB). I also have a 1 MB version of the same files, just with most of the data removed, as a test version.

When I run the pipeline with the full file, I get the error: DF-Executor-InternalServerError. It gives me no other useful information.

However, when I run the short version, it runs just fine. Along with a few other discussions on this error, I think the problem is simply that I need to increase execution/computer resources and/or timeout time. I'm just not sure where I should be doing that in the chain.

Has anyone run into anything like this?

Improve this question
2
  • Everything has a timeout. How long does it run until it fails? Commented Jul 13, 2023 at 8:52
  • 30 minutes, so I feels to me it's on the resource side, where it just can't handle processing the 100 mb excel. I just don't know where on the azure side I can pump up compute. Commented Jul 13, 2023 at 15:06

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.