0

Setup Info

I have an application deployed on a k8s cluster on perm. My application uses many utility apps like Vault, RabbitMQ, and Redis, and the application is exposed via ingress. My application is in .NET Core, and the frontend is Angular. The database is an MSSQL database cluster again on perm.

Problem:

My application speed is very inconsistent. When I use any module where a search operation is required with a database, sometimes the application takes 32 seconds to load, and other times it takes just 2 seconds. If I keep changing the search number, it is always different times, sometimes 11 seconds, sometimes above 30 sec. I am not able to find the problem and need help to investigate this issue.

What have I done?

First, I check the database to make sure the query is not slow. After working with the DB admin, it seems fine, and it returns the heaviest data search under 1 sec. We also use curl to hit the DB to search for an ID number from the cluster VM. It was super fast, so the query is not the issue here.

I checked the logs of coredns, and its logs are clean; also, the ingress logs show no error. I am running out of ideas to test and find out the issue of these inconsistencies and delays.

What I am looking for:

I need some advice to dig into this issue and find out the core issue here. Any professional advice will be appreciated.

Improve this question

1 Answer 1

Reset to default
1

The best way to really get under the hood of what's happening is to use telemetry data, using open telemetry (https://opentelemetry.io/docs/what-is-opentelemetry/)

You will first need to instrument your app by using the relevant opentelemetry packages (e.g. nodejs - @opentelemetry/instrumentation-http, there will be python equivalents). Use the opentelemetry SDK packages also to initiate and set up an exporter.

You can then export the span data directly to the OTEL (this is generated by the packages you added to your code above), and you can run the collector in your cluster to ingest the data. And then send it on to an open source telemetry monitoring platform such as Jaeger, where you can visualise the span data - it will break down the requests

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.