Millions of credential validations on the Active Directory from SSIS

While monitoring the event viewer on our Domain Controller, we started seeing a ever increasing number of credential validations each second, that we could not fully understand at first. I will attempt to give a brief run-through of our scenario, observations, fix and conclusion, which I made in conjunction with my colleague Henrik Jensen.

Scenario
1 CRM server, 2 SQL servers (1 CRM SQL and 1 BI SQL, where several SSIS jobs are running) and a domain controller.

Issue
Severe performance reduction on CRM and SQL servers. Increasing load times on several servers connected to the domaincontroller, and some login attempts being shut down.

Observation
The Active Directory were logging around 2 million events pr hour – all Credential Validation from the BI SQL service account.

We immediately suspected the SSIS jobs to be causing the problems, and we began to troubleshoot what could be causing it. We knew the bisql user was in the thick of things, so we started working out, what the user was doing, in regards to the SSIS jobs, and how we were creating the jobs.

When creating the jobs on the SQL server, the consultant would create a new job, create the steps, set a schedule, and active the job – something we have done a million times.

Solution
We realized, that the error was when the SSIS job was triggered, and the job was run in context of the BISQL user, but as the job had a different owner than the user running it, the server had to check the credentials with the Active Directory – the check was fine, and the job ran as it was supposed. However, each and every time any job ran, it had to check with the Active Directory, as the login was not stored after the initial login. The event thrown is not an error – just a log that a validation was performed successfully. Basically, the problem was caused because the SSIS jobs had the creating admin as the owner of the job. When I changed the owner from the creating admin to the service account, the event logging stopped creating the Credential Validations.

Conclusion
We searched for information all around the web in regards to this problem, but we were unable to find any information on this issue. This isn’t really a problem for most installations, but it is something to have in mind, when you are running multiple jobs on the same environment. I changed the ownership of all the jobs, and it have completely fixed the performance issues and login times.