Score:2

Windows Server 2019 Throttling for Domain Users and not Admins?

cc flag

Morning All,

I've got a fairly new environment running the latest ESXi, all WS19S servers AD joined and everything seems to be working as it should and is very responsive as all of the servers are over provisioned in resources. Recently, we installed an application that runs from a file share on one of the servers, app seems to do well until it has to loop through a folder that has 40+GBs of PDFs (roughly 250K of files).

Here's the interesting part, when you access the app from any client or server as a domain user(did a lot of testing for this), it can take up to 40 seconds for the app to iterate through those PDFs (I know it's a poor design), need to mention, everything else seems fairly responsive. Now, log in as a domain admin on the same machine as the user did, it only takes a few seconds to go through the same file. Have had many discussions with the software vendor and they are as stumped as I am.

Has anyone seen this type of behavior before and if so, how do you overcome it?

I've disabled things like the AV and other security products to see if maybe they were scanning or limiting read/writes but nothing changed. Also, I only a have a few GPO items configured, mostly having to do with passwords.

Stumped is not the word right now!!!

Jevgenij Martynenko avatar
us flag
To confirm your assumption, try to do more thorough testing using the same account. Create a brand new test account, asign it domain admin rights, and measure the time. Then remove admin rights, relogin (important), make sure admin rights are gone by running `whoami /groups`, and try again.
Davids Learnin avatar
cc flag
Thanks but that has already been done, ran as described above after I removed the test account from the domain admin group (confirmed by using dumpsec). Also ran on a clean install in another environment with the same outcome.
Massimo avatar
ng flag
What is the app actually doing with those files? Do you notice this slowdown only when using the app, or does it happen also when f.e. browsing the network share? Also, what happens if you access the folder on the server itself instead of using the network?
Jevgenij Martynenko avatar
us flag
Have you checked if assigning local admin to the account make things run faster? Please update your question with test results to shine more light on this. Anyway, my guess is this not related to some network or storage throttling, but the app is doing something else, which it can't do fast enough when run as a non-admin user. This action (or attempted action) is appear to be slowing the app. For further debuging I would go with Process Monitor from Sysinternals. It might point you to some failed actions. Especially, if you compare app activity running with and without admin rights
Davids Learnin avatar
cc flag
The app is creating a ticket, the folder in question, stores image files of the ticket in pdf format, what the process is doing, its itinerating through the folder to see if a name of a pdf file exist. If it doesnt find the file, it creates a new one. There are no errors being generated any where, I can open one of their log files and see the timestamps of each process, seeing 40-60 seconds on searching the folder for domain users vs 2-3 seconds for the same process for a domain admin. I should also add, the users can read ever file, open, modify, create and delete as needed in that folder.
Davids Learnin avatar
cc flag
Forgot to add, the app has to be ran as an local administrator, does change how it acts at all.
Massimo avatar
ng flag
Iterating through a folder to see if a file name is in use? This is not poor design, this is *astonishingly bad* design. It looks like checking for the file existing, or even trying to open the file and checking for an error, was just too much of a leap...
Massimo avatar
ng flag
Can you run the app on the server itself (so to rule out networking issues)?
Massimo avatar
ng flag
Also, you can try using https://learn.microsoft.com/en-us/sysinternals/downloads/procmon to check what is going on at the file system level; maybe the standard user is hitting access denied errors which cause the slowdown.
cn flag
I find it odd that there isn't any information in the question about the security of the files in question, and if the Domain Admins group has elevated/different permissions on said documents. Domain Admins should not have permissions, that is an inferior design.
Davids Learnin avatar
cc flag
Massimo, yes, we've tried running on the server in question with the same result, could also eliminate network issues because the way it runs with domain admins. Greg, domain admins have full rights in our organization as well as most, not inferior but essential in smaller environments (lack of personal which dictates segregation of duties)
Score:1
sy flag

We had the same issue as described. For us turning off access based enumeration on the file share fixed the issue.

https://theitbros.com/config-access-based-enumeration-windows-server-2016/

Score:0
ng flag

I'll venture a guess: there are different ACLs on the individual files.

If this is the case, when you are a standard user you can only access some of them; instead when you have admin rights you get the full load.

Depending on what the application is doing with them, having to process 1k or 10k files instead of 250k could make a significant difference.

Davids Learnin avatar
cc flag
Thanks for your feedback but that is kind of opposite of the problem I feel meaning if it was restricted to a user, their time would be shorter, also, if they weren't seeing all of the files, the application would create a new imagine because it wouldn't exist because they don't have access Good thought but I dont think it applies..
Massimo avatar
ng flag
My bad, I completely misread your question and thought the app was slower for admins than users...
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.