Waffle going with Basic instead of Negotiate when many sessions

Nov 10, 2010 at 11:20 AM

Hi,

I'm asking for your help again. We wanted to use waffle in our production environment after a long time of implementing a new version and testing. In development and also in testing, there were no issues at all. But after installing into production we got a problem. After some time it happens that user get the Basic authentication window instead of automatic authentication and also it happens random, sometimes it authenticates and sometime not. I'm clueless what the cause of this problem is. And the worst is we cannot reproduce this behavior in the Test environment at all. The test environment and the production environment are the same as for OS and IE versions and also hardware.

We have a APP server and then a terminal server - Windows server 2003 R2 - Standard x64 Edition - Service Pack 2 - 2 x QuadCore CPU with IE 7 on which the users log on and then open the application which connects to the APP Server, so basically its two computers communicating, but there are lot of users, each user has some windows of IE with the application open...

We tried already to sniff the communication and the only thing that I'm seeing is that there is the situation like the client gets Unauthorized, so it sends NTLMSSP_NEGOTIATE, but then gets another Unauthorized.

It goes like this:

LOCK /app/webdav/Vorr%C3%A4teabrechnung%2001-05-2009%20bis%2001-05-.4345986.xlsx HTTP/1.1
Content-Language: en-us
Accept-Language: de, en-us;q=0.2
Content-Length: 0
Timeout: Second-180
Translate: f
Content-Type: text/xml
Depth: 0
User-Agent: Microsoft Data Access Internet Publishing Provider DAV
Connection: Keep-Alive
Cookie: JSESSIONID=A98084FE40716AB3B6E5DBA8870BD48C
Authorization: NTLM TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAFAs4OAAAADw==

NOT working - When switching to Basic:

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: Basic realm="BasicSecurityFilterProvider"
WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
Connection: close
Content-Type: text/html;
charset=utf-8
Content-Length: 954
Date: Tue, 09 Nov 2010 09:04:50 GMT

Working:

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: NTLM TlRMTVNTUAACAAAACgAKADgAAAAFgomizRybg0w/6lwAAAAAAAAAALIAsgBCAAAABgBxFwAAAA9BADAAMAAwAEQAAgAKAEEAMAAwADAARAABAA4AQQAwADAAMABTADEAMQAEACIAQQAwADAAMABEAC4AZwByAGUAYwBvAG4AZQB0AC4AYQB0AAMAMgBBADAAMAAwAFMAMQAxAC4AQQAwADAAMABEAC4AZwByAGUAYwBvAG4AZQB0AC4AYQB0AAUAIgBBADAAMAAwAEQALgBnAHIAZQBjAG8AbgBlAHQALgBhAHQABwAIAHlN4Srtf8sBAAAAAA==
Connection: keep-alive
Transfer-Encoding: chunked
Date: Tue, 09 Nov 2010 09:04:50 GMT

The setting in web.xml is:

<filter>
<filter-name>SecurityFilter</filter-name>
<filter-class>waffle.servlet.NegotiateSecurityFilter</filter-class>   

<init-param>
<param-name>principalFormat</param-name>
<param-value>fqn</param-value>
</init-param>
<init-param>
<param-name>roleFormat</param-name>
<param-value>both</param-value>
</init-param>
<init-param>
<param-name>allowGuestLogin</param-name>
<param-value>false</param-value>
</init-param>
<init-param>
<param-name>securityFilterProviders</param-name>
<param-value>
waffle.servlet.spi.BasicSecurityFilterProvider
waffle.servlet.spi.NegotiateSecurityFilterProvider
</param-value>
</init-param>
<init-param>          
<param-name>waffle.servlet.spi.NegotiateSecurityFilterProvider/protocols</param-name>           
<param-value>NTLM</param-value>         
</init-param>
</filter> 

The thing I wonder about is that everything works fine for some time but after some time or maybe after some number of session open it starts to happen.  Also we cannot debug it at all, because it happens just in the production.

Do you have some idea?

Coordinator
Nov 10, 2010 at 11:29 AM

Your BASIC provider is first in the list, so your client is probably preferring it because of its version, settings or policy. Swap the order of securityFilterProviders in the web.xml.

<param-value> 
waffle.servlet.spi.NegotiateSecurityFilterProvider 
waffle.servlet.spi.BasicSecurityFilterProvider 
</param-value> 

Nov 10, 2010 at 11:35 AM

We tried that and the problem still came... I forgot that to mention in my first post. I also tried to add Kerberos to protocols but didn't help also

Coordinator
Nov 10, 2010 at 11:45 AM

Every single time it has been a client problem. Make the change, get the complete HTTP trace from a client that's doing BASIC. It's usually a security setting on the browser where the client doesn't believe it's in the intranet (trusted site, etc.) or has Windows Authentication disabled.

Nov 11, 2010 at 10:48 AM

the first try to log on:

 

GET /App/WelcomeToApp.po HTTP/1.1
Accept: */*
Accept-Language: de-AT,en-US
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Host: ServerHost
Connection: Keep-Alive

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=E29F4220B7A4C888C5427018EAA57489; Path=/App; Secure
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
WWW-Authenticate: Basic realm="BasicSecurityFilterProvider"
Connection: keep-alive
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Thu, 11 Nov 2010 10:56:10 GMT

GET /App/WelcomeToApp.po HTTP/1.1 NTLMSSP_NEGOTIATE
Accept: */*
Accept-Language: de-AT,en-US
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Host: ServerHost
Connection: Keep-Alive
Cookie: JSESSIONID=E29F4220B7A4C888C5427018EAA57489
Authorization: Negotiate TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAFAs4OAAAADw==

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
WWW-Authenticate: Basic realm="BasicSecurityFilterProvider"
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Thu, 11 Nov 2010 10:56:10 GMT

 

After that comes the Basic window and the try again:

 

GET /App/WelcomeToApp.po HTTP/1.1 NTLMSSP_NEGOTIATE
Accept: */*
Accept-Language: de-AT,en-US
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Host: ServerHost
Connection: Keep-Alive
Authorization: Negotiate TlRMTVNTUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAFAs4OAAAADw==
Cookie: JSESSIONID=E29F4220B7A4C888C5427018EAA57489

HTTP/1.1 401 Unauthorized NTLMSSP_CHALLENGE
Server: Apache-Coyote/1.1
WWW-Authenticate: Negotiate TlRMTVNTUAACAAAACgAKADgAAAAFgominkecxbp4KfYAAAAAAAAAALIAsgBCAAAABgBxFwAAAA9BADAAMAAwAEQAAgAKAEEAMAAwADAARAABAA4AQQAwADAAMABTADEAMQAEACIAQQAwADAAMABEAC4AZwByAGUAYwBvAG4AZQB0AC4AYQB0AAMAMgBBADAAMAAwAFMAMQAxAC4AQQAwADAAMABEAC4AZwByAGUAYwBvAG4AZQB0AC4AYQB0AAUAIgBBADAAMAAwAEQALgBnAHIAZQBjAG8AbgBlAHQALgBhAHQABwAIALGw8BePgcsBAAAAAA==
Connection: keep-alive
Transfer-Encoding: chunked
Date: Thu, 11 Nov 2010 10:56:29 GMT

GET /App/WelcomeToApp.po HTTP/1.1 NTLMSSP_AUTH, USer: Domain\user
Accept: */*
Accept-Language: de-AT,en-US
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Host: ServerHost
Connection: Keep-Alive
Authorization: Negotiate TlRMTVNTUAADAAAAGAAYAHAAAAAYABgAiAAAAAoACgBIAAAAEAAQAFIAAAAOAA4AYgAAAAAAAACgAAAABYKIogUCzg4AAAAPYQAwADAAMABkAHAAYQBsAGUAYwBlAGsAdwBBADAAMAAwAFMAMgAzAEHUAzHBgJdfAAAAAAAAAAAAAAAAAAAAAGs7NjlPbVvVbBhIspiqdsmIXMYgA+VhGg==
Cookie: JSESSIONID=E29F4220B7A4C888C5427018EAA57489

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=E29F4220B7A4C888C5427018EAA57489; Path=/PartnerVerwaltung
Expires: -1
Cache: no-cache
Cache-Control: no-cache
Content-Type: text/html;charset=UTF-8
Content-Length: 47484
Date: Thu, 11 Nov 2010 10:56:29 GMT

 

It was with the altered configuration:

<init-param>
	<param-name>securityFilterProviders</param-name>
	<param-value>
		waffle.servlet.spi.NegotiateSecurityFilterProvider
		waffle.servlet.spi.BasicSecurityFilterProvider
	</param-value>
</init-param>
<init-param>
	<param-name>waffle.servlet.spi.NegotiateSecurityFilterProvider/protocols</param-name> 
        <param-value>
		Negotiate
		NTLM
	</param-value> 
</init-param>

It happens mostly with more than 50 users connected...

Coordinator
Nov 11, 2010 at 1:36 PM

Ok. I think we have a real problem here. The initial diagnosis was wrong.

You're seeing a popup, but not a BASIC authentication one. It looks the same, but it's a popup that asks for LM credentials (theoretically allows you to login as a different domain user). It's caused by the server denying the logon repeatedly for perfectly valid credentials.

  1. Do you think you can find a server-side log for a session like this with Waffle logging at DEBUG level?
  2. Are you running 1.3? Can you try swapping it for 1.4 beta? I've definitely corrected a couple of volume/scale problems - it would be a long shot, but I could stretch a reasonable explanation.
  3. You say it's happening with 50+ users connected. Does it permanently bring the system to a failure (ie. all new users can't logon)?