A good understanding of kerberos is required before reading this post.
First of all, SPN is used to identify/locate a service.
Second of all, interoperability could mean any of the following 4 things:
1) An AD account can log into AD domain from a non-Windows box;
2) An service running on non-Windows box can leverage AD to authenticate it's clients using kerberos
3) An AD account can log into a MIT kerberos 5 realm from its AD domain.
4) Trust relationship between an AD domain and a kerberos realm
We will discuss 2) only in this post
When a client wants to use Kerberos as its authentication method, it has to know the name of the service it seeks and the host that provides the service in order to compose a SPN in its service request.
On the server side, of course, the SPN(s) should be registered properly.
When the KDC gets a Service Ticket Request from a client, it extracts the SPN from the request, locate the server in its database(in Windows world, it's AD) based on SPN. KDC then prepares the ticket, encrypted it with the resource server's master key or service account's master key, and returns the ticket to client.
Now the client presents the ticket to the resource server. Resource server is able to decrypt the ticket because the ticket is encrypted with its master key (or the host has service account's master key). Finally a service will or will not be provided based on the analysis of client's token.
If a resource server is a non-windows box, it will not have its account in AD, neither does it have master key. That is where ktpass comes into play. First, we create an account to repensent this non-windows box in AD. Second, we need to register SPN(s) against this account. Third, we need to generate and transfer the master key to the non-windows box so it can use it later to decrypt tickets.
Note: A master key is a hash of an account's password. It's stored in both AD and locally on a Windows box. For a non-Windows box, it's in a type of file called keytab
To perform the second and third tasks, run ktpass as follows:
ktpass -princ fakedServiceName/fakedMachineName@ad.domain.name.COM -mapuser adDomainNetBiosName\theAccnt4unixBox -pass pwdOfTheUnixBox -out c:\keyTabfileToBeTransferedToUnixBox.keytab
What the above command does is to
1) register SPN fakedServiceName/fakedMachineName@ad.domain.name.COM on the account theAccnt4unixBox;
2) generate a keytab file that can be transferred to non-Windows box later
3) enable the account option "Use DES ecryption types for this account".
The option in item 3) has to be enabled because AD supports only DES-CBC-CRC and DES-CBC-MD5 encryption. For the same reason, non-Windows client should use only the above 2 methods as their kerberos ticket encryption type. For Sun Java client, it can be specified in krb5.conf file or krb5.ini file.
So whenever the corss-platform kerberos authentication doesn't work, it's good to check the following:
1) Is the environment set up properly? For example, is the SPN registered? Is your network letting kerberos traffic get thru?
2) Does the corresponding user account in AD have "DES" option enabled? If you just enable the option, you need to reset the password and generated new keytab file
3) If you change the password, a new keytab file will need to be generated and merged into non-Windows box again.
4) What is configured in krb5.conf (or krb5.ini)? This link explains some basic settings
5) If all above didn't yield good results, a network trace will be very helpful
Side notes:
1) To set up a windows workstation for a MIT kerberos 5 realm, use ksetup
2) Keyword: "KDC has no support for encryption type (14)", "KRB5KDC_ERR_ETYPE_NOSUPP", unix, "active directory"
3) keytab file can be created either on unix host (such as "net ads keytab create") or on DC (ktpass)