Search This Blog

Dec 7, 2006

Can you trust ADU&C ?

Yesterday I was surprised to realize that ADUC won't report complete group membership, dsget and ldp report results ok.

This is explained in KB833883 and supposedly fixed in Windows 2003 sp1. However, while this hotfix is included in sp1, it's still needed to make the registry changes outlined in the same KB to make the hotfix work. Not entirely know why couldn't MS include that registry change altogether in sp1.

Please see the usernet thread I raised in details.

Dec 5, 2006

Required Active Directory Attributes for Email Delivery

--- copied from “Exchange 2003 Transportation and Routing Guide”

When you are troubleshooting an NDR, verify that all mail-enabled attributes that Message Categorizer requires exist for that recipient in Active Directory. In Exchange 2000, multiple attributes must be correct for messages to be categorized:

homeMDB
homeMTA
legacyExchangeDN
mail
mailNickname
msExchHomeServerName
msExchMailboxGuid
msExchMailboxSecurityDescriptor
proxyAddresses

This list of required attributes is valid only if the recipient is a mailbox-enabled object in Active Directory (for example, an Exchange 2003 recipient). However, if the recipient is an Exchange Server 5.5 recipient, the only attributes that have to be present are:

legacyExchangeDN
homeMDB
homeMTA

For mail-enabled objects (for example, a custom recipient) and alternate addresses, the targetAddress attribute is required. If the targetAddress attribute is not present, the fallback is to the mail attribute.

If an e-mail message is missing any of the required attributes or if they are incorrect, the message may remain in the categorizer, and no events are created in Event Viewer. If you track the message, it appears in Message Categorizer or it generates an NDR, depending on which attribute is missing. If you want to check these attributes for a user in Active Directory, use the LDP tool or ADSI Edit. For more information about the LDP tool or ADSI Edit, see the Windows online documentation.

Dec 4, 2006

Why outlook shows only sender email address instead of Display Name?

There are many possibilities. One of them is a by-design mechanism to prevent spoofed emails.

To prevent spoofing, Exchange 2003 requires authentication before a sender’s name is resolved in GAL. So be alarmed when you see SMTP address instead of display name in the sender field when the sender is supposed to be an internal user.

How To Setup Exchange To Receive Emails For A Shared SMTP Domain

Assume that we have 2 Exchange organizations, one is responsible for *@MainCompany.com emails (MainOrg), the other is responsible for *@subCompany.com emails (SubOrg). Now we want MainOrg to receive emails on behalf of SubOrg, meaning all emails that are sent to *@subCompany.com address should go to Exchange server in MainOrg.

Note: SubOrg doesn't have to be Exchange, it could be any mail system

1. For all users in SubOrg, create contacts in MainOrg
2. Create a Recipient Policy that will generate exactly same @subCompany.com email addresses for contacts you created in step 1. This Recipient Policy should NOT be authoritative for subComapany.com
3. Change public MX record of subCompany.com so it now points to MainCompany Exchange server instead of subCompany Exchange server
4. Create a SMTP connector on MainOrg Exchange server, specify subCompany.com as its space
5. Enable "relay for this domain" on the connector created in step 3, forward all mail to subCompany exchange server (subOrg Exchange as smart host)
6. Restart Routing Engine and SMTP services

Caution: subOrg must be configured as "authoritative" for @subCompany.com

Nov 29, 2006

Exchange routing considerations

- Internal messages always go for shortest route available
- A connector will be considered off ONLY when all bridgehead server(s) on that connector are down
- For external messages, a route is chosen with closest SMTP name space matching first regardless the cost. For example, an email destined to *.net will go to connector that is responsible for *.net even it has higher cost than the one that is responsible for * space.
- Routing does not fail over from a connector with a specific address space to a connector with a less specific space. So when there is problem with all *.net connector(s), emails will be queued up in *.net connector(s)
- The above 2 rules don’t apply to user who doesn’t have permission to the specific connector. Consider connector that a user doesn’t have permission as non-exist when routing his emails.

Nov 27, 2006

SMTP Virtual Server vs. SMTP Connector

SMTP Virtual Server vs. SMTP Connector
1. SMTP virtual server is the protocol stack that actually does the work - sending/receiving SMTP emails. SMTP virtual server alone gives you ability to send/receive Internet emails.
2. SMTP Connector is built on the top of virtual server and provides you more control - such as dispatching emails to different domains to different routes, applying different restrictions, etc.
3. SMTP virtual server only sends/receives emails to/from the IP address it is bound.
4. As for DNS, either specify external DNS servers on SMTP virtual servers or specify forwarder on DNS server that Exchange server uses.
5. The benefits of SMTP Connector are 1) ease of administration; 2) to simplify troubleshooting when issue surfaces.
6. You can either have your SMTP connector delivery the emails directly (given that the connector is able to resolve external domain names - using one of 2 settings in item 4 )
- or -
You can have your SMTP connector forward all emails to a smart host. Although you can specify a smart host on an virtual server, it’s better to set it on the connector. The smart host setting on the connector overrides any smart hosts on the virtual server.

Nov 21, 2006

VBscripting tips

  1. To make sure you fetch the output only after the command finishes, add the following between shell.exec and stdout.all
    Do While oExec.Status = 0
    WScript.Sleep 100
    Loop
  2. To pipe output from one process to another
    Dim oUserDN
  • set oShell = createobject("wscript.shell")
    mystr = "dsquery user -samid %username% -o dn dsget user -memberof"
    set oUserDN = oShell.exec("%ComSpec% /c """ & mystr & """")
    Wscript.Echo oUserDN.StdOut.readall

Oct 31, 2006

Troubleshooting 1030/1058 event ID error - one of the many cases

Userenv event ID 1030/1058 pair are very common errors that could be caused by various things. Common causes would be secure channel broken, domain account corrupt, client side group policy extension issue, etc. A brutal but simple solution will be quit/rejoin domain provided you are VERY sure your network and DCs are working properly. If DC or client side extension are in question, you would most likely want to enable userenv log as per KB 221833.The example below is certainly not the most difficult one but shows typical troubleshooting tools when the network is in doubt.

1030/1058 pair in application log, GP not applied

  • Event Type: Error
  • Event Source: Userenv
  • Event Category: None
  • Event ID: 1058
  • Date: 10/31/2006Time: 10:09:28 AM
  • User: NT AUTHORITY\SYSTEM
  • Computer: server1
  • Description:Windows cannot access the file gpt.ini for GPO cn={B8F49320-CB4A-4660-9BF5-806926B63906},cn=policies,cn=system,DC=samle,,DC=com. The file must be present at the location <\\sample.com\SysVol\sample.com\Policies\{B8F49320-CB4A-4660-9BF5-806926B63906}\gpt.ini>. (Configuration information could not be read from the domain controller, either because the machine is unavailable, or access has been denied. ). Group Policy processing aborted.

- nltest /sc_query:shortDomainName OK (to rule out security channel issue)

- netdiag /v: DC List Test failed. (The DC is down)Trust Relationship test failed

Note: if you open a cmd as local system, the domain membership test will be failed in netdiag - this is normal

The following 3 commands to test connectivity

  • ping shortDomainName OK
  • portqry test against all required ports OK
  • ping -l oversizePacket -f OK

Further troubleshooting:

  • Suspected NetBIOS was not enabled, went into TCP/IP setting, but it's ticked
  • Nslookup: Short domain name not resolved.
    Action: change suffix then flush DNS cache
    Result: short domain name resolved, but "DC list test" still failed
  • Found another symptom: Couldn't log into this box via TS, error "RPC server is not available". Check the RPC service and RPC Locator service, found the later was not running while this was a W2k member server.
    Action: start RPC Locator service
    Result: not difference
  • Checked application log again, 1030/1058 started on Oct 26. Checked System Log to see if there was anything happened at that day, it reported "tcp/ip NetBIOS helper" service terminated unexpectedly". This service is NetBIOS over TCP/IP itself.
    Action: started it.
    Result: netdiag came back clean

Actually I should have checked services status before even begun troubleshooting. Those set to "automatic" should be running - this is a quick way to find problems sometimes. But anyways, this can serve as an example of using different tools when network is in doubt.

Oct 26, 2006

How To User Mailbox Management Policy

  1. Define a policy in "Recipient Policies" container. Make sure you have correct filters and actions defined.
  2. Apply the policy immediately/manually or wait for RUS kicks in overnight
  3. Define a Mailbox Management schedule on *ALL* involved servers
  4. At this point, you can wait for the scheduled Mailbox Management process to launch or manually run it by selecting ?Start Mailbox Manager?

Note: Mailbox Management policy is not the same as System Policy that applies to either mailbox stores or servers. Mailbox Management policy is defined in Receipient Policy container while System Policy is defined under server object.

Oct 25, 2006

DNSBL - email block list

A DNS-based Blackhole List (DNSBL, also known as Real-time Blackhole List or RBL), is a means by which an Internet site may publish a list of IP addresses, in a format which can be easily queried by computer programs on the Internet. As the name suggests, the technology is built on top of the Internet DNS or Domain Name System. DNSBLs are chiefly used to publish lists of addresses linked to spamming. Most mail transport agent (mail server) software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

How Block List Providers Match Offending IP Addresses

After you set up your connection filter, when an e-mail message is sent to your organization, Exchange contacts the block list provider. The provider checks for the existence of an A (host) record in its DNS. Exchange queries for this information in a specific format. For example, if the connecting address is 192.168.5.1, and the block list provider's organization is contoso.org, then Exchange queries for the existence of the following record:

;. IN A 127. 0.0.x

which, in this case, is:

1.5.168.192..RBLprovider.org

If this IP address is found on the provider's list, the provider returns a 127.0.0.x status code that indicates an offending IP address and the type of offense. All block list providers return a response code of 127.0.0.x, where x indicates the type of offense. This number varies, depending on the block list provider.

Although few people object to the principle that mail-receiving sites should be able to reject undesired mail systematically, many have voiced objections to specific DNSBLs for the following reason:

For Joe Blow to refuse emails is legal (though it's bad policy, akin to "shooting the messenger"). But if Joe and ten million friends all gang up to make a blacklist, they are exercising illegal monopoly power.

Tips:

  • After a filter is defined, it will not take effect until it's applied to a smtp virtual server
  • Recipient filtering rules apply only to anonymous connections. Authenticated users and Exchange servers by pass these validations
  • Likewise, connection filters apply only to external connections. Connections made by Exchange servers within the same organization will not be affected.

Oct 13, 2006

NetBIOS name resource types

NameNumber(h)TypeUsage
00UWorkstation service
01UMessenger service
01GMaster Browser
03UMessenger service
06URemote Access Server service
1FUNetDDE service
20UFile Server service
21URemote Access Server client service
22UExchange Interchange (MSMail Connector)
23UExchange Store
24UExchange Directory
30UModem Sharing Server service
31UModem Sharing client service
43USMS Clients Remote Control
44USMS Administrators Remote Control Tool
45USMS Clients Remote Chat
46USMS Clients Remote Transfer
87UMicrosoft Exchange MTA
6AUMicrosoft Exchange IMC
BEUNetwork Monitor Agent
BFUNetwork Monitor Application
03UMessenger service
00GDomain Name
1BUDomain Master Browser
1CGDomain Controllers
1DUMaster Browser
1EGBrowser Service Elections
1CGIIS
00UIIS

Network issues that affect TCP/IP and RPC traffic across firewall or VPN after Windows 2003 SP1

After you install Windows Server 2003 Service Pack 1 (SP1), you may
experience issues that affect server-to-server communication for TCP/IP
traffic or remote procedure call (RPC) traffic across firewall or
virtual private network (VPN) products in rare and specific network
configurations.

Before carrying any troubleshooting, check the following KBs:

Q899148 Some firewalls may reject network traffic that originates from
Windows Server 2003 Service Pack 1-based computers
Q898060 Installing security update MS05-019 or Windows Server 2003
Service Pack 1 may cause network connectivity between clients and
servers to fail

Sep 13, 2006

DFSR notes 1

  1. Basic: DFS Namespace and DFS Replication are totally separate things despite the fact that they can work together and often work together
  2. You configure DFSR to replicate physical folders then publish them into namespaces
  3. You can’t create file under DFS namespace? (Access denied.) However you can do so on physical folders
    Answer: this is because target folder can be created only on shared folders. And I grant Read only on those shared folders.
  4. You can add(associate) multiple target folders into one folder in namespace. DFS Namespace will give you chance to create RG(you don’t have to). You will be referred to one of the targets when accessing using DFS namespace.
  5. Looks like we have to create RB then create namespace folder from RB, or vice versa. We can’t create both ready and connect them.
  6. To configure one-way replication, we can disable one of the sending connections in RG. However this is not recommended by Microsoft (see Planning Guide). Instead, NTFS permissions should be used to control replication direction.
  7. Domain controllers don’t have to run on R2. On servers that are involved in replication or act as name space server have to run R2. R2 schema (version 31) is that is required.

 

End#

Aug 3, 2006

Virtual memory, physical memory, page file, and 32bit/64bit limitations

Despite the huge amount of documents available on the Net, there are still the same huge amount of misunderstanding of above concepts.

Physical Memory: This is the easiest to under stand. Read it as RAM
Virtual Memory: Addressible space to the OS. It would be much more accurate if we call it Virtual Space
Page File: Due to the expense of RAM, we use disk space as an extension of physical memory.
Page: The allocation unit for memories
Paging Out: When certain criteria are met, an occupied page in RAM will be written to Page File to make space for other process. For example when a page in RAM occupied by a process has not been accessed for a period of time, and another process requests space while there is not free space, Paging Out happens.
Paging In: When a process refers a page that is not in RAM, the page will be Paging In.
4GB Virtual Memory Limit: All 32bit OS will have 4GB Virtual Memory for its processes
4GB Physical Memory Limit: All 32bit computers can address on 4GB RAM if no other technology, such as PAE, is deployed
32bit Windows OS running on 32bit hardware happen to have same Virtual Memory limit and Physical Memory limit, but they are different limits.
Available Memory: It could be one of three things
- Space that is still not used by process in its Virtual Memory
- Space that is still available in Physical Memory (RAM)
- Space that is still available in RAM+Page File

Most of the time, it means Available Physical Memory. But it could often refer to Available Virtual Memory. This is what becomes confusing.
/3GB Switch: By default, 32bit Windows divide the 4GB Virtual Space into two 2GB spaces, one for kernel mode, one for user mode. For applications, such as Exchange, that need a lot space in user mode, you can specify /3GB switch to increase user mode space by 1GB by reducing kernel mode space. This feature is available only on Windows 2000 Advance Edition and higher
PAE: PAE is a technology to extend the RAM space that can be access by a 32bit CPU. It's similar to LBA in hard disk.

So as you can see, you may use up both Virtual Memory and Physical Memory. It could be very confusing as 1) not everyone understands the memory management completely; 2) not all application/OS reports memory allocation errors clearly. But there are a few things you REALLY need to remember:

- If you use up Virtual Memory(again, really it's a space resource only!), it won't help by adding additional RAM. Imaging that if a city is using up its phone numbers, it won't help to have more phones! The only solution will be either reducing the number of users or increasing the length of phone number system)
- If you use up Physical Memory, you can buy more RAM up to the hardware limit. If you have 4GB already on a 32bit computer, you are not getting better to have another 1GB RAM (if PAE is not used).
- If you use up Physical Memory, do a little troubleshooting before rush for more RAM. More than often this is caused by a poor-coded program that leaks memory.

A few important performance counters:
- Memory, committed Bytes: memory that has been allocated for processes ( either those in RAM or those in page file)
- Memory: Pages Input/Sec: how many pages were read from page file into RAM
- Memory: Pages Output/Sec: see above
- Memory, Available MBytes: how much free RAM
- Prcess, Working Set: The number of pageable memory blocks in a process' address space that were recently referenced (i.e. still in physical RAM). When a process references pageable memory that is not currently in its working set, a page fault occurs.
- Paging File, %pagefile in use: this is an indicator if your pagefile has a proper size. It should be 50-75% used.

Jun 2, 2006

How to identify what service is listening on what port?

Open a command prompt and type

Netstat -ano

This command outputs a list of all listening ports, associated process IDs, and the port number

Now run

tasklist /svc

This will give you the service name that maps to a particular PID

Jul 13.
Further to above, a "new" option, -b, is available in netstat command after Windows 2003 SP1. This will show you what image occupies which port.

May 9, 2006

DFS, FRS, DFSR

- FRS is a service for replication files between servers
- DFS is a servcie to maintain shared namespaces between servers. It provides both unique name space and referral so users can have access to shared folders by using unique namespace without having to know where the resources are
- DFS is NOT responsible for replications. Synchronization between different link targets can be achieved by using FRS, manual copy(robocopy etc.) or thirdparty tools
- Sysvol folder on DCs is replicated using FRS

- In Windows 2003 R2, DFSR, DFS' own replication service replaces FRS (or any other replication service) to sync the folders/files
- FRS is not designed for replicating big files or huge amount of data

Apr 24, 2006

How to control remote desktop permissions

There are 3 ways to control who can have remote access via RDP/Terminal Service

1. Add/Remove users to/from Remote Desktop Users group. This is the recommended way;
2. Define "Allow log on through Terminal Services" in security policy
3. Terminal Services Configuration/Connections/The connection you want to change/Properties/Security/Advanced
Grant or clear permissions here. To enable remote login, a user needs at least the following permissions: Query Information, Logon, and Connect.

By default, administrators can log into DC remotely, admin and RD Users can log into non-DC machine remotely.

Apr 17, 2006

Kerberos, SPN, ktpass, and AD interoperability with Non-windows clients

A good understanding of kerberos is required before reading this post.

First of all, SPN is used to identify/locate a service.
Second of all, interoperability could mean any of the following 4 things:
1) An AD account can log into AD domain from a non-Windows box;
2) An service running on non-Windows box can leverage AD to authenticate it's clients using kerberos
3) An AD account can log into a MIT kerberos 5 realm from its AD domain.
4) Trust relationship between an AD domain and a kerberos realm

We will discuss 2) only in this post

When a client wants to use Kerberos as its authentication method, it has to know the name of the service it seeks and the host that provides the service in order to compose a SPN in its service request.

On the server side, of course, the SPN(s) should be registered properly.

When the KDC gets a Service Ticket Request from a client, it extracts the SPN from the request, locate the server in its database(in Windows world, it's AD) based on SPN. KDC then prepares the ticket, encrypted it with the resource server's master key or service account's master key, and returns the ticket to client.

Now the client presents the ticket to the resource server. Resource server is able to decrypt the ticket because the ticket is encrypted with its master key (or the host has service account's master key). Finally a service will or will not be provided based on the analysis of client's token.

If a resource server is a non-windows box, it will not have its account in AD, neither does it have master key. That is where ktpass comes into play. First, we create an account to repensent this non-windows box in AD. Second, we need to register SPN(s) against this account. Third, we need to generate and transfer the master key to the non-windows box so it can use it later to decrypt tickets.

Note: A master key is a hash of an account's password. It's stored in both AD and locally on a Windows box. For a non-Windows box, it's in a type of file called keytab

To perform the second and third tasks, run ktpass as follows:

ktpass -princ fakedServiceName/fakedMachineName@ad.domain.name.COM -mapuser adDomainNetBiosName\theAccnt4unixBox -pass pwdOfTheUnixBox -out c:\keyTabfileToBeTransferedToUnixBox.keytab

What the above command does is to
1) register SPN fakedServiceName/fakedMachineName@ad.domain.name.COM on the account theAccnt4unixBox;
2) generate a keytab file that can be transferred to non-Windows box later
3) enable the account option "Use DES ecryption types for this account".

The option in item 3) has to be enabled because AD supports only DES-CBC-CRC and DES-CBC-MD5 encryption. For the same reason, non-Windows client should use only the above 2 methods as their kerberos ticket encryption type. For Sun Java client, it can be specified in krb5.conf file or krb5.ini file.

So whenever the corss-platform kerberos authentication doesn't work, it's good to check the following:
1) Is the environment set up properly? For example, is the SPN registered? Is your network letting kerberos traffic get thru?
2) Does the corresponding user account in AD have "DES" option enabled? If you just enable the option, you need to reset the password and generated new keytab file
3) If you change the password, a new keytab file will need to be generated and merged into non-Windows box again.
4) What is configured in krb5.conf (or krb5.ini)? This link explains some basic settings
5) If all above didn't yield good results, a network trace will be very helpful

Side notes:
1) To set up a windows workstation for a MIT kerberos 5 realm, use ksetup
2) Keyword: "KDC has no support for encryption type (14)", "KRB5KDC_ERR_ETYPE_NOSUPP", unix, "active directory"
3) keytab file can be created either on unix host (such as "net ads keytab create") or on DC (ktpass)

Mar 21, 2006

How often intrasite replication happens?

In Windows 2000 AD, once a DC gets a modification, it sends out a notification to its closest replication partner 5 minutes later, and send the subsequent notifications to the other replication partners with 30 seconds pause.

If you change the forest level to Windows 2003, the numbers change to 15 seconds and 3 seconds respectively.

These values are stored in the following registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\
Parametersms\Replicator notify pause after modify (secs)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\
Parametersms\Replicator notify pause between DSAs (secs)

In AD, you can find the value on the cross-reference object for each directory partition in the Configuration container.
DSReplicationNotifyFirstDSADelay
DSReplicationNotifySubsequentDSADelay

Feb 12, 2006

Mailbox-enabled, mail-enalbed account, mail-enabled contact, vs. distribution list

New users to Exchange (or AD) more than often are confused with the above recipient types. Below is a brief explanation.

- Mailbox-enabled User: full functional object in AD/Exchange. Can send/receive email using local Exchange org, has SID and access to domain resources, and has mailbox on local Exchange server
- Mail-enabled User: has external email address only so can send/receive email only externally. Can be seen in GAL. Has SID and access to domain resources
- Mail-enalbed Contact (or just Contact): Same as Mail-enabled User but lacks of SID therefore doesn't have access to domain resources
- DL: A security group or distribution group with email address. If it's for email purpose only, make it a distribution group.

This is a very basic knowledge for whoever works on Exchange/AD, but I was amazed by the number of people who didn't know.

Jan 17, 2006

A time consuming case with kerbreos error

Situation:
==========
two DCs domain
PDC failed due to hard disk corruption, Exchange stopped. After that the customer tried to seize all fsmo roles but failed. (Side note: this was because he didn't have a DNS on BDC).
Then what he did was to restored BDC using a cloned image, changed the time back to the date when the image was made.

Still the exchange services failed to start. So he built a brand new forest+exchange and that ran properly.

Goal
====
Now he wanted to get old mails back

Troubleshooting
===============
1. Created DNS and verified that all records were registerred correctly
2. Exchange services still couldn't start
3. Netdiag reported:
DC list test . . . . . . . . . . . : Failed
[WARNING] Cannot call DsBind to dc.xxx.com (192.168.231.104). [SEC_E_WRONG_PRINCIPAL]

System Log reported:

Event ID : 4
Raw Event ID : 4
Category : None
Source : Kerberos
Type : Error
Generated : 2005-12-6 0:43:43
Written : 2005-12-6 0:43:43
Machine : MAIL
Message : The kerberos client received a KRB_AP_ERR_MODIFIED error from the
server host/dc.xxx.com. The target name used was ldap/dc.xxx.com/xxx.com@xxx.com. This indicates that the password used to encrypt the kerberos service ticket
is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (xxx.COM), and the client realm.
Please contact your system administrator.

5. checked into AD with ldifde dump, there was not any duplicate machine account, service principal name whatsoever.
6. There was not dupliate A records in either forward lookup zone or reverse lookup zone
7. All report on DC was perfectly OK
8. Reset secure channel on Exchange server was successful, but problem persisted
9. Found that we could not open ADUC on Exchange server, nor could we remotely connect to DC using Event Viewer
10. Capture network traffic trace, the error was "krb5krb_ap_err_modified", which still pointed to duplicated machine accounts or SPNs. But we did not have any duplicates

*** What else that is not correctly reported by all above information could be wrong ? ****

11. Disjoined Exchange from domain (in order to get a fresh computer account in domain)
12. Got error "target principal name incorrect" when we re-joined it back
13. We got another copy of network trace. In it we found
KERBEROS: Error code (error-code[6]) = Pre-authentication information was invalid
14. we tried everything we could think of, such as to stop antivirus, check Stored User and Password, etc. No luck.

Resolution:
===========
15. Reset the secure channel for the DC itself. (But weird enough that DC didn't report anything wrong if there had been secure channel issue!)

You can reset single DC environment password as follows:
nl_test /sc_change_pwd:domain
or
netdom resetpwd /server:IPofDC ......(do NOT stop KDC)