Tools of the Trade - Office 365 Tools for Hybrid Deployments and Migrations - Part 3

Other articles in this series:

• Tools of the Trade - Office 365 Tools for Hybrid Deployments and Migrations - Part 1
• Tools of the Trade - Office 365 Tools for Hybrid Deployments and Migrations - Part 2

Office 365 Tools Recap

In the two previous posts in this series, we took a look at some tools that are used in the planning and deployment phases of an Office 365 hybrid implemenation or migration.  We explored a couple of the tools used for evaluating your environment's  preparedness for directory synchronization in the first post. Then we devoted the entire second post to the Microsoft Azure Active Directory Sync tool. Now we will wrap up this 3-part series with a look a few helpful tools for diagnosing and troubleshooting connectivity issues after an Office 365 deployment.

Intro to the Tools

There are plenty of tools out there, many by  Microsoft, for troubleshooting environments, and many could be used for troubleshooting connectivity with an Office 365 or hybrid environment. However, in this post we will focus on three tools that will be most helpful for ensuring our public facing configuration is setup properly: the Microsoft Remote Connectivity Analyzer (MRCA), the Microsoft Connectivity Analyzer (MCA), and the Microsoft Lync Connectivity Analyzer.

I felt the above image was appropriate for two reasons. First, Nick Burns is freakin' hilarious. If you have never seen an SNL Nick Burns skit, let me do you a  favor:

The second reason that I felt the above meme was appropriate was because Microsoft sure didn't do us any favors with how similarly these tools are named! Especially when it comes time to take the certification exams, trying to remember which functions matched to which tool when they all sound very similar can be a bit challenging. At any  rate, I have digressed quite a bit, so lets move on to the Microsoft Remote Connectivity Analyzer.

Microsoft Remote Connectivity Analyzer

This tool is simply awesome, and has the capability to help troubleshoot, diagnose, and verify settings for various public-facing applications. For example, it can conduct analsis on Office 365 setups, on-prem Lync and OCS environments, and on-prem Exchange environments, among others. The tool is a website, making it reachable from anywhere, and not at all burdensome to your local device. It can be reached at: https://testconnectivity.microsoft.com.

Once you have arrived at the landing page for the tool, you will immediately notice the availability to test the various server components via the tabbed structure, simplifying navigation. The various tests will help you verify things like your required public DNS records, whether or not SSL certificates check out as they should, and whether or not the servers are responding on the expected ports. They can also test autodiscover settings, and verify credentials for remote access. 

Obviously the best way to get familiar with the individual tests and analysis options is to visit the site and look around on each tab. To get even more familiar with the tool, try feeding it some values for tests that are relevant to your own environment, and explore the output.

Microsoft Connectivity Analyzer

See what I mean about the tools being named similarly? At any rate, this tool is called out as a companion to the MRCA tool, but unlike the MRCA, it is a tool that must be downloaded locally; it is not a web interface. To run this tool, your client must be running Windows Vista or newer, and if running it on a server, the server needs to be running Windows Server 2008 or newer. In each case, the .NET Framework 4.5 is required.

The Microsoft Connectivity Analyzer can be used by administrators and end users, alike. Its main purpose is to test connectivity between email clients and servers running Exchange server. This is where I must stop and ask, why couldn't this tool have been names something like "Exchange Connectivity Analyzer" or "Microsoft Email Client Connectivity Analyzer"? Honestly, wouldn't either of these have been more descriptive and less confusing for the overall collection of tools? I digress....The tool is simply meant to simulate various client logon and mail flow scenarios, and would therefore be a useful tool for related situations in which there is doubt about a client's logon ability, or what mail flow pattern is being experienced.

Microsoft Lync Connectivity Analyzer

This last tool, as you can see WAS named a in a little more descriptive manner. This tool has a very specific purpose. It is meant to determine if a Lync environment, either on-prem or Skype for Business Online, meets the necessary requirements for client connectivity from mobile clients and the Lync Windows Store App. While the name of the tool would almost indicate that the tool was used for testing all connectivity to a Lync environment, it is in fact only used for the tests I just mentioned.

The tool, in the course of its analysis, tests several components. Among them, it checks public DNS records to make sure the  necessary A records and SRV records are in place. It also checks the proxy configuration for the environment. Lastly, but definitely not least important, it checks on the validity of the SSL certificates that are in place.

Wrapping it up...

Well, that about does it for our review of a great set of tools for preparing for,  deploying, and troubleshooting Office 365. I hope you all have found this information helpful, and hopefully it will come in handy while you are transitioning your company to the awesomeness that is Office 365. 

Stay techy, my friends!

Lync 2013 and OWA Integration Troubles

Gratuitous Intro to the Situation

Ok, so we'll keep this short(ish) and sweet. Integrating your Lync Server 2013 on-premise environment with your Exchange Server 2013 on-premise environment is a fairly common goal within companies that wish to leverage even a few of the awesome "unified communication" features that come with the grouping of these awesome applications. Moreover, there are exactly 12,073,258 (Disclaimer: the preceding statistic is a completely bogus number) different blog posts and forum entries on the web detailing how to go about integrating Lync with Exchange. So, this issue assumes that you already have integration setup and verified to be working. At least partly working...

The Actual Problem

Some sporadic, not all, Exchange mailboxes were unable to login to Outook Web App (OWA) and successfully sign-in to Lync from within OWA. If the user is enabled for both Lync and Exchange using the same SIP domain (Example: @msucguy.com), they should be able to log into OWA, and sign in to Lync from within OWA, at which point they can initiate chats from their web browser. Below is what one would expect to see in such a scenario:

However, for the sporadic few users that were not seeing this, they would instead see the below message once logged into OWA:

The notorious, "There's a problem with IM. Please try again later. If the problem continues, contact your helpdesk" error. 

How to Troubleshoot

There are a few things to consider when trying to get to the bottom of this pesky error. Let's dig into a few of the key things to check for.

1. While this post assumed that the Lync integration with Exchange had already been setup successfully (partially, anyway), let's take a step back and forget about the fact that we already know some users are successfully logging into Lync via OWA. If we know of no instances of users accessing IM within OWA, we want to ensure that Lync does in fact trust Exchange (and OWA). To do this, let's open up the ever-so-handy-dandy Lync Server Management Shell. Once in the Lync Server Management Shell, we will run the cmdlet "Get-CsTrustedApplicationPool". We should see the below results, listing out your OWA URL up top, specifying your Lync Pool as the Registrar, and listing "urn:application:outlookwebaccess" in the Application field:
   
2. Next, with the Lync Server Management Shell still opened, we will run the "Get-CsTrustedApplication" cmdlet. The results should look like below. Note the port number 5199. If you have a pretty tightly locked down network internally, you will want to make sure that your Lync Front End servers can reach your OWA servers on port 5199.
   Ok, I realize that most of this image is "whited out", but you get the picture. Make sure your FQDNs are where the white out sections are, and make sure "outlookwebaccess" shows up where it should.
3. Now that we have verified that our Lync environment is properly set up to trust the Exchange servers, let's go back to assuming that we now know for a fact that IM works for some users within OWA. If we know that it does work, at least for some, then we need to verify that there is not, in fact, problem with "IM" (a.k.a., Lync Server 2013). Of course we can verify this by checking to make sure that we are able to log into our desktop or mobile clients with Lync, but this does not rule out that there is an isolated problem in the environment. To make sure all is well, quickly log into the Lync Server Control Panel (LSCP), and browse to the "Topology" tab. On this tab, ensure that all your relevant servers show up with a green check mark for their Replication Status, and then ensure that all servers report a healthy status for their services. We will assume that all is well on these checks, and that the environment is perfectly healthy.

The Resolution (at least in my case)

Alright, well now that we verified that Lync is in fact set up to trust Exchange and OWA properly, that the Lync environment is not having any sort of issues, and that IM within OWA does actually work for most users, we need to do some checking on the individual user level. A topical inspection of the user's settings within Exchange and Lync didn't reveal anything that stood out right away, so we decided to do a side-by-side comparison of a user that was experiencing the issue versus a user that was not experiencing the issue by looking at their properties within ADSIEdit. ADSIEdit allows us to get a really granular view of all possible attributes that can be set for a user within the realms of Active Directory-integrated applications, like Lync and Exchange.

Sure enough, there was a single attribute that did not get set for the accounts experiencing the problem: msExchOWAPolicy. And wouldn't you know, that attribute sounds like it just may have something to do with the problem we are experiencing. The value for the problematic user was set to "<not set>", when it should have been set to the Default policy, like so:

Technet does a dandy little job explaining how a default policy is created when Exchange is installed, and that this policy has ALL options enabled by default, BUT this default is not set on any mailboxes by default: https://technet.microsoft.com/en-us/library/dd335142%28v=exchg.150%29.aspx. So, if your Exchange admins/engineers don't have a provisioning process that sets this value for each user, or if the process allows for a user or two to slip through the cracks, you might just find yourself with a handful of users that magically can not sign into IM within OWA.

Well, I supposed I completely blew the attempt to keep this short and sweet, but I hope this is helpful for someone out there, and that they don't have to spend too long trying to track the problem down.

Stay techy, my friends!

The Rapidly Growing RTCAB Database Log File

​I wanted to share a little helpful info on an issue I have run into a few times with recent Lync Server 2013 Enterprise Edition deployments​​​​ regarding the backend databases that are stored on SQL Server. In particular, the 'rtcab' database is the source of the problem that I am covering in this post. Let me be more specific: the rtcab database log file is the source of the problem. Namely, for reasons that this post will not really explore or dive into, this log file can tend to grow very fast and very large.

For anyone that has every manually created SQL Server Mirroring, they understand how troublesome a Mirroring configuration can be to implement, and how frustrating it can be to troubleshoot. On the flip side of that coin, those same people truly understand how freakin' awesome it is that the Lync Server 2013 Topology Builder will completely take care of the Mirroring setup  on its own if the proper permissions are in place, including a Witness! With such beautiful automation in place regarding the backend SQL configurations, it can be easy to forget about the the SQL portion of the Lync Server 2013 environment all together; you almost have a false sense of "set it and forget it".

Obviously, it would be incredibly foolish to purposefully forget about the SQL environment after Lync installation, but let's just say in the excitement of finally getting all the pieces successfully deployed and having a functional Lync deployment, you forget to go back onto the primarly SQL server in the Mirror (or just the SQL server, if Mirroring was not set up), and set up the necessary maintenance plans. Why would you need maintenance plans for backups if you have a real-time copy of the databases on your Mirror at all times, you ask? Well, Mirroring requires that the databases have a Full recovery model, which means that their logs will continue to grow, without being truncated, until they cannot grow any longer.

Going back to our hypothetical, you have this successfully setup Lync environment, and a pristine SQL Mirror setup, and life is good. Then suddenly life is not so good when suddenly your address book in Lync is not displaying all the results that it should. Eventually you discover that the Address Book is either corrupt or not functioning propery, and you discover that the drive on your SQL server that contains the database log files is full. Why is it full? Well, that pesky rtcab database gets a LOT of transactions and updates, for some reason, and that transaction log has grown to a few hundred GB in size! None of the other logs may have grown much at all, but that rtcab log is just ornery.

So, first thing to do is realize that we should have made a maintenance plan within SQL Server Management Studio from the get-go to keep our databases regularly backed up with a Full backup, and then a second maintenance plan to backup all the Transactions Logs, which would in turn truncate the logs. However, in many cases, an hourly transaction log backup might be necessary for keeping the rtcab log from growing, Alright, so we are a bit lit, but we get those maintenance plans in place, and try to manually backup the database and transaction log, which is successful, but we notice that the transaction log is still several hundred GB in size. So, we try to shrink the rtcab log file, and while it seems like the command completed successfully in the Management Console, the file is STILL the same size.

Next, we turn to good ole' Google to find out why the Shrink command is not working on the log file (make sure when you try to shrink the log file in the Management console that you choose "Log" from the drop-down instead of leaving it on the default of "Data"). Google isn't much help because all that every DBA and their mother wants to do is scream about why "you should never Shrink the database; best practice is to have full backups and transactional log backups running on a schedule." This is the point where we are thinking, "Yeah, thanks for that overwhelming helpfulness, DBAs. We are aware of what Best Practice is, and will be using that moving forward, but how in the world do I free up the hundreds of GBs on my drive in the mean time?"

So, now that I have made you read through all this to find out why that rtcab log file won't shrink after using the Management Console, or using the DBCC SHRINKFILE command, the answer is simple: because the log is still being utilized in the Mirroing relationship, it cannot be shrunk. You must go into the Properties of the database, go to the Mirroring tab, and then Pause the mirror. This leaves the Mirror in tact for the database, but shows a status of Suspended. Once this is done, you should be able to run the Shrink operation, and the file should instantly shrink down to almost nothing. BOOM! You now have hundreds of GBs of space back. And as long as your Maintenance Plans are setup properly with a retention period (use a Maintenance Cleanup Task), you will keep your space back! Oh, almost forgot, don't forget to go back in to the Mirroring tab and Resume the Mirror!

Stay techy, my friends!