William T. Esrey Sprint Corporation 2330 Shawnee Mission Parkway Westwood, KS 66208 Mr. Esrey, I am a beta deployment customer of Sprint ION. The Sprint ION technology is obviously strategic to your business, but the service is presently so fraught with political turf battles and systemic problems that it is difficult to see how its technical issues can be resolved with its present organization. During the beta cycle, I have identified some issues, which, as a real, paying customer, I would consider unacceptable. Most people think of a beta cycle as a stage to identify and troubleshoot problems with hardware and software. However, it is also a good time to identify and troubleshoot problems with customer support procedures and equipment. It is my suspicion that these issues are not being made clear to top-level executives at Sprint, and that it will take definitive action from that level to effect positive change. Slow Response to Voice Outages Voice service not being functional is a very big deal and the response needs to be immediate and definitive, like the response of the fire department. For over 100 years, customers have been able to pick up their home telephone and expect a dialtone. This expectation has been met, in general, over 99.9% of the time-even if the power is out or the weather is inclement. Sprint ION customers are unlikely to change their expectations of reliability, nor are they likely to be willing to accept a slow response. An immediate response, combined with a bias for action is essential. What is an immediate response combined with a bias for action? Lake County 911 receives a call that the old warehouse at 211 East Main in Westville is on fire. Actually the warehouse is half located in Eastville and half located in Westville, but because it's closer, 911 dispatches the Westville fire department. They arrive 3 minutes later at the scene of a 5-alarm blaze. The fire department locates a hydrant, connects a fire hose, and starts spraying water on the fire. Clearly unable to deal with the problem with only their limited resources, the Westville fire department requests mutual aid over the radio. 5 minutes later, the Southville fire department arrives, closely followed by the Northville and Eastville fire departments. In this case, the most likely cause of the problem is fire. The most likely solution is extinguishing the fire with water. Of course, it is possible that there isn't a fire at all-or that the fire can't be extinguished with water and requires chemicals to douse. It is also possible that the fire started on the Eastville side of the warehouse, which means that the fire might not technically be the Westville fire department's problem and Eastville should be calling the shots. However, a fire is burning, lives and property are at stake, and nobody thinks about these things-such concerns, even if technically valid, are unimportant in light of the larger issue. How could Sprint ION improve? If Sprint ION were in the business of putting out fires, they would not put out very many of them before the buildings burned down. When a customer calls (presumably from a payphone or cellular phone) to report an outage, he is often greeted with a long delay before the call is answered. After the call is answered, the standard procedure is to ask the customer to reset the router-which means that the customer may need to leave the location (pay phone) from which he is calling, and go reboot the router, only to discover that the problem has not been solved and have to wait again to get through. After a genuine problem is uncovered, there may be multiple fix agencies in disparate geographical locations that need to address the problem. Some agencies may be unable to effectively diagnose or troubleshoot a problem on their own, and the others who could help might be unable or unwilling to assist. Finally, internal politics apparently make some agencies unwilling to take a definitive response to a possible problem, unless all other possibilities that don't point to them as a cause are eliminated. An Example of Current Resolution Procedures My voice service went out on Friday morning. I could make or receive any telephone call I wanted, as long as it was no more than 20 seconds in length and as long as I only needed to hear the other party speaking and they didn't need to hear me. My data service also experienced a substantial degradation in performance. I called to report the problem, dutifully reset the router as instructed (didn't fix the problem), and heard nothing at all back until the following day. The following day, I was awakened at 8AM by the ISMC and asked to verify whether the problem still existed (it did), then reboot the router. This didn't fix the problem, any more than it did the previous day. At that point, I was informed that the ticket would be "escalated to tier 2." The escalation procedure evidently consists of putting the customer on a 12-hour long teleconference, the majority of which consists of fixing problems in Kansas City (I am in Seattle). During the time that my problem was addressed, I asked two questions-"When was this last working, and what has changed since then?" The answer to the question came after a flurry of e-mail, since there is apparently no centralized point where maintenance is tracked: my service was last working before Sprint changed to a Lucent 24-port DSLAM line card, and what changed since then is that Sprint installed the 24-port DSLAM line card. The obvious solution, according to Sprint? "Did you reboot your router? How about your computer?" "Now let's fix a bunch of software problems in Kansas City that probably don't have anything to do with you, but might!" After some additional research, it was also discovered that the other two ION customers in the same central office also lost voice service the day that the 24-port DSLAM line card was installed. The obvious solution, according to Sprint? Call the other customers to find out if their data was also slow, notwithstanding the identical voice issues. When I suggested that Sprint take a bias for action and replace the line card, I was rebuffed with "Sir, we will solve this problem our way." More than 72 hours after originally reporting the outage, my service is still not restored-although the BLNO finally dispatched a technician to my central office to test the 24-port DSLAM line card (curiously, what I suggested yesterday morning, but wasn't the Sprint ION solution). Five Suggestions for Improvement * A unified trouble ticket processing system with severity one problems placed at the top of the queue, with aging. When stuff breaks, people who can fix it need to know right away, and the oldest problems need to be fixed first. There also need to be enough people to fix multiple simultaneous problems. Idea: You can save 100% of your headcount budget: just lose all of your customers. Having insufficient resources to solve customers' problems is a sure path to these savings. * A centralized maintenance tracking system. Anything done by any fix agency that might affect a customer's service should be easily available for view. That way, if a software upgrade takes place and suddenly 5 similar trouble tickets come in from the same area where the software was upgraded, you have a prime suspect. * A bias for action in problem-solving. You may not be able to definitively tell whether the software upgrade in the above example is to blame. But if that is your prime suspect, the first step should be backing out the upgrade to see if the problem goes away-not the last step. In other words, fix it first and then worry about why it didn't work. * "Five alarm mutual aid." If you can't fight a fire on your own, you can't be too proud to ask for help. If you are, the building will burn down. Similarly, if someone asks for your help, you can't be too proud to provide it-or when you need help, your buildings will burn down. When Sprint ION divisions fail to assist one another, withhold information, or attempt to blame one another (or their vendors) for problems, it slows or prevents resolution of the customer's problem. * Acknowledge that the average customer doesn't care why it does not work, just that it is not working (and he's mad if it doesn't). You don't care whether there were 5.5 regression issues with the Lotus Notes connector on NT4 service pack 3 and above in a mixed-mode Exchange beta environment, you just want your e-mail to get through (and you're mad if it doesn't). Although I am not an average customer, the average customer doesn't care whether a vendor, an ILEC or another fix agency is to blame for the problem. He just wants his phone calls to get through (and he's mad if they don't). Idea: Customers don't stay mad for long before they're former customers. Lack of Communication With Customers This problem takes three forms. The first is that customers' concerns are often miscommunicated, so that technicians try to solve the wrong problem or believe that there isn't a problem and give up. Then, rather than communicating directly with the customer, technicians simply resolve the ticket and send it back to the customer service department for confirmation of resolution, adding another layer of bureaucracy and a possible lengthy delay when the problem is not fixed and the trouble report was not understood. The second is that it's assumed that all customers are non-technical and that they cannot provide information that is not obvious to troubleshooting tools (and that those troubleshooting tools always report correct results). The third problem is that customers are not informed of the progress of their service order. Running Around In Circles Some technical problems are difficult to describe, as well as transcribe. Additionally, there can sometimes be multiple, related problems with service. If the problem is not effectively communicated, then it might not be solved. For example, I opened a trouble ticket that Sprint titled "10 digit local numbers fail to connect." The actual problem was that when a number within the same NPA that was long distance was dialed as 7 digits, the improper announcement played (the proper announcement is that 11 digits are required for long distance calls, the actual announcement was that the number is invalid). Obviously, a much different problem. To the ISMC's credit, they had their technicians contact me to perform a call trace, and that gave me the chance to explain the actual problem (which was eventually fixed by a Telcordia patch). However, similarly unclear trouble tickets that don't require customer interaction could result in improper repairs, or no repairs at all. It is essential for a technician to contact the customer and confirm the nature of the problem, then call again after performing repairs and make sure that the problem has been corrected. It is also essential for trouble tickets to contain enough examples and information for technicians to resolve them effectively. You can't always believe the computer (or the vendor) If the Lucent 24-port DSLAM card that was installed in my central office is to be believed, my DSL loop operates at 998Kbps upstream and 8Mbps downstream. This sounds impressive, especially since according to Sprint ION's engineers, similar results are unattainable in the laboratory with only 20 feet of wire (much less the 1.1 miles of "real-world" cable between my home and the central office). However, Sprint ION's technicians were insistent that the same card's troubleshooting results were unimpeachable after resetting the card and manually forcing it to 6.5Mbps downstream and 640Kbps upstream (the settings maintained by the previous 12-port card). It is my suspicion that these results may not be correct. When other evidence contradicts troubleshooting tools, further investigation is in order-especially when dealing with notoriously unreliable Ascend/Lucent equipment. Idea: If you're too cheap to buy good (read: Cisco) equipment, you can't be too cheap to have people on hand to fix it when it breaks. Not all customers are dumb, and many can help you The first step in troubleshooting any technical problem with a customer should be confirming the symptoms that the customer is experiencing, if they cannot be obviously determined. With a high-tech service such as Sprint ION, the customer base is likely to be, at least initially, reasonably technical. For example, I have over 8 years of experience in the technology industry, have a degree in computer science and numerous technical certifications, extensive experience in the networking, datacommunications and telecommunications fields, and work for a software company writing complex technical documentation for mission-critical server platforms. I am also, without question, the expert on the symptoms I am experiencing when I report a problem. Granted, this does not make me an expert on Sprint ION in general. However, given my experience with similar technology, it is very possible that I could be helpful to you in diagnosing problems with my service-and I expect that this could be the case with many of your customers, especially beta test customers. This can only happen, however, if there is a two-way flow of information concerning the problem, and information provided by the customer is considered as part of the possible solution. Not fixing the most obvious problem just because the customer pointed it out and "he doesn't know anything" is self-defeating. Even if the customer might be wrong, if he has a good point it should be followed up. In this light, it may be tempting to exclude customers from the problem-solving process entirely, in order to avoid possible embarrassment by a customer pointing out something painfully obvious, or possible inconvenience by a customer demanding a fix that might not work. However, it shouldn't matter where a solution came from-all that matters to the customer is that his problem gets fixed. Idea: "You mind your business and we'll mind ours" is a bad attitude, because your business and the customer's business are inseparable. The customer wants to stay informed In general, customers are also very forgiving of technical problems if definitive action is being taken to solve them. However, if the customer is not informed what the problem is, and that steps are being taken to correct it, he'll feel neglected, anxious, uncomfortable, and eventually angry. When a severe problem occurs, the customer should be informed in whatever way possible of the nature of the problem, and the steps being taken to solve it. This,. of course, only works to a point. Waiting a day before calling back after a trouble ticket is opened on a severity 1 problem, and then taking 2 additional days to fix it, will stretch the patience of even the most understanding customer. Additionally, frequent, severe problems will also test the patience of customers. However, the importance of good communication cannot be underestimated. In Conclusion Sprint ION integrates so many technologies, many of which are so immature, that outages will occur. Consequently, Sprint ION must create an action plan for a quick, definitive, and action-oriented response. Ideally, this response should be transparent to the customer, but there are many cases in which it cannot be-and transparency is much less important to customers than prompt restoration of service. If Sprint ION were strictly an Internet service, slow response and poor service would not substantially differ from the rest of the industry (of course, there would be nothing to set it apart from the rest of the industry either, so customer acquisition costs would probably be high). When asking customers to trust Sprint with their telephone, however, the service expectations are substantially higher. Customers simply do not expect their home telephone to crash, and they especially don't expect a 3-day wait to get it fixed if it does. It is my sincere hope that you can effect positive change in this area.