You probably haven’t thought much about what happens when you make a phone call. Neither did I until my friend and tennis partner Fred Rosenberg explained it. I was fascinated, and for years had planned to get a few more details from him so I could share it with you. But now he’s gone, having recently died unexpectedly. He was in his late 60s and at the peak of his career, and his tennis game.
When you punch in the numbers to make a call, it opens a connection with a powerful computer called a switch that’s operated by your phone company. The switch then routes your call across the phone network.
When you hang up, the switch notes the length of your call and creates a record in a database. But there’s still more for the switch to do. It then needs to rate the call: that is, it needs to figure out how much to charge you and how much the call is costing the phone company.
Let’s say you make a call to the UK. The switch looks up your account in a separate database and sees how much you pay for calls to the UK. (A corporate customer with 500 lines will likely be paying less for that call than you.) The switch also needs to connect with a database of the network in the UK that handles your call, to see how much the company is paying the network.
Here’s the kicker: After Fred finished describing this to me, he said, “And all of this takes place in a fraction of a second.”
Fred’s job was to administer the call-record database used for billing. The whole company, known as XCast Labs, which currently has about 30 employees and is headquartered in Los Angeles, depended on the database working. That level of responsibility would keep me awake at night, worrying that things would go wrong.
And they did go wrong. Often at our weekly tennis match Fred would tell me about the latest crisis.
There are actually two call-record databases. One rates the calls in real-time, as they’re made. This is a quick-and-dirty rating to track activity and send alerts of fraud if suddenly an account was making thousands more calls than usual.
The other, which Fred administered, would access that real-time database at 2 a.m. every morning and download the previous day’s call records. Fred’s database would then rate all the calls again, but more thoroughly. For example, if a customer exceeded a specific data allotment on a cell phone account, this rating would know to charge for the overage. Plus, it would add in any extra surcharges, and calculate taxes.
So what could go wrong? One challenge Fred faced was simply the growth of the company. His database was rating perhaps 5 million call records a day when we first talked about it. But then that came close to doubling, and his server was only capable of rating 10 million calls in 24 hours. If there were more than 10 million calls, then he’d be in trouble because they’d already be in the next 24-hour period but would still be rating the previous batch of calls.
Because they’re a small company, adding an additional server is a big expense. Instead Fred would try to figure out how to make his database more efficient, so it could rate more calls in a 24-hour period. And he did. Sometimes his experiments would break things, but then he’d get it working again.
Still the volume increased, and so the company eventually bought a second server, which could in theory handle an additional 10 million per day. But you can’t have two separate databases. Fred’s next challenge was to figure out how to make those two servers work in concert. After trying various approaches, he got it to work seamlessly.
But still the volume increased. Soon they were bumping up against 20 million per day. Fred then came up with his most elegant solution yet. The brains of their servers each had 16 cores—almost as if there were 16 separate processors. Because today’s chips are reaching the speed limit for silicon technology, chipmakers have increased power by using multiple cores.
But software often can’t take advantage of this extra power. Fred figured it out, though. He found a way to set up the server so that a sub-group of cores would act like a single computer. In essence, a single server was now like four different computers, with each capable of rating 10 million calls in 24 hours. Suddenly the company had much more capacity. Which was good, because at the time of Fred’s passing, they were handling over 30 million calls a day.
Fred was a hero, working quietly in the background to keep the call-record database humming so that the monthly billing could go off without a hitch.
If Fred’s small company handles over 30 million calls a day, imagine the volume a much larger company would be handling. And imagine similar huge databases in other industries around the world. Working quietly in the background behind each enterprise are the Freds of the world. I salute them, and I salute Fred.
See column archives at jimkarpen.com.