This post takes a look at Covid data with a particular focus on the number of new daily cases and the growth (or reduction) of those daily cases over time. If this were physics, we’d be looking at speed and acceleration, rather than the total distance traveled. I won’t try to convince you of anything, but rather just try to build an understanding of where we’ve been, where we are, and what to expect in the next few months.

Let’s start with the growth in daily cases for US states since March 10th, for states reporting at least 20 cases:

Each dot represents the growth in the number of new daily cases for a US state on a given day. I discuss methodology further at the end of this post if you’re interested. ^[1]

We can clearly see a few crucial trends in this chart. Growth was furious for all states in mid-March (20% daily growth means doubling in 3.8 days, as you’ve surely heard) and showed a lot of variance. Then nearly all states issued stay-at-home orders between March 23rd and April 3rd. ^[2] These orders, no doubt coupled with some amount of anxiety and precautions from the population, quickly reduced growth rates, which were clustered around 0% by mid-April. This was a significant accomplishment. Sadly, we were unable to improve from there, and never brought growth figures consistently or substantially below zero. Here’s the same data seen by week in a slightly different way:

We started out red-hot and worsening in mid-March, but that gave way to slower growth and calmer colors. The initial success was followed by stagnation, and slight worsening in the last two weeks. Let’s look at our nationwide figures:

New daily cases peaked on April 10 in the US at about 32,000 cases/day. They have since fallen to 21,500 cases/day. ^[1:1] Growth peaked at 40% on March 24, shortly before the lockdowns started, then fell sharply hitting 0% on April 15.

Now consider this: we had about 7,000 cases/day on March 25, as we headed into lockdowns, and we have 21,500 cases/day now, as we are leaving them. That might feel a little disheartening. What happened? Was there any point to this whole thing? Did we just destroy countless jobs, businesses, and dreams for no good reason?

There are three good answers here. The first is that the precipitous fall in growth brought about by the lockdowns was a major win that probably averted total disaster. However, unless you look at a plot of growth rates, or at least look at daily cases and appreciate the trend, this win is somewhat hidden. I hope the charts so far have done a decent job of showing this aspect of our journey.

The second is that the lockdowns were indeed somewhat pointless. Not because they are inherently so, but because we’ve done a bad job and failed to significantly bring down case numbers while we had a perfect opportunity to do it. We bought the lockdown with trillions of dollars and untold sacrifice, and then squandered it.

The third answer is that we have to consider states separately to really analyze the situation, because national data is just too blunt. States had varying levels of success and peaked at different times, and to understand what worked we need to factor that in.

Let’s look at what other countries achieved with their lockdowns:

Those curves show the kind of drastic reduction in the number of daily cases that well-organized societies can achieve. They are able to push growth significantly below zero and keep it there long enough to bring case numbers down an order of magnitude or more. A smaller outbreak is then more amenable to containment by well-design policies while economic and social activity is restored.

Let’s look at more countries for better context. Here are the ten countries most successful at containing the pandemic from a peak of at least 70 cases:

I have excluded China from the list due to controversies around their data. They would have been 4th place with a 99.8% reduction from a peak of 4,687 cases/day. We see some islands in there, some smaller populations, and also small peaks. It’s worth pointing out that neither islandness nor a small population are any guarantees, as the history of smallpox in Iceland can attest. ^[3] Still, countries like Switzerland and Austria vanquished pretty large outbreaks and are not islands last I checked. Social cohesion and good policies seem like the overriding factors. But let’s look at a more diverse group of places:

Sweden is the only wealthy country in this list doing worse than the US. This was not cherry picked: that remains true when you look at the whole world, where the US ranks 62nd by this metric. In the last week Sweden’s top epidemiologist has admitted mistakes in their strategy. ^{[4] [5]} However, the overall number of infections is low in Sweden, and their growth has been kept mostly in check, never spiraling out of control. They are a highly conscientious society that took a daring (and often misrepresented) approach with a clear understanding of the trade-offs involved.

The situation in the bottom countries is catastrophically different. They all have strong growth of already sizable outbreaks, with Brazil in an especially dire situation, no doubt the worst in the world, having recently overtaken the US for the top spot in daily cases amid continued growth. Their president is now attempting to censor Covid numbers, and it’s possible Brazilian data will no longer be reliable over the next few weeks. ^[6]

Even if we ignore any mistakes made before mid-March, it is clear from this data that the US has not done a great job containing the pandemic. Despite remaining in a fairly strict lockdown for weeks, we performed worse than all but one rich nation in reducing case numbers. But let’s not yet worry about whether we’re a failed state or have been made great again.

After all, the US is a large and heterogeneous place, and looking at national aggregate data obscures a lot of the story. States like Alaska, Montana and Wyoming never had more than 25 cases/day, while New York reached 9900 cases/day, a peak greater than every nation’s except for Brazil and Russia. Having seen what other countries look like, here is what happened in US states with a peak of at least 70 cases/day:

A handful of states managed substantial reductions in daily cases, including New York, which had by far the largest outbreak in the US. That’s cup half full. Still, at 91.3% decrease New York is behind most developed countries. It is striking that none of our states have managed to do as well as Spain, Italy, or Germany when it comes to reducing case numbers.

And then there are the states at the bottom of this list. When you see 0% that means no reduction: these states are currently at their historical maximum and growing, and we don’t know when and where they’ll peak.

Keep in mind the decreases in the chart above show the reduction in each state’s daily cases measured against its own peak. To get an idea of how states changed since the national peak, and how the outbreak decreased in some areas and increased in others, here are the most substantial deltas in daily cases by state since the US peaked on April 10th:

Since we peaked nationally on April 10, we have reduced daily cases by about 10,500/day, with most of the reduction coming from New York (9,000 cases/day) and New Jersey (3,000 cases/day). It might strike you as odd that the national decrease (10,500 cases/day) is smaller than the decrease from just New York and New Jersey (a combined 12,000 cases/day). And sure enough, if we exclude those two states, daily cases have actually increased in the rest of the US since our national peak. Without NY and NJ, on April 10 we were at 18,300 cases/day, then we peaked on May 6 at 21,400 cases/day, and are now at 20,000 cases/day, for a reduction of 7%.

So let’s talk the future and make some predictions. Think about these two questions:

1. How many states will see a daily cases peak at least 30% greater than any peak they’ve had so far?
2. How many states will be forced back into lockdown?

Then consider these facts: compared to other developed nations we have done a much worse job reducing our outbreak; we did not use our lockdown period to develop comprehensive policies to fight Covid; we have not used leadership to galvanize the population to fight the pandemic and adopt practices that mitigate spread - quite the opposite, we have started a culture war around wearing masks, social distancing and whether to even take Covid seriously; many American leaders undermine mitigation by deed and word; even while in lockdown, we have only been able to achieve modest daily reductions in case numbers; people feel like they have done their duty and should now be able to resume life, being generally sick of hearing about Covid and all its controversies and conspiracies; places highly prone to spread, such as gyms, churches, and restaurants, will resume operations; domestic travel will resume so that any counties with larger outbreaks might seed those with fewer cases; finally, if daily growth increases even to a modest 5%, cases will double in two weeks under the inexorable march of exponential increase.

Offsetting these is the fact that a large part of the population is much more careful and attuned to the spread of Covid. Humans are remarkably adaptable, and maybe smart on-demand interventions at the county and state levels can curb local outbreaks.

Before answering those two questions, let’s take a look at the familiar case-and-growth plots for the 40 states with cases/day currently over 70:

Many of those curves don’t look great. Keep in mind some of the spikes we see mid-graph are due to specific incidents like outbreaks at a prison.

But enough of the charts, let’s try our hand at divination. Only eight states have managed a decrease of 70% or more in their daily cases (nine if we count Pennsylvania at 69.8%). These are the states most likely to keep things under control: most have seen a serious situation, all have been effective by US standards, and they are further down from their peaks. I’ll round up and say 10 states will avoid a greater peak in the future. The other 40 will see a peak at least 30% greater than their current peak. And of these 40, at least half will adopt lockdown measures before the end of the year that affect a majority of their population.

This is all the data I’ve got for now, but if you’ve read this far, you might as well stick around for a few broader considerations.

First, the trade-off between economic outcomes and epidemiological outcomes has become grossly overstated. The more infection we have, the more the economy will be affected as people shy away from economic activity.^[7] A failure to intelligently fight Covid is an economic failure as well. Brazil is a sad example of this, as the out-of-control outbreak has wreaked havoc in the economy.

Almost every containment strategy - personal behaviors, contact tracing, widespread testing, effective quarantine of sick patients, etc. - ultimately benefits the economy. Every leader who has mocked or sabotaged Covid containment is hurting economic output. And plenty of economic activity can be encouraged with low risk, especially if smart mitigation is applied. Even where a trade-off seems obvious, say opening up restaurants without restrictions, things are not so simple: the net economic effect needs to account for the consequences of the greater spread of Covid, which unfortunately is very likely in restaurants.

The trade-off is much more direct when it comes to personal freedom. Church services are a perfect example. They are simultaneously: 1) prone to spreading Covid, 2) not responsible for a lot of economic activity, and 3) extremely important to a large part of the population.

Or to pick a different demographic, look at skiing in Colorado. Plenty of people here would be willing to risk infection in order to ski, yet this choice was denied to them. This may seem like a trivial sacrifice, but to many it is deeply meaningful. Skiing is a complex trade-off since it does involve a lot of economic activity and also enormous Covid risk, as we saw when tourists started various outbreaks in our ski towns. Yet there is also a strong personal freedom component embedded in it. It is interesting that the restrictions which most incensed Michigan protesters were related to personal freedoms, like the use of personal boats.

The moral calculus around Covid trade-offs is complex. Risk to self; risk to others you might infect; risk to society at large if we overrun the health system; how to weigh death against hardship, enjoyment, and freedom; how much we value the life of elderly people and those at greater risk of complications, and so on.

But there are a lot of actions and personal decisions that remain invariant no matter how you feel about trade-offs. God knows we are all sick of Covid, now that the novelty wore off and this looks like a long haul. But stay as safe as possible, and for whatever degree of risk-taking you decide on, mitigate as much as possible.

I hope this has been useful and informative. Thanks for reading!

I’ve been doing analysis of Covid cases to try to understand what to expect in terms of lockdown length and disease progress, especially in Colorado and Brazil, the places I spend the most time in. There are a lot of data sources around, and it took me a few hours to find and test a number of them. I hope this saves time for anyone interested in crunching Covid numbers. If you have suggestions and tips on data sources, please open a PR or issue in my Github repo. Here we go.

John Burn-Murdoch and his team at the Financial Times have done a great job reporting visually on the pandemic. They have fewer and simpler charts than many other sites but their charts are done exquisitely and distill a lot of data to provide you the clearest picture available of each country’s situation, plus a few of the regional hotspots around the world.

Our World in Data is a wonderful project based at Oxford University that attempts to explain the world using rigorous data sources and beautiful charts. They have been producing a lot of great Covid content since the pandemic broke out. If you have some time, I suggest exploring the non-covid areas of the site as well (and if you enjoy that, I highly recommend the book Factfulness). All of their work is open sourced.

The OWID data is in a GitHub repo. Their main source is the European CDC, which publishes confirmed cases and deaths aggregated by date and country for most of the world (not just Europe) in JSON, CSV, and XML files.

Johns Hopkins University has built a wildly popular dashboard available in desktop and mobile versions. Their repository is public and it aggregates data from a variety of sources into easy to use CSV files (there’s also a JSON mirror). In addition to worldwide national totals, data is available for individual US counties and states. It includes number of cases and deaths along with recovered and active patients. Since they aggregate data from the US CDC, China CDC, European CDC and several other national institutions, this is a great way to get your hands at worldwide data.

The New York Times offers a plethora of high-quality Covid maps and visualizations. It’s not a surprise Mike Bostock, creator of the D3.js library, used to work there. The NYT open sourced a repository providing high-quality and painstakingly verified data for US cases at both the state and county level. This is probably the best source of data for analyzing number of cases and deaths in the US.

Another outstanding US data source is the Covid Tracking Project, powered by dozens of volunteers attempting to collect data on number of tests performed, positive and negative results, hospitalizations, patients in the ICU and on ventilators, and so on. They face a severe dearth of information in the US and the complete lack of centralized reporting, but they’re making the best of it. If you want to attempt more sophisticated analysis, this is a good source. But mind the gaps.

I’m sure you’ve hit Worldometer while googling covid information. They provide encyclopedic amounts of data about Covid infection worldwide through an effective bare-bones interface with good charts. Data is aggregated by country and includes deaths and active cases, both by day and totalized.

Finally, if you are interested in more regional data for other countries, there are great repositories for Spain and Italy. It’s not easy to aggregate UK data, but Tom White has a good repo. Álvaro Justen has done the same for Brazil, while research lab Fiocruz has a good web UI for Brazilian data.

If you know of other high-quality regional repos, please send a PR or GitHub issue. I’d love to expand this post with the best repos for each region.

(Some security recommendations are summarized at the end.)

I. The Robbery

This past summer I was walking around in the neighborhood where I grew up, happy-go-lucky, when some guy jumped off a motorcycle pointing a gun at me. It was my first time at gunpoint, and from the outset the weapon was positively spellbinding. As I gazed at it, strange thoughts hit me: “Am I going to get shot by this rusty piece of shit? What a sorry way to die! And what if I get tetanus?”

Those were thoughts I wouldn’t have anticipated, but as Dan Carlin says, humans in extreme situations often behave unexpectedly. And while a gun-toting thug is a far cry from the Battle of Verdun, it is pretty extreme for me. This post tells the story of the robbery and its surprising information security developments. There are lessons here for both users and designers of technology.

My daughter and I were visiting Brazil in July, taking a carefree walk in a boulevard lined with lush trees. She had just gotten into “good kid, m.A.A.d. city”, ironically enough an album about growing up in Compton amid dire violence. So we were deep in conversation about the US criminal justice system, drug laws, and the ideas of people like Bryan Stevenson and Michelle Alexander^[1].

Growing up in Brazil you get a crash course in street smarts. I was mugged twice as a 10-year-old and once at 15. That’s counting only the times when stuff was actually taken. There were scores of near-muggings I dodged by either talking my way out or running my way out.

But after 20 trouble-free years, I let my guard down. Absorbed in conversation, I barely noticed the motorcycle driving on the other side of the street. By the time it veered the wrong way into traffic and sped towards the sidewalk we were on, it was too late. The passenger jumped out while the motorcycle was still moving, gun in hand pointed squarely at me.

The scene felt strangely removed - it’s cliche, but it really did feel like a movie. Instead of panicked confusion, there was a strong pragmatic voice in my head. I had thought about “what if” scenarios plenty of times before and they kicked in. Who is the attacker? What is their motivation? What’s the best course of action?

There are career criminals in Brazil who are downright professional. I know somebody whose house was invaded while they were home and the robbers let them know how long the “job” was going to take, offered them water, and made sure nobody freaked out. Better than some moving companies I’ve used.

But when someone is robbing random people on the street using a gun, that’s pretty far from professional. Way too volatile a situation with huge risks and beggarly payoff. These were at best lowlifes and at worst jittery crackheads. I felt two strong imperatives. First, keep the situation as absolutely relaxed as possible. When they get nervous, they get scared. And when they get scared, that’s when I accidentally get shot. But second, and more importantly: if they want to kidnap my daughter, fight it at any cost whatsoever. Better to die on that sidewalk than to let them take her.

I remember thinking, “take a deep breath, raise hands slowly, move smoothly, stay relaxed, hand everything over.” It worked. Who knows, maybe Andy from the Headspace app saved my life. The bandits were gone, along with our two iPhones and my watch. But the real fun was still to come.

After we got home, I logged into iCloud and put both phones in lost mode. They had been turned off, predictably. Plenty of crooks have been caught by way of “Find iPhone,” but they’ve learned by now. Thinking of my data in criminal hands was uncomfortable, but the fact that iOS exploits sell for $1.5 million made me feel a lot better. No small-timer is breaking into an iPhone. I figured they would wipe it out and sell it.

I have two-factor authentication in all the accounts that matter, and whenever possible my second factor is an iPhone app that generates time-based one-time passwords (TOTP) for authentication. Google Authenticator is a popular app for this, but I use OTP Auth instead because it is more flexible (more on this in the recommendations). Here’s what it looks like, slightly sped up to make it more exciting:

When it’s time to log into one of your accounts, you provide your login and password as you normally would, plus the temporary code being shown by the app.

I also use a password manager with unique, long passwords for each site. So my main concern at that point was minimizing the impact of this whole thing on my kid. We had dinner planned with friends, tasting menu at a good Japanese place, so I thought it best to go, have a good time, laugh and hopefully cushion the blow. Later I could call T-Mobile and suspend the cell lines.

II. The Hacking

A couple of hours later we were back, much happier, imbued with friendship and, in my case, plenty of sake. I opened Gmail and got some shockers:

Wuh-wait what? I wasn’t expecting to see any of these, but least of all the Facebook password reset. Before you read on, take a good look at those emails. It’s fun to work out what happened here. Done? Let’s dig in:

Whoah! Facebook password reset by phone number? How? Did they unlock my phone? But also… why? At once I felt the sinking realization I misread the situation. They seemed to be more sophisticated that I thought - not the motorcycle crew themselves, but someone else in the operation (his identity would be unmasked later that night).

The idea that somebody was hacking into my accounts right at that moment, with my phone in hand, was deeply unsettling. A malevolent twist to the emotional roller coaster of that evening. But this was a technical problem, so it was time to sober up as best as I could and work methodically.

The “how” was simple. The attacker took the SIM card out of the stolen iPhone and put it in another phone. At that point he found out my phone number, whereas previously he had no information on me. More importantly, he could also receive my SMS text messages. He then attempted to log into Facebook using my phone number as a login, clicked on “Forgot Password,” and reset the password via SMS.

So here is a big screw up and a couple of lessons. As I said, I have 2FA (two-factor authentication) in the accounts “that matter.” But I rarely use Facebook, so I didn’t enable 2FA there. Oops. Turns out it’s not such a great idea to have an account in the world’s most popular app as a weak link in your defenses.

Now consider Facebook’s account recovery policies. If the account has 2FA enabled, passwords can only be reset by email. That’s good. But without 2FA, if an account has an associated phone number, the password can be reset via SMS. In such a case, a SIM card is an instant ticket to the account: find it and reset its password in one fell swoop.

That’s a disaster. Facebook single handedly provides a way for attackers to go from a SIM card, or hijacked SMS messages, to a trove of personal information for the vast majority of people out there. By contrast, attackers made zero progress in hacking my kid’s accounts, mostly because she doesn’t use Facebook.

But why the hell was this wretch logging into my FB account? I suspect it wasn’t for my cousin’s mad political rants. Already shaken from the armed robbery, my mind played tricks on me as paranoid thoughts of identity theft and fraudulent bank transfers loomed.

I immediately logged into t-mobile.com and suspended both cell lines, disabling the attacker’s main weapon. As an aside, T-Mobile has been great for international travel. I love you guys, keep your website safe. I tested sending SMS messages to my suspended numbers and happily all attempts generated errors.

On to the other emails. The Facebook password reset arrived at 9:46pm Brazil time. Curiously, at 11:23pm they briefly turned my phone back on with its SIM card, and the phone went into lost mode and flashed on Find iPhone here. But then there is that fourth email with a subject line of “iPhone SE 64GB Silver Was Found!” arriving at 12:20am. Here it is:

The phone model and storage capacity are exactly right. The spelling, grammar, and layout are pretty well done. It was sent to my primary personal email, lifted from Facebook. Imagine a regular user receiving this right after their phone has been stolen, while they’re somewhat shaken, and when they’re not native English speakers to boot. What are the odds they’ll realize this is a phishing attempt for iCloud credentials?

Apart from checking the URL, the biggest clue is the exclamation mark in the subject line, a little too enthusiastic for Apple. Either way, this is a nearly perfect phishing piece, made more so by impeccable timing. Maybe iCloud accounts should be placed in some sort of restricted state after a device is put into lost mode.

It’s stressful to face an ongoing, targeted, personal attack. Deep breath again. Time to methodically check every account for suspicious activity, change passwords just in case, and recover compromised accounts. My main Google account, protected by 2FA, was safe throughout the ordeal. I reset my Facebook password by email and got back in. GitHub, AWS, and other professional accounts were also on 2FA and had no unauthorized activity. Audit logs never tasted so sweet. It was a relief knowing I wouldn’t have to tell clients, “Hey, how are you? Great! So, listen, this iPhone thieving ring probably has all your data, isn’t that funny? Hah! But never mind that! Those Bitcoin prices, huh?”

Then I tried a secondary Gmail account I use for some mailing lists and other non-critical tasks. You know… the kind of account for which one might leave 2FA disabled. Sure enough, the wretch had been there, and the password was changed via SMS password reset. And he only found the account by the phone number in the first place. Familiar? Here’s a quick recap:

This Gmail account did not have a recovery email set up, and ironically I couldn’t use SMS anymore. Google offers a recovery algorithm where you try to answer different questions with the ability to “Try another way” if you don’t have a particular answer (quick: in which month and year did you create your Google account?). I was locked out for a while, long enough to start thinking I had lost the account, but eventually produced a couple of answers and got back in.

Finally, all of my accounts were safe again. It was getting late, but I had to find out why this person was frantically probing my accounts, and maybe, with some luck, who they were.

I knew the data in the iPhones was safe, as per the Apple vs. government showdown after the San Bernardino terrorist attack. But earlier I had assumed the phones could still be wiped clean and used normally. But maybe they couldn’t, and this whole rigmarole was about breaking into my iCloud account. Hence the phishing.

A quick search confirmed the idea. Since iOS 7, released in 2013, Apple has provided the Activation Lock feature, whereby if a device is linked to an iCloud account, activating it requires the password to that account. This has created some misery among people buying and selling used iPhones: if the seller forgets to unlink the device from their iCloud account, the device is bricked until they do so.

A warm wave of righteous schadenfreude washed over me: all the robbers had were parts! They would fetch little money from this whole thing, especially since my kid’s screen was cracked. You go, kid! Glad I hadn’t replaced it yet. Also glad for activation lock, though perversely my digital torment was its side effect: the world is complicated. It turns out there was no sinister plot, just a miserable scheme for a few hundred dollars. Straight to the depths of hell is where those cowards going.

It was sobering to realize the attacker almost succeeded. Up until a few months prior to the robbery, my iCloud account did not have 2FA enabled and it used the compromised Gmail address as the recovery email. If the robbery had happened then, they would have been able to get in, unlink the phones, and sell them at full (used) price. They would have changed my iCloud password in the process, and might have erased and locked my other Apple devices for the hell of it, which would have been disastrous and possibly ruinous, depending on whether I could get back into the account. Whatever little data I have in iCloud would have been stolen as well.

I hope this motivates you to enable 2FA on all of your accounts, even the unimportant ones. They can interact in incremental and unexpected ways to become your undoing. Moreover, using TOTP apps as the second factor is far safer than SMS.

Apple has done a fantastic job with iOS security and Find iPhone, curbing everything from malware to exploits to theft. But further improvements can be made to better protect its customers. In the next post you’ll see week-long sustained hacking attempts and meet the maggot behind the attacks, operating in a wretched hive of “iPhone unlockers.”

III. Recommendations

• Make sure your accounts cannot be hacked via text message (SMS) password resets. You can often do so by enabling two-factor authentication (2FA) for an account, particularly if you use a time-based one-time password (TOTP) app as the second factor. Two such apps are Google Authenticator and OTP Auth. You could also withhold your phone number from certain accounts. Another advantage of TOTP is that if you’re unable to receive SMS messages for whatever reason, you can still log in.

• Beware of your unprotected “less critical” accounts. They might provide a path to your sensitive ones.

• If you decide to go with a TOTP app, choose one that allows you to make an encrypted backup of your account secrets. OTP Auth provides that along with encrypted iCloud sync, all optional and controlled by the user. Authy is another good option. If you use Google Authenticator, make sure losing your phone won’t lock you out of any accounts.

• If you design apps, be careful with password resets via SMS. SIM cards are an easy target, cell providers are subject to social engineering that could lead to intercepted messages^[2], and SMS notifications can be seen on lock screens in most phones. Allow users to choose TOTP as a second factor.

• If your iPhone is lost or stolen, go to iCloud.com immediately, put it in lost mode, and provide a phone number where you can be reached. Once you’ve done that, you might want to temporarily suspend your phone line (many carriers offer this on their websites). If you do so, you can no longer call your own phone, and unless it’s on wifi, you also can’t “Play Sound” or “Erase iPhone” via iCloud - keep that in mind. On the upside, nobody can use your SIM card to hack your accounts via SMS password reset. It’s a trade-off.

• If your iPhone is definitely stolen, rather than lost, it will probably appear off in iCloud. Put it in lost mode anyway. If you provide a phone number, know that it might be targeted for iCloud phishing or social engineering as crooks try to hack into your iCloud account (that’s why the attacker briefly turned my phone on: to get a phone number to target). You almost surely want to suspend your cell service immediately. You lose the tracking and other goodies, but thieves generally know to keep the phone off, and handing them a working SIM card is fraught with peril. Tread carefully and may the force be with you.

• You might want to protect your SIM card with a PIN. This requires you to enter the PIN whenever you turn your phone on. Attackers are thus unable to transplant your SIM card to another device and use it. However, if you lose your iPhone and the battery dies, or the person who finds it turns it off, it’s game over. Even if the phone is later turned on, it won’t connect to the Internet, enter Lost Mode, show a phone number where you can be reached, or “Play Sound” (this is true even if a known wifi is in range^[3]). A phone that otherwise might have been found could be bricked and lost forever.

• Beware of “Your iPhone was found” emails, text messages, and Whatsapp messages. Scammers attempt to phish for your iCloud credentials in devious ways soon after an iPhone is stolen. If you provided a lost mode phone number, thieves will attempt to use it against you while trying to break into your iCloud account.

• Read Tech Solidarity’s security guide. It’s overkill for a regular user, but know the rules before breaking them.

Thank you for reading.

Many programmers are surprised to find out that the goto statement is still widely used in modern, high-quality codebases. Here are some examples, using the first codebases that come to mind:

The ratio to usages of the continue keyword is provided to normalize for lines of code and the prevalence of loops in the code. This is not limited to C code bases. Lucene.net for example has 1,511 goto usages and a ratio of 3 goto usages to each continue usage. The C# compiler, written itself in C#, clocks in at 297 goto usages and 0.22 ratio.

People who take “goto is evil” as dogma will point out that each of these usages could be rewritten as gotoless alternatives. And that’s true, of course. But it will come at a price: duplication of code, introduction of flags, several if statements, and overall added complexity. These are highly reviewed codebases written by talented people. When they use goto, it’s because they find it to be the simplest approach.

This is exactly how dogma hurts software development. We take a sensible rule that works most of the time and promote it to sacred edict, deeming violators as inferior programmers, producers of unclean code. Thus something that would have been a helpful guideline becomes a hard constraint. Pile up enough of these, and code that could have been simple ends up in a tangled mess, all in the name of “purity.”

We have a long tradition of dogmas, but goto is the seminal example, denounced in Edsger Dijkstra’s famous letter, Go To Statement Considered Harmful. Just barely over a page, it’s a good case study. The letter is good advice in the vast majority of cases: misuse of goto will quickly land you in a maze of twisty little passages, all alike. Less helpful were the creation of a social taboo (goto is the province of inferior programmers) and the absolutist calls for abolition. Dijkstra himself came to regret how “others were making a religion” out of his position, as quoted in Donald Knuth’s more level-headed paper, Structured Programming with go to Statements.

Taboos tend to accrete over time. For example, overzealous object-oriented design has produced a lot of lasagna code (too many layers) and a tendency towards overly complex designs. Chasing semantic markup purity, we sometimes resorted to hideous and even unreliable CSS hacks when much simpler solutions were available in HTML. Now, with microservices, people sometimes break up a trivial app into a hard-to-follow spiderweb of components. Again, these are cases of people taking a valuable guideline for an end in itself. Always keep a hard-nosed pragmatic aim at the real goals: simplicity, clarity, generality.

When Linus Torvalds started the Linux kernel in 1991, the dogma was that “monolithic” kernels were obsolete and that microkernels, a message-passing alternative analogous to microservices, were the only way to build a new OS. GNU had been working on microkernel designs since 1986. Torvalds, a pragmatist if there was ever one, tossed out this orthodoxy to build Linux using the much simpler monolithic design. Seems to have worked out.

Every programmer pays lip service to simplicity. But when push comes to shove, most will readily give up simplicity to satisfy dogma. We should be willing to break generic rules when the circumstances call for it. Keep it simple.

TLDR: I launched Grokbit, a code search and browsing tool.

When I was programming as a kid, I longed for a hardcore codebase, like a compiler or operating system, to really understand computers. That stuff just sounded so magical, some kind of Elvish secret way beyond mortals. There were decent books explaining how things worked, but that’s a poor substitute for code.

Then the Internet reached Brazil when I was about 14, and all of a sudden there was this “GNU C” compiler that was allegedly better than Microsoft’s, and the code was completely open! And what’s more, there was an open Unix you could run on your 386! No need to convince your parents to sell the car and buy a SPARCstation!

This was the best present any kid could hope for. This is why, despite a lot of issues in the tech community, my gratitude to these people is overwhelming. You might think Stallman is a lunatic, and you might be right, but damn - he’s the geek black-bearded Santa who brought the source to the children.

Now, reading this code was hard. The kernel, in particular, is tough to follow because entry points and flow of execution are unclear. There was no tmux, I didn’t know Vim, my regexes were weak. So I printed out a whole bunch of code in my parent’s dot matrix printer, and spread it on the floor. A poor man’s multi-pane Vim session. Here’s an interrupt handler, there’s a “bottom half”, and hey, look!, the syscall is right over there by the socks.

I think reading code is second only to writing code in making you a better programmer. When I write stuff like Anatomy of a program in memory or What does an idle CPU do?, a big part of the kick is sharing what I think are beautiful designs with people who haven’t seen them before.

Still, I always wished I could give readers a better interface to actually dive into the code. But our tools for handling an entire code base, especially in the browser, are just not good enough at the moment. Searching also needs a lot of improvement: I think we can do better than regexes and general full-text searching when it comes to searching code.

That’s why I built Grokbit. It has an indexing and search engine that’s entirely tailored to code, so you can search semantically, like “give me the definition of foo” or “search for an identifier named bar”. It’s also wicked fast: you get real-time suggestions even in the largest repos I could try.

But searching was only half the battle. Having a rich UI - especially a multi-pane one, was an absolute requirement for me. When you have function A calling B calling C, it can be enormously helpful to have the 3 of them in the screen at once, and navigate seamlessly. Plus being able to load multiple large files, having back/forward in the browser work well, and an overall smooth UX.

So that’s what I went for with Grokbit. It’s still crude, lots of low hanging fruit, but it has already been very useful, as you’ll be able to tell in future blog posts. But before I put more weekends into it, I’d like some feedback. Try it out, let me know what could be better, or which search features I should build sooner (many easy wins here). I hope it’s as fun to use as it was to build.

Finally, if you are interested in working on the project, reach out. I don’t know if this will become a SaaS app, or a feature in another product, or an open source project, but I am hellbent on solving this problem. Let’s carry the ring into Mordor.

For a number of years I’ve configured my desktops so that most tasks can be done using only home row keys on the keyboard, a technique I call home row computing. It takes the Vi idea of staying on the home row to every app, all the time, but without using modes so things are simpler.

I’ve described an implementation for Windows, but I have since moved to Macs and back to a qwerty keyboard (away from Dvorak). The current setup is described in this post. It uses familiar Vi key bindings and is far more suitable. It’s fairly painless to configure on the Mac and has never given me any problems, thanks to Takayama Fumihiko’s awesome keyboard apps.

Using this is a joy. It’s really fast, easy on the hands, and makes you feel like a geek god. If you don’t use Vim, you’ll now have one of its benefits in your favorite editor and in other apps, plus a weapon against smug Vimmers. If you already use Vim, your cherished hjkl keys become universal and pressing Esc gets a hell of a lot easier.

Some of the important keys that must be moved to home row are the arrow keys, Esc, delete (backspace) and forward delete. Another helpful home row task is moving and resizing windows. The key to all this is remapping Caps Lock to allow combinations of Caps Lock plus a home key to do these tasks. Again, there are no modes involved here, Caps Lock works as a modifier like the cmd and fn keys. Here’s a good start:

I have left several keys unmapped so you can customize your own setup, and we’ll get to window management in a moment. The first step is to set Caps Lock to No Action in System Preferences > Keyboard > Modifier keys:

Now we must remap the Caps Lock key code to something else. To do so, you need a small tool called Seil (open source). You can map Caps Lock to any other key, like cmd or option. So if you don’t want to go all-out home row, you can still benefit from the remapping.

I like to remap Caps Lock into something that guarantees no conflicts ever for our combos. So I use key code 110, which is the Apps key on a Windows keyboard and is safely absent from Apple keyboards:

Now we’re in business, the world - or at least the keyboard - is our oyster. The maker of Seil also makes Karabiner, open as well and an outstanding keyboard customizer for OS X. I have no affiliation with these tools, apart from being a happy user for years. If you end up using them, please donate. So go ahead and install Karabiner, and you’ll see a plethora of keyboard tweak possibilities:

Each of the tweaks can be toggled on and off. There are even native Vi, Vim, and Emacs modes. However, I don’t like the built-in ones, so I built my own config. Go to Misc & Uninstall and click Open private.xml:

In this file, ~/Library/Application Support/Karabiner/private.xml, you can define your own keyboard remapping scheme. I actually symlink that to a Dropbox file to keep the configuration consistent across my machines, but at any rate, here is a file you can use to implement what we have discussed so far. Drop the file in, click ReloadXML and you’ll have this:

Home Row Computing is at the top (prefixed with ! for sorting). Toggle it on, and you’re done. Enjoy your new keyboard layout, do a search on Spotlight and see how fast and smooth it is to choose an option.

Finally, there is window management. That’s an area where you can fumble quite a bit, resizing and moving about clumsily with a mouse. My favorite options to make it fast and homerow-friendly are ShiftIt (open) and Moom (best $10 I ever spent, no affiliation). There are some others, but to me Moom towers above the rest. It has a great two-step usage, where one hot key activates it:

And the following key triggers a command you get to define using window primitives like move, zoom, resize, and change monitors. You can also define shortcuts that run commands directly. Moom has some handy default actions:

Out of box, arrow keys can be used to send a window to the left, right, top, or bottom of the screen, and Moom natively interprets hjkl as arrows making it easy to stay on home row. You can associate keys with various commands and precise window positions:

This is gold for large monitors like Apple Thunderbolts. I remap Caps Lock + M into the global Moom shortcut for painless activation. This allows me to set the shortcut itself to something bizarre that won’t conflict with anything but would be a dog to type. Currently it’s an improbable Fn + Control + Command + M. I also have Caps Lock + N activating a Moom command that cycles a window between my two monitors. Both of these shortcuts are in the keyboard map I provided.

If you have any questions, let me know. I know a number of keyboard nuts out there use this scheme on Windows and Linux, and I hope this makes it easy to do so on Macs.

I hate to break it to you, but a user application is a helpless brain in a vat:

Every interaction with the outside world is mediated by the kernel through system calls. If an app saves a file, writes to the terminal, or opens a TCP connection, the kernel is involved. Apps are regarded as highly suspicious: at best a bug-ridden mess, at worst the malicious brain of an evil genius.

These system calls are function calls from an app into the kernel. They use a specific mechanism for safety reasons, but really you’re just calling the kernel’s API. The term “system call” can refer to a specific function offered by the kernel (e.g., the open() system call) or to the calling mechanism. You can also say syscall for short.

This post looks at system calls, how they differ from calls to a library, and tools to poke at this OS/app interface. A solid understanding of what happens within an app versus what happens through the OS can turn an impossible-to-fix problem into a quick, fun puzzle.

So here’s a running program, a user process:

It has a private virtual address space, its very own memory sandbox. The vat, if you will. In its address space, the program’s binary file plus the libraries it uses are all memory mapped. Part of the address space maps the kernel itself.

Below is the code for our program, pid, which simply retrieves its process id via getpid(2):

1
2
3
4
5
6
7
8
9

#include 
#include 
#include 

int main()
{
    pid_t p = getpid();
    printf("%d\n", p);
}

In Linux, a process isn’t born knowing its PID. It must ask the kernel, so this requires a system call:

It all starts with a call to the C library’s getpid(), which is a wrapper for the system call. When you call functions like open(2), read(2), and friends, you’re calling these wrappers. This is true for many languages where the native methods ultimately end up in libc.

Wrappers offer convenience atop the bare-bones OS API, helping keep the kernel lean. Lines of code is where bugs live, and all kernel code runs in privileged mode, where mistakes can be disastrous. Anything that can be done in user mode should be done in user mode. Let the libraries offer friendly methods and fancy argument processing a la printf(3).

Compared to web APIs, this is analogous to building the simplest possible HTTP interface to a service and then offering language-specific libraries with helper methods. Or maybe some caching, which is what libc’s getpid() does: when first called it actually performs a system call, but the PID is then cached to avoid the syscall overhead in subsequent invocations.

Once the wrapper has done its initial work it’s time to jump into hyperspace the kernel. The mechanics of this transition vary by processor architecture. In Intel processors, arguments and the syscall number are loaded into registers, then an instruction is executed to put the CPU in privileged mode and immediately transfer control to a global syscall entry point within the kernel. If you’re interested in details, David Drysdale has two great articles in LWN (first, second).

The kernel then uses the syscall number as an index into sys_call_table, an array of function pointers to each syscall implementation. Here, sys_getpid is called:

In Linux, syscall implementations are mostly arch-independent C functions, sometimes trivial, insulated from the syscall mechanism by the kernel’s excellent design. They are regular code working on general data structures. Well, apart from being completely paranoid about argument validation.

Once their work is done they return normally, and the arch-specific code takes care of transitioning back into user mode where the wrapper does some post processing. In our example, getpid(2) now caches the PID returned by the kernel. Other wrappers might set the global errno variable if the kernel returns an error. Small things to let you know GNU cares.

If you want to be raw, glibc offers the syscall(2) function, which makes a system call without a wrapper. You can also do so yourself in assembly. There’s nothing magical or privileged about a C library.

This syscall design has far-reaching consequences. Let’s start with the incredibly useful strace(1), a tool you can use to spy on system calls made by Linux processes (in Macs, see dtruss(1m) and the amazing dtrace; in Windows, see sysinternals). Here’s strace on pid:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

~/code/x86-os$ strace ./pid

execve("./pid", ["./pid"], [/* 20 vars */]) = 0
brk(0)                                  = 0x9aa0000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7767000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18056, ...}) = 0
mmap2(NULL, 18056, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7762000
close(3)                                = 0

[...snip...]

getpid()                                = 14678
fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7766000
write(1, "14678\n", 614678
)                  = 6
exit_group(6)                           = ?

Each line of output shows a system call, its arguments, and a return value. If you put getpid(2) in a loop running 1000 times, you would still have only one getpid() syscall because of the PID caching. We can also see that printf(3) calls write(2) after formatting the output string.

strace can start a new process and also attach to an already running one. You can learn a lot by looking at the syscalls made by different programs. For example, what does the sshd daemon do all day?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

~/code/x86-os$ ps ax | grep sshd
12218 ?        Ss     0:00 /usr/sbin/sshd -D

~/code/x86-os$ sudo strace -p 12218
Process 12218 attached - interrupt to quit
select(7, [3 4], NULL, NULL, NULL

[
  ... nothing happens ...
  No fun, it's just waiting for a connection using select(2)
  If we wait long enough, we might see new keys being generated and so on, but
  let's attach again, tell strace to follow forks (-f), and connect via SSH
]

~/code/x86-os$ sudo strace -p 12218 -f

[lots of calls happen during an SSH login, only a few shown]

[pid 14692] read(3, "-----BEGIN RSA PRIVATE KEY-----\n"..., 1024) = 1024
[pid 14692] open("/usr/share/ssh/blacklist.RSA-2048", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 14692] open("/etc/ssh/blacklist.RSA-2048", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid 14692] open("/etc/ssh/ssh_host_dsa_key", O_RDONLY|O_LARGEFILE) = 3
[pid 14692] open("/etc/protocols", O_RDONLY|O_CLOEXEC) = 4
[pid 14692] read(4, "# Internet (IP) protocols\n#\n# Up"..., 4096) = 2933
[pid 14692] open("/etc/hosts.allow", O_RDONLY) = 4
[pid 14692] open("/lib/i386-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 4
[pid 14692] stat64("/etc/pam.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 14692] open("/etc/pam.d/common-password", O_RDONLY|O_LARGEFILE) = 8
[pid 14692] open("/etc/pam.d/other", O_RDONLY|O_LARGEFILE) = 4

SSH is a large chunk to bite off, but it gives a feel for strace usage. Being able to see which files an app opens can be useful (“where the hell is this config coming from?”). If you have a process that appears stuck, you can strace it and see what it might be doing via system calls. When some app is quitting unexpectedly without a proper error message, check if a syscall failure explains it. You can also use filters, time each call, and so so:

1
2
3
4
5
6
7
8
9

~/code/x86-os$ strace -T -e trace=recv curl -silent www.google.com. > /dev/null

recv(3, "HTTP/1.1 200 OK\r\nDate: Wed, 05 N"..., 16384, 0) = 4164 <0.000007>
recv(3, "fl a{color:#36c}a:visited{color:"..., 16384, 0) = 2776 <0.000005>
recv(3, "adient(top,#4d90fe,#4787ed);filt"..., 16384, 0) = 4164 <0.000007>
recv(3, "gbar.up.spd(b,d,1,!0);break;case"..., 16384, 0) = 2776 <0.000006>
recv(3, "$),a.i.G(!0)),window.gbar.up.sl("..., 16384, 0) = 1388 <0.000004>
recv(3, "margin:0;padding:5px 8px 0 6px;v"..., 16384, 0) = 1388 <0.000007>
recv(3, "){window.setTimeout(function(){v"..., 16384, 0) = 1484 <0.000006>

I encourage you to explore these tools in your OS. Using them well is like having a super power.

But enough useful stuff, let’s go back to design. We’ve seen that a userland app is trapped in its virtual address space running in ring 3 (unprivileged). In general, tasks that involve only computation and memory accesses do not require syscalls. For example, C library functions like strlen(3) and memcpy(3) have nothing to do with the kernel. Those happen within the app.

The man page sections for a C library function (the 2 and 3 in parenthesis) also offer clues. Section 2 is used for system call wrappers, while section 3 contains other C library functions. However, as we saw with printf(3), a library function might ultimately make one or more syscalls.

If you’re curious, here are full syscall listings for Linux (also Filippo’s list) and Windows. They have ~310 and ~460 system calls, respectively. It’s fun to look at those because, in a way, they represent all that software can do on a modern computer. Plus, you might find gems to help with things like interprocess communication and performance. This is an area where “Those who do not understand Unix are condemned to reinvent it, poorly.”

Many syscalls perform tasks that take eons compared to CPU cycles, for example reading from a hard drive. In those situations the calling process is often put to sleep until the underlying work is completed. Because CPUs are so fast, your average program is I/O bound and spends most of its life sleeping, waiting on syscalls. By contrast, if you strace a program busy with a computational task, you often see no syscalls being invoked. In such a case, top(1) would show intense CPU usage.

The overhead involved in a system call can be a problem. For example, SSDs are so fast that general OS overhead can be more expensive than the I/O operation itself. Programs doing large numbers of reads and writes can also have OS overhead as their bottleneck. Vectored I/O can help some. So can memory mapped files, which allow a program to read and write from disk using only memory access. Analogous mappings exist for things like video card memory. Eventually, the economics of cloud computing might lead us to kernels that eliminate or minimize user/kernel mode switches.

Finally, syscalls have interesting security implications. One is that no matter how obfuscated a binary, you can still examine its behavior by looking at the system calls it makes. This can be used to detect malware, for example. We can also record profiles of a known program’s syscall usage and alert on deviations, or perhaps whitelist specific syscalls for programs so that exploiting vulnerabilities becomes harder. We have a ton of research in this area, a number of tools, but not a killer solution yet.

And that’s it for system calls. I’m sorry for the length of this post, I hope it was helpful. More (and shorter) next week, RSS and Twitter. Also, last night I made a promise to the universe. This post is dedicated to the glorious Clube Atlético Mineiro.

In the last post I said the fundamental axiom of OS behavior is that at any given time, exactly one and only one task is active on a CPU. But if there’s absolutely nothing to do, then what?

It turns out that this situation is extremely common, and for most personal computers it’s actually the norm: an ocean of sleeping processes, all waiting on some condition to wake up, while nearly 100% of CPU time is going into the mythical “idle task.” In fact, if the CPU is consistently busy for a normal user, it’s often a misconfiguration, bug, or malware.

Since we can’t violate our axiom, some task needs to be active on a CPU. First because it’s good design: it would be unwise to spread special cases all over the kernel checking whether there is in fact an active task. A design is far better when there are no exceptions. Whenever you write an if statement, Nyan Cat cries. And second, we need to do something with all those idle CPUs, lest they get spunky and, you know, create Skynet.

So to keep design consistency and be one step ahead of the devil, OS developers create an idle task that gets scheduled to run when there’s no other work. We have seen in the Linux boot process that the idle task is process 0, a direct descendent of the very first instruction that runs when a computer is first turned on. It is initialized in rest_init, where init_idle_bootup_task initializes the idle scheduling class.

Briefly, Linux supports different scheduling classes for things like real-time processes, regular user processes, and so on. When it’s time to choose a process to become the active task, these classes are queried in order of priority. That way, the nuclear reactor control code always gets to run before the web browser. Often, though, these classes return NULL, meaning they don’t have a suitable process to run - they’re all sleeping. But the idle scheduling class, which runs last, never fails: it always returns the idle task.

That’s all good, but let’s get down to just what exactly this idle task is doing. So here is cpu_idle_loop, courtesy of open source:

1
2
3
4
5
6
7
8
9
10
11

while (1) {
    while(!need_resched()) {
        cpuidle_idle_call();
    }

    /*
      [Note: Switch to a different task. We will return to this loop when the
      idle task is again selected to run.]
    */
    schedule_preempt_disabled();
}

I’ve omitted many details, and we’ll look at task switching closely later on, but if you read the code you’ll get the gist of it: as long as there’s no need to reschedule, meaning change the active task, stay idle. Measured in elapsed time, this loop and its cousins in other OSes are probably the most executed pieces of code in computing history. For Intel processors, staying idle traditionally meant running the halt instruction:

1
2
3
4

static inline void native_halt(void)
{
    asm volatile("hlt": : :"memory");
}

hlt stops code execution in the processor and puts it in a halted state. It’s weird to think that across the world millions and millions of Intel-like CPUs are spending the majority of their time halted, even while they’re powered up. It’s also not terribly efficient, energy wise, which led chip makers to develop deeper sleep states for processors, which trade off less power consumption for longer wake-up latency. The kernel’s cpuidle subsystem is responsible for taking advantage of these power-saving modes.

Now once we tell the CPU to halt, or sleep, we need to somehow bring it back to life. If you’ve read the last post, you might suspect interrupts are involved, and indeed they are. Interrupts spur the CPU out of its halted state and back into action. So putting this all together, here’s what your system mostly does as you read a fully rendered web page:

Other interrupts besides the timer interrupt also get the processor moving again. That’s what happens if you click on a web page, for example: your mouse issues an interrupt, its driver processes it, and suddenly a process is runnable because it has fresh input. At that point need_resched() returns true, and the idle task is booted out in favor of your browser.

But let’s stick to idleness in this post. Here’s the idle loop over time:

In this example the timer interrupt was programmed by the kernel to happen every 4 milliseconds (ms). This is the tick period. That means we get 250 ticks per second, so the tick rate or tick frequency is 250 Hz. That’s a typical value for Linux running on Intel processors, with 100 Hz being another crowd favorite. This is defined in the CONFIG_HZ option when you build the kernel.

Now that looks like an awful lot of pointless work for an idle CPU, and it is. Without fresh input from the outside world, the CPU will remain stuck in this hellish nap getting woken up 250 times a second while your laptop battery is drained. If this is running in a virtual machine, we’re burning both power and valuable cycles from the host CPU.

The solution here is to have a dynamic tick so that when the CPU is idle, the timer interrupt is either deactivated or reprogrammed to happen at a point where the kernel knows there will be work to do (for example, a process might have a timer expiring in 5 seconds, so we must not sleep past that). This is also called tickless mode.

Finally, suppose you have one active process in a system, for example a long-running CPU-intensive task. That’s nearly identical to an idle system: these diagrams remain about the same, just substitute the one process for the idle task and the pictures are accurate. In that case it’s still pointless to interrupt the task every 4 ms for no good reason: it’s merely OS jitter slowing your work ever so slightly. Linux can also stop the fixed-rate tick in this one-process scenario, in what’s called adaptive-tick mode. Eventually, a fixed-rate tick may be gone altogether.

That’s enough idleness for one post. The kernel’s idle behavior is an important part of the OS puzzle, and it’s very similar to other situations we’ll see, so this helps us build the picture of a running kernel. More next week, RSS and Twitter.

Here’s a question: in the time it takes you to read this sentence, has your OS been running? Or was it only your browser? Or were they perhaps both idle, just waiting for you to do something already?

These questions are simple but they cut through the essence of how software works. To answer them accurately we need a good mental model of OS behavior, which in turn informs performance, security, and troubleshooting decisions. We’ll build such a model in this post series using Linux as the primary OS, with guest appearances by OS X and Windows. I’ll link to the Linux kernel sources for those who want to delve deeper.

The fundamental axiom here is that at any given moment, exactly one task is active on a CPU. The task is normally a program, like your browser or music player, or it could be an operating system thread, but it is one task. Not two or more. Never zero, either. One. Always.

This sounds like trouble. For what if, say, your music player hogs the CPU and doesn’t let any other tasks run? You would not be able to open a tool to kill it, and even mouse clicks would be futile as the OS wouldn’t process them. You could be stuck blaring “What does the fox say?” and incite a workplace riot.

That’s where interrupts come in. Much as the nervous system interrupts the brain to bring in external stimuli - a loud noise, a touch on the shoulder - the chipset in a computer’s motherboard interrupts the CPU to deliver news of outside events - key presses, the arrival of network packets, the completion of a hard drive read, and so on. Hardware peripherals, the interrupt controller on the motherboard, and the CPU itself all work together to implement these interruptions, called interrupts for short.

Interrupts are also essential in tracking that which we hold dearest: time. During the boot process the kernel programs a hardware timer to issue timer interrupts at a periodic interval, for example every 10 milliseconds. When the timer goes off, the kernel gets a shot at the CPU to update system statistics and take stock of things: has the current program been running for too long? Has a TCP timeout expired? Interrupts give the kernel a chance to both ponder these questions and take appropriate actions. It’s as if you set periodic alarms throughout the day and used them as checkpoints: should I be doing what I’m doing right now? Is there anything more pressing? One day you find ten years have got behind you.

These periodic hijackings of the CPU by the kernel are called ticks, so interrupts quite literally make your OS tick. But there’s more: interrupts are also used to handle some software events like integer overflows and page faults, which involve no external hardware. Interrupts are the most frequent and crucial entry point into the OS kernel. They’re not some oddity for the EE people to worry about, they’re the mechanism whereby your OS runs.

Enough talk, let’s see some action. Below is a network card interrupt in an Intel Core i5 system. The diagrams now have image maps, so you can click on juicy bits for more information. For example, each device links to its Linux driver.

Let’s take a look at this. First off, since there are many sources of interrupts, it wouldn’t be very helpful if the hardware simply told the CPU “hey, something happened!” and left it at that. The suspense would be unbearable. So each device is assigned an interrupt request line, or IRQ, during power up. These IRQs are in turn mapped into interrupt vectors, a number between 0 and 255, by the interrupt controller. By the time an interrupt reaches the CPU it has a nice, well-defined number insulated from the vagaries of hardware.

The CPU in turn has a pointer to what’s essentially an array of 255 functions, supplied by the kernel, where each function is the handler for that particular interrupt vector. We’ll look at this array, the Interrupt Descriptor Table (IDT), in more detail later on.

Whenever an interrupt arrives, the CPU uses its vector as an index into the IDT and runs the appropriate handler. This happens as a special function call that takes place in the context of the currently running task, allowing the OS to respond to external events quickly and with minimal overhead. So web servers out there indirectly call a function in your CPU when they send you data, which is either pretty cool or terrifying. Below we show a situation where a CPU is busy running a Vim command when an interrupt arrives:

Notice how the interrupt’s arrival causes a switch to kernel mode and ring zero but it does not change the active task. It’s as if Vim made a magic function call straight into the kernel, but Vim is still there, its address space intact, waiting for that call to return.

Exciting stuff! Alas, I need to keep this post-sized, so let’s finish up for now. I understand we have not answered the opening question and have in fact opened up new questions, but you now suspect ticks were taking place while you read that sentence. We’ll find the answers as we flesh out our model of dynamic OS behavior, and the browser scenario will become clear. If you have questions, especially as the posts come out, fire away and I’ll try to answer them in the posts themselves or as comments. Next installment is tomorrow on RSS and Twitter.

The last post in this series looks at closures, objects, and other creatures roaming beyond the stack. Much of what we’ll see is language neutral, but I’ll focus on JavaScript with a dash of C. Let’s start with a simple C program that reads a song and a band name and outputs them back to the user:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#include 
#include 

char *read()
{
    char data[64];
    fgets(data, 64, stdin);
    return data;
}

int main(int argc, char *argv[])
{
    char *song, *band;

    puts("Enter song, then band:");
    song = read();
    band = read();

    printf("\n%sby %s", song, band);

    return 0;
}

If you run this gem, here’s what you get (=> denotes program output):

1
2
3
4
5
6
7

./stackFolly
=> Enter song, then band:
The Past is a Grotesque Animal
of Montreal

=> ?ǿontreal
=> by ?ǿontreal

Ayeee! Where did things go so wrong? (Said every C beginner, ever.)

It turns out that the contents of a function’s stack variables are only valid while the stack frame is active, that is, until the function returns. Upon return, the memory used by the stack frame is deemed free and liable to be overwritten in the next function call.

Below is exactly what happens in this case. The diagrams now have image maps, so you can click on a piece of data to see the relevant gdb output (gdb commands are here). As soon as read() is done with the song name, the stack is thus:

At this point, the song variable actually points to the song name. Sadly, the memory storing that string is ready to be reused by the stack frame of whatever function is called next. In this case, read() is called again, with the same stack frame layout, so the result is this:

The band name is read into the same memory location and overwrites the previously stored song name. band and song end up pointing to the exact same spot. Finally, we didn’t even get “of Montreal” output correctly. Can you guess why?

And so it happens that the stack, for all its usefulness, has this serious limitation. It cannot be used by a function to store data that needs to outlive the function’s execution. You must resort to the heap and say goodbye to the hot caches, deterministic instantaneous operations, and easily computed offsets. On the plus side, it works:

The price is you must now remember to free() memory or take a performance hit on a garbage collector, which finds unused heap objects and frees them. That’s the fundamental tradeoff between stack and heap: performance vs. flexibility.

Most languages’ virtual machines take a middle road that mirrors what C programmers do. The stack is used for value types, things like integers, floats and booleans. These are stored directly in local variables and object fields as a sequence of bytes specifying a value (like argc above). In contrast, heap inhabitants are reference types such as strings and objects. Variables and fields contain a memory address that references these objects, like song and band above.

Consider this JavaScript function:

1
2
3
4
5

function fn()
{
    var a = 10;
    var b = { name: 'foo', n: 10 };
}

This might produce the following:

I say “might” because specific behaviors depend heavily on implementation. This post takes a V8-centric approach with many diagram shapes linking to relevant source code. In V8, only small integers are stored as values. Also, from now on I’ll show strings directly in objects to reduce visual noise, but keep in mind they exist separately in the heap, as shown above.

Now let’s take a look at closures, which are simple but get weirdly hyped up and mythologized. Take a trivial JS function:

1
2
3
4
5

function add(a, b)
{
        var c = a + b;
        return c;
}

This function defines a lexical scope, a happy little kingdom where the names a, b, and c have precise meanings. They are the two parameters and one local variable declared by the function. The program might use those same names elsewhere, but within add that’s what they refer to. And while lexical scope is a fancy term, it aligns well with our intuitive understanding: after all, we can quite literally see the bloody thing, much as a lexer does, as a textual block in the program’s source.

Having seen stack frames in action, it’s easy to imagine an implementation for this name specificity. Within add, these names refer to stack locations private to each running instance of the function. That’s in fact how it often plays out in a VM.

So let’s nest two lexical scopes:

1
2
3
4
5
6
7
8
9

function makeGreeter()
{
    return function hi(name) {
        console.log('hi, ' + name);
    }
}

var hi = makeGreeter();
hi('dear reader'); // prints "hi, dear reader"

That’s more interesting. Function hi is built at runtime within makeGreeter. It has its own lexical scope, where name is an argument on the stack, but visually it sure looks like it can access its parent’s lexical scope as well, which it can. Let’s take advantage of that:

1
2
3
4
5
6
7
8
9

function makeGreeter(greeting)
{
    return function greet(name) {
        console.log(greeting + ', ' + name);
    }
}

var heya = makeGreeter('HEYA');
heya('dear reader'); // prints "HEYA, dear reader"

A little strange, but pretty cool. There’s something about it though that violates our intuition: greeting sure looks like a stack variable, the kind that should be dead after makeGreeter() returns. And yet, since greet() keeps working, something funny is going on. Enter the closure:

The VM allocated an object to store the parent variable used by the inner greet(). It’s as if makeGreeter's lexical scope had been closed over at that moment, crystallized into a heap object for as long as needed (in this case, the lifetime of the returned function). Hence the name closure, which makes a lot of sense when you see it that way. If more parent variables had been used (or captured), the Context object would have more properties, one per captured variable. Naturally, the code emitted for greet() knows to read greeting from the Context object, rather than expect it on the stack.

Here’s a fuller example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

function makeGreeter(greetings)
{
    var count = 0;
    var greeter = {};

    for (var i = 0; i < greetings.length; i++) {
        var greeting = greetings[i];

        greeter[greeting] = function(name) {
            count++;
            console.log(greeting + ', ' + name);
        }
    }

    greeter.count = function() { return count; }

    return greeter;
}

var greeter = makeGreeter(["hi", "hello", "howdy"])
greeter.hi('poppet'); // prints "howdy, poppet"
greeter.hello('darling'); // prints "howdy, darling"
greeter.count(); // returns 2

Well… count() works, but our greeter is stuck in howdy. Can you tell why? What we’re doing with count is a clue: even though the lexical scope is closed over into a heap object, the values taken by the variables (or object properties) can still be changed. Here’s what we have:

There is one common context shared by all functions. That’s why count works. But the greeting is also being shared, and it was set to the last value iterated over, “howdy” in this case. That’s a pretty common error, and the easiest way to avoid it is to introduce a function call to take the closed-over variable as an argument. In CoffeeScript, the do command provides an easy way to do so. Here’s a simple solution for our greeter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

function makeGreeter(greetings)
{
    var count = 0;
    var greeter = {};

    greetings.forEach(function(greeting) {
        greeter[greeting] = function(name) {
            count++;
            console.log(greeting + ', ' + name);
        }
    });

    greeter.count = function() { return count; }

    return greeter;
}

var greeter = makeGreeter(["hi", "hello", "howdy"])
greeter.hi('poppet'); // prints "hi, poppet"
greeter.hello('darling'); // prints "hello, darling"
greeter.count(); // returns 2

It now works, and the result becomes:

That’s a lot of arrows! But here’s the interesting feature: in our code, we closed over two nested lexical contexts, and sure enough we get two linked Context objects in the heap. You could nest and close over many lexical contexts, Russian-doll style, and you end up with essentially a linked list of all these Context objects.

Of course, just as you can implement TCP over carrier pigeons, there are many ways to implement these language features. For example, the ES6 spec defines lexical environments as consisting of an environment record (roughly, the local identifiers within a block) plus a link to an outer environment record, allowing the nesting we have seen. The logical rules are nailed by the spec (one hopes), but it’s up to the implementation to translate them into bits and bytes.

You can also inspect the assembly code produced by V8 for specific cases. Vyacheslav Egorov has great posts and explains this process along with V8 closure internals in detail. I’ve only started studying V8, so pointers and corrections are welcome. If you know C#, inspecting the IL code emitted for closures is enlightening - you will see the analog of V8 Contexts explicitly defined and instantiated.

Closures are powerful beasts. They provide a succinct way to hide information from a caller while sharing it among a set of functions. I love that they truly hide your data: unlike object fields, callers cannot access or even see closed-over variables. Keeps the interface cleaner and safer.

But they’re no silver bullet. Sometimes an object nut and a closure fanatic will argue endlessly about their relative merits. Like most tech discussions, it’s often more about ego than real tradeoffs. At any rate, this epic koan by Anton van Straaten settles the issue:

The venerable master Qc Na was walking with his student, Anton. Hoping to prompt the master into a discussion, Anton said “Master, I have heard that objects are a very good thing - is this true?” Qc Na looked pityingly at his student and replied, “Foolish pupil - objects are merely a poor man’s closures.”

Chastised, Anton took his leave from his master and returned to his cell, intent on studying closures. He carefully read the entire “Lambda: The Ultimate…” series of papers and its cousins, and implemented a small Scheme interpreter with a closure-based object system. He learned much, and looked forward to informing his master of his progress.

On his next walk with Qc Na, Anton attempted to impress his master by saying “Master, I have diligently studied the matter, and now understand that objects are truly a poor man’s closures.” Qc Na responded by hitting Anton with his stick, saying “When will you learn? Closures are a poor man’s object.” At that moment, Anton became enlightened.

And that closes our stack series. In the future I plan to cover other language implementation topics like object binding and vtables. But the call of the kernel is strong, so there’s an OS post coming out tomorrow. I invite you to subscribe and follow me.

In this penultimate post about the stack, we take a quick look at tail calls, compiler optimizations, and the proper tail calls landing in the newest version of JavaScript.

A tail call happens when a function F makes a function call as its final action. At that point F will do absolutely no more work: it passes the ball to whatever function is being called and vanishes from the game. This is notable because it opens up the possibility of tail call optimization: instead of creating a new stack frame for the function call, we can simply reuse F's stack frame, thereby saving stack space and avoiding the work involved in setting up a new frame. Here are some examples in C and their results compiled with mild optimization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

int add5(int a)
{
        return a + 5;
}

int add10(int a)
{
        int b = add5(a); // not tail
        return add5(b); // tail
}

int add5AndTriple(int a) {
        int b = add5(a); // not tail
        return 3 * add5(a); // not tail, doing work after the call
}

int finicky(int a) {
        if (a > 10) {
                return add5AndTriple(a); // tail
        }

        if (a > 5) {
                int b = add5(a); // not tail
                return finicky(b); // tail
        }

        return add10(a); // tail
}

You can normally spot tail call optimization (hereafter, TCO) in compiler output by seeing a jump instruction where a call would have been expected. At runtime TCO leads to a reduced call stack.

A common misconception is that tail calls are necessarily recursive. That’s not the case: a tail call may be recursive, such as in finicky() above, but it need not be. As long as caller F is completely done at the call site, we’ve got ourselves a tail call. Whether it can be optimized is a different question whose answer depends on your programming environment.

“Yes, it can, always!” is the best answer we can hope for, which is famously the case for Scheme, as discussed in SICP (by the way, if when you program you don’t feel like “a Sorcerer conjuring the spirits of the computer with your spells,” I urge you to read that book). It’s also the case for Lua. And most importantly, it is the case for the next version of JavaScript, ES6, whose spec does a good job defining tail position and clarifying the few conditions required for optimization, such as strict mode. When a language guarantees TCO, it supports proper tail calls.

Now some of us can’t kick that C habit, heart bleed and all, and the answer there is a more complicated “sometimes” that takes us into compiler optimization territory. We’ve seen the simple examples above; now let’s resurrect our factorial from last post:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#include 

int factorial(int n)
{
	int previous = 0xdeadbeef;	

	if (n == 0 || n == 1) {
		return 1;
	}

	previous = factorial(n-1);
	return n * previous;
}

int main(int argc)
{
	int answer = factorial(5);
	printf("%d\n", answer);
}

So, is line 11 a tail call? It’s not, because of the multiplication by n afterwards. But if you’re not used to optimizations, gcc’s result with O2 optimization might shock you: not only it transforms factorial into a recursion-free loop, but the factorial(5) call is eliminated entirely and replaced by a compile-time constant of 120 (5! == 120). This is why debugging optimized code can be hard sometimes. On the plus side, if you call this function it will use a single stack frame regardless of n’s initial value. Compiler algorithms are pretty fun, and if you’re interested I suggest you check out Building an Optimizing Compiler and ACDI.

However, what happened here was not tail call optimization, since there was no tail call to begin with. gcc outsmarted us by analyzing what the function does and optimizing away the needless recursion. The task was made easier by the simple, deterministic nature of the operations being done. By adding a dash of chaos (e.g., getpid()) we can throw gcc off:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#include 
#include 
#include 

int pidFactorial(int n)
{
        if (1 == n) {
                return getpid(); // tail
        }

        return n * pidFactorial(n-1) * getpid(); // not tail
}

int main(int argc)
{
        int answer = pidFactorial(5);
        printf("%d\n", answer);
}

Optimize that, unix fairies! So now we have a regular recursive call and this function allocates O(n) stack frames to do its work. Heroically, gcc still does TCO for getpid in the recursion base case. If we now wished to make this function tail recursive, we’d need a slight change:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#include 
#include 
#include 

int tailPidFactorial(int n, int acc)
{
        if (1 == n) {
                return acc * getpid(); // not tail
        }

        acc = (acc * getpid() * n);
        return tailPidFactorial(n-1, acc); // tail
}

int main(int argc)
{
        int answer = tailPidFactorial(5, 1);
        printf("%d\n", answer);
}

The accumulation of the result is now a loop and we’ve achieved true TCO. But before you go out partying, what can we say about the general case in C? Sadly, while good C compilers do TCO in a number of cases, there are many situations where they cannot do it. For example, as we saw in our function epilogues, the caller is responsible for cleaning up the stack after a function call using the standard C calling convention. So if function F takes two arguments, it can only make TCO calls to functions taking two or fewer arguments. This is one among many restrictions. Mark Probst wrote an excellent thesis discussing Proper Tail Recursion in C where he discusses these issues along with C stack behavior. He also does insanely cool juggling.

“Sometimes” is a rocky foundation for any relationship, so you can’t rely on TCO in C. It’s a discrete optimization that may or may not take place, rather than a language feature like proper tail calls, though in practice the compiler will optimize the vast majority of cases. But if you must have it, say for transpiling Scheme into C, you will suffer.

Since JavaScript is now the most popular transpilation target, proper tail calls become even more important there. So kudos to ES6 for delivering it along with many other significant improvements. It’s like Christmas for JS programmers.

This concludes our brief tour of tail calls and compiler optimization. Thanks for reading and see you next time.

Recursion is magic, but it suffers from the most awkward introduction in programming books. They’ll show you a recursive factorial implementation, then warn you that while it sort of works it’s terribly slow and might crash due to stack overflows. “You could always dry your hair by sticking your head into the microwave, but watch out for intracranial pressure and head explosions. Or you can use a towel.” No wonder people are suspicious of it. Which is too bad, because recursion is the single most powerful idea in algorithms.

Let’s take a look at the classic recursive factorial:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#include 

int factorial(int n)
{
        int previous = 0xdeadbeef;

        if (n == 0 || n == 1) {
                return 1;
        }

        previous = factorial(n-1);
        return n * previous;
}

int main(int argc)
{
        int answer = factorial(5);
        printf("%d\n", answer);
}

The idea of a function calling itself is mystifying at first. To make it concrete, here is exactly what is on the stack when factorial(5) is called and reaches n == 1:

Each call to factorial generates a new stack frame. The creation and destruction of these stack frames is what makes the recursive factorial slower than its iterative counterpart. The accumulation of these frames before the calls start returning is what can potentially exhaust stack space and crash your program.

These concerns are often theoretical. For example, the stack frames for factorial take 16 bytes each (this can vary depending on stack alignment and other factors). If you are running a modern x86 Linux kernel on a computer, you normally have 8 megabytes of stack space, so factorial could handle n up to ~512,000. This is a monstrously large result that takes 8,971,833 bits to represent, so stack space is the least of our problems: a puny integer - even a 64-bit one - will overflow tens of thousands of times over before we run out of stack space.

We’ll look at CPU usage in a moment, but for now let’s take a step back from the bits and bytes and look at recursion as a general technique. Our factorial algorithm boils down to pushing integers N, N-1, … 1 onto a stack, then multiplying them in reverse order. The fact we’re using the program’s call stack to do this is an implementation detail: we could allocate a stack on the heap and use that instead. While the call stack does have special properties, it’s just another data structure at your disposal. I hope the diagram makes that clear.

Once you see the call stack as a data structure, something else becomes clear: piling up all those integers to multiply them afterwards is one dumbass idea. That is the real lameness of this implementation: it’s using a screwdriver to hammer a nail. It’s far more sensible to use an iterative process to calculate factorials.

But there are plenty of screws out there, so let’s pick one. There is a traditional interview question where you’re given a mouse in a maze, and you must help the mouse search for cheese. Suppose the mouse can turn either left or right in the maze. How would you model and solve this problem?

Like most problems in life, you can reduce this rodent quest to a graph, in particular a binary tree where the nodes represent positions in the maze. You could then have the mouse attempt left turns whenever possible, and backtrack to turn right when it reaches a dead end. Here’s the mouse walk in an example maze:

Each edge (line) is a left or right turn taking our mouse to a new position. If either turn is blocked, the corresponding edge does not exist. Now we’re talking! This process is inherently recursive whether you use the call stack or another data structure. But using the call stack is just so easy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#include 
#include "maze.h"

int explore(maze_t *node)
{
	int found = 0;

    if (node == NULL) {
        return 0;
    }

    if (node->hasCheese) {
        return 1; // found cheese
    }

	found = explore(node->left) || explore(node->right);
	return found;
}

int main(int argc)
{
        int found = explore(&maze);
}

Below is the stack when we find the cheese in maze.c:13. You can also see the detailed GDB output and commands used to gather data.

This shows recursion in a much better light because it’s a suitable problem. And that’s no oddity: when it comes to algorithms, recursion is the rule, not the exception. It comes up when we search, when we traverse trees and other data structures, when we parse, when we sort: it’s everywhere. You know how pi or e come up in math all the time because they’re in the foundations of the universe? Recursion is like that: it’s in the fabric of computation.

Steven Skienna’s excellent Algorithm Design Manual is a great place to see that in action as he works through his “war stories” and shows the reasoning behind algorithmic solutions to real-world problems. It’s the best resource I know of to develop your intuition for algorithms. Another good read is McCarthy’s original paper on LISP. Recursion is both in its title and in the foundations of the language. The paper is readable and fun, it’s always a pleasure to see a master at work.

Back to the maze. While it’s hard to get away from recursion here, it doesn’t mean it must be done via the call stack. You could for example use a string like RRLL to keep track of the turns, and rely on the string to decide on the mouse’s next move. Or you can allocate something else to record the state of the cheese hunt. You’d still be implementing a recursive process, but rolling your own data structure.

That’s likely to be more complex because the call stack fits like a glove. Each stack frame records not only the current node, but also the state of computation in that node (in this case, whether we’ve taken only the left, or are already attempting the right). Hence the code becomes trivial. Yet we sometimes give up this sweetness for fear of overflows and hopes of performance. That can be foolish.

As we’ve seen, the stack is large and frequently other constraints kick in before stack space does. One can also check the problem size and ensure it can be handled safely. The CPU worry is instilled chiefly by two widespread pathological examples: the dumb factorial and the hideous O(2ⁿ) recursive Fibonacci without memoization. These are not indicative of sane stack-recursive algorithms.

The reality is that stack operations are fast. Often the offsets to data are known exactly, the stack is hot in the caches, and there are dedicated instructions to get things done. Meanwhile, there is substantial overhead involved in using your own heap-allocated data structures. It’s not uncommon to see people write something that ends up more complex and less performant than call-stack recursion. Finally, modern CPUs are pretty good and often not the bottleneck. Be careful about sacrificing simplicity and as always with performance, measure.

The next post is the last in this stack series, and we’ll look at Tail Calls, Closures, and Other Fauna. Then it’ll be time to visit our old friend, the Linux kernel. Thanks for reading!

Last week we looked at how the stack works and how stack frames are built during function prologues. Now it’s time to look at the inverse process as stack frames are destroyed in function epilogues. Let’s bring back our friend add.c:

1
2
3
4
5
6
7
8
9
10
11

int add(int a, int b)
{
    int result = a + b;
    return result;
}

int main(int argc)
{
    int answer;
    answer = add(40, 2);
}

We’re executing line 4, right after the assignment of a + b into result. This is what happens:

The first instruction is redundant and a little silly because we know eax is already equal to result, but this is what you get with optimization turned off. The leave instruction then runs, doing two tasks for the price of one: it resets esp to point to the start of the current stack frame, and then restores the saved ebp value. These two operations are logically distinct and thus are broken up in the diagram, but they happen atomically if you’re tracing with a debugger.

After leave runs the previous stack frame is restored. The only vestige of the call to add is the return address on top of the stack. It contains the address of the instruction in main that must run after add is done. The ret instruction takes care of it: it pops the return address into the eip register, which points to the next instruction to be executed. The program has now returned to main, which resumes:

main copies the return value from add into local variable answer and then runs its own epilogue, which is identical to any other. Again the only peculiarity in main is that the saved ebp is null, since it is the first stack frame in our code. In the last step, execution has been returned to the C runtime (libc), which will exit to the operating system. Here’s a diagram with the full return sequence for those who need it.

You now have an excellent grasp of how the stack operates, so let’s have some fun and look at one of the most infamous hacks of all time: exploiting the stack buffer overflow. Here is a vulnerable program:

1
2
3
4
5
6
7
8
9
10

void doRead()
{
        char buffer[28];
        gets(buffer);
}

int main(int argc)
{
        doRead();
}

The code above uses gets to read from standard input. gets keeps reading until it encounters a newline or end of file. Here’s what the stack looks like after a string has been read:

The problem here is that gets is unaware of buffer's size: it will blithely keep reading input and stuffing data into the stack beyond buffer, obliterating the saved ebp value, return address, and whatever else is below. To exploit this behavior, attackers craft a precise payload and feed it into the program. This is what the stack looks like during an attack, after the call to gets:

The basic idea is to provide malicious assembly code to be executed and overwrite the return address on the stack to point to that code. It is a bit like a virus invading a cell, subverting it, and introducing some RNA to further its goals.

And like a virus, the exploit’s payload has many notable features. It starts with several nop instructions to increase the odds of successful exploitation. This is because the return address is absolute and must be guessed, since attackers don’t know exactly where in the stack their code will be stored. But as long as they land on a nop, the exploit works: the processor will execute the nops until it hits the instructions that do work.

The exec /bin/sh symbolizes raw assembly instructions that execute a shell (imagine for example that the vulnerability is in a networked program, so the exploit might provide shell access to the system). The idea of feeding raw assembly to a program expecting a command or user input is shocking at first, but that’s part of what makes security research so fun and mind-expanding. To give you an idea of how weird things get, sometimes the vulnerable program calls tolower or toupper on its inputs, forcing attackers to write assembly instructions whose bytes do not fall into the range of upper- or lower-case ascii letters.

Finally, attackers repeat the guessed return address several times, again to tip the odds ever in their favor. By starting on a 4-byte boundary and providing multiple repeats, they are more likely to overwrite the original return address on the stack.

Thankfully, modern operating systems have a host of protections against buffer overflows, including non-executable stacks and stack canaries. The “canary” name comes from the canary in a coal mine expression, an addition to computer science’s rich vocabulary. In the words of Steve McConnell:

Computer science has some of the most colorful language of any field. In what other field can you walk into a sterile room, carefully controlled at 68°F, and find viruses, Trojan horses, worms, bugs, bombs, crashes, flames, twisted sex changers, and fatal errors?

At any rate, here’s what a stack canary looks like:

Canaries are implemented by the compiler. For example, GCC’s stack-protector option causes canaries to be used in any function that is potentially vulnerable. The function prologue loads a magic value into the canary location, and the epilogue makes sure the value is intact. If it’s not, a buffer overflow (or bug) likely happened and the program is aborted via __stack_chk_fail. Due to their strategic location on the stack, canaries make the exploitation of stack buffer overflows much harder.

This finishes our journey within the depths of the stack. We don’t want to delve too greedily and too deep. Next week we’ll go up a notch in abstraction to take a good look at recursion, tail calls and other tidbits, probably using Google’s V8. To end this epilogue and prologue talk, I’ll close with a cherished quote inscribed on a monument in the American National Archives:

Earlier we’ve explored the anatomy of a program in memory, the landscape of how our programs run in a computer. Now we turn to the call stack, the work horse in most programming languages and virtual machines. Along the way we’ll meet fantastic creatures like closures, recursion, and buffer overflows. But the first step is a precise picture of how the stack operates.

The stack is so important because it keeps track of the functions running in a program, and functions are in turn the building blocks of software. In fact, the internal operation of programs is normally very simple. It consists mostly of functions pushing data onto and popping data off the stack as they call each other, while allocating memory on the heap for data that must survive across function calls. This is true for both low-level C software and VM-based languages like JavaScript and C#. A solid grasp of this reality is invaluable for debugging, performance tuning and generally knowing what the hell is going on.

When a function is called, a stack frame is created to support the function’s execution. The stack frame contains the function’s local variables and the arguments passed to the function by its caller. The frame also contains housekeeping information that allows the called function (the callee) to return to the caller safely. The exact contents and layout of the stack vary by processor architecture and function call convention. In this post we look at Intel x86 stacks using C-style function calls (cdecl). Here’s a single stack frame sitting live on top of the stack:

Right away, three CPU registers burst into the scene. The stack pointer, esp, points to the top of the stack. The top is always occupied by the last item that was pushed onto the stack but has not yet been popped off, just as in a real-world stack of plates or $100 bills.

The address stored in esp constantly changes as stack items are pushed and popped, such that it always points to the last item. Many CPU instructions automatically update esp as a side effect, and it’s impractical to use the stack without this register.

In the Intel architecture, as in most, the stack grows towards lower memory addresses. So the “top” is the lowest memory address in the stack containing live data: local_buffer in this case. Notice there’s nothing vague about the arrow from esp to local_buffer. This arrow means business: it points specifically to the first byte occupied by local_buffer because that is the exact address stored in esp.

The second register tracking the stack is ebp, the base pointer or frame pointer. It points to a fixed location within the stack frame of the function currently running and provides a stable reference point (base) for access to arguments and local variables. ebp changes only when a function call begins or ends. Thus we can easily address each item in the stack as an offset from ebp, as shown in the diagram.

Unlike esp, ebp is mostly maintained by program code with little CPU interference. Sometimes there are performance benefits in ditching ebp altogether, which can be done via compiler flags. The Linux kernel is one example where this is done.

Finally, the eax register is used by convention to transfer return values back to the caller for most C data types.

Now let’s inspect the data in our stack frame. These diagram shows precise byte-for-byte contents as you’d see in a debugger, with memory growing left-to-right, top-to-bottom. Here it is:

The local variable local_buffer is a byte array containing a null-terminated ascii string, a staple of C programs. The string was likely read from somewhere, for example keyboard input or a file, and it is 7 bytes long. Since local_buffer can hold 8 bytes, there’s 1 free byte left. The content of this byte is unknown because in the stack’s infinite dance of pushes and pops, you never know what memory holds unless you write to it. Since the C compiler does not initialize the memory for a stack frame, contents are undetermined

• and somewhat random - until written to. This has driven some into madness.

Moving on, local1 is a 4-byte integer and you can see the contents of each byte. It looks like a big number, with all those zeros following the 8, but here your intuition leads you astray.

Intel processors are little endian machines, meaning that numbers in memory start with the little end first. So the least significant byte of a multi-byte number is in the lowest memory address. Since that is normally shown leftmost, this departs from our usual representation of numbers. It helps to know that this endian talk is borrowed from Gulliver’s Travels: just as folks in Lilliput eat their eggs starting from the little end, Intel processors eat their numbers starting from the little byte.

So local1 in fact holds the number 8, as in the legs of an octopus. param1, however, has a value of 2 in the second byte position, so its mathematical value is 2 * 256 = 512 (we multiply by 256 because each place value ranges from 0 to 255). Meanwhile, param2 is carrying weight at 1 * 256 * 256 = 65536.

The housekeeping data in this stack frame consists of two crucial pieces: the address of the previous stack frame (saved ebp) and the address of the instruction to be executed upon the function’s exit (the return address). Together, they make it possible for the function to return sanely and for the program to keep running along.

Now let’s see the birth of a stack frame to build a clear mental picture of how this all works together. Stack growth is puzzling at first because it happens in the opposite direction you’d expect. For example, to allocate 8 bytes on the stack one subtracts 8 from esp, and subtraction is an odd way to grow something.

Let’s take a simple C program:

1
2
3
4
5
6
7
8
9
10
11

int add(int a, int b)
{
	int result = a + b;
	return result;
}

int main(int argc)
{
	int answer;
	answer = add(40, 2);
}

Suppose we run this in Linux without command-line parameters. When you run a C program, the first code to actually execute is in the C runtime library, which then calls our main function. The diagrams below show step-by-step what happens as the program runs. Each diagram links to GDB output showing the state of memory and registers. You may also see the GDB commands used and the whole GDB output. Here we go:

Steps 2 and 3, along with 4 below, are the function prologue, which is common to nearly all functions: the current value of ebp is saved to the top of the stack, and then esp is copied to ebp, establishing a new frame. main’s prologue is like any other, but with the peculiarity that ebp is zeroed out when the program starts.

If you were to inspect the stack below argc (to the right) you’d find more data, including pointers to the program name and command-line parameters (the traditional C argv), plus pointers to Unix environment variables and their actual contents. But that’s not important here, so the ball keeps rolling towards the add() call:

After main subtracts 12 from esp to get the stack space it needs, it sets the values for a and b. Values in memory are shown in hex and little-endian format, as you’d see in a debugger. Once parameter values are set, main calls add and it starts running:

Now there’s some excitement! We get another prologue, but this time you can see clearly how the stack frames form a linked list, starting at ebp and going down the stack. This is how debuggers and Exception objects in higher-level languages get their stack traces. You can also see the much more typical catching up of ebp to esp when a new frame is born. And again, we subtract from esp to get more stack space.

There’s also the slightly weird reversal of bytes when the ebp register value is copied to memory. What’s happening here is that registers don’t really have endianness: there are no “growing addresses” inside a register as there are for memory. Thus by convention debuggers show register values in the most natural format to humans: most significant to least significant digits. So the results of the copy in a little-endian machine appear reversed in the usual left-to-right notation for memory. I want the diagrams to provide an accurate picture of what you find in the trenches, so there you have it.

With the hard part behind us, we add:

There are guest register appearances to help out with the addition, but otherwise no alarms and no surprises. add did its job, and at this point the stack action would go in reverse, but we’ll save that for next time.

Anybody who’s read this far deserves a souvenir, so I’ve made a large diagram showing all the steps combined in a fit of nerd pride.

It looks tame once it’s all laid out. Those little boxes help a lot. In fact, little boxes are the chief tool of computer science. I hope the pictures and register movements provide an intuitive mental picture that integrates stack growth and memory contents. Up close, our software doesn’t look too far from a simple Turing machine.

This concludes the first part of our stack tour. There’s some more byte spelunking ahead, and then it’s on to see higher level programming concepts built on this foundation. See you next week.

Previously we looked at how the kernel manages virtual memory for a user process, but files and I/O were left out. This post covers the important and often misunderstood relationship between files and memory and its consequences for performance.

Two serious problems must be solved by the OS when it comes to files. The first one is the mind-blowing slowness of hard drives, and disk seeks in particular, relative to memory. The second is the need to load file contents in physical memory once and share the contents among programs. If you use Process Explorer to poke at Windows processes, you’ll see there are ~15MB worth of common DLLs loaded in every process. My Windows box right now is running 100 processes, so without sharing I’d be using up to ~1.5 GB of physical RAM just for common DLLs. No good. Likewise, nearly all Linux programs need ld.so and libc, plus other common libraries.

Happily, both problems can be dealt with in one shot: the page cache, where the kernel stores page-sized chunks of files. To illustrate the page cache, I’ll conjure a Linux program named render, which opens file scene.dat and reads it 512 bytes at a time, storing the file contents into a heap-allocated block. The first read goes like this:

After 12KB have been read, render's heap and the relevant page frames look thus:

This looks innocent enough, but there’s a lot going on. First, even though this program uses regular read calls, three 4KB page frames are now in the page cache storing part of scene.dat. People are sometimes surprised by this, but all regular file I/O happens through the page cache. In x86 Linux, the kernel thinks of a file as a sequence of 4KB chunks. If you read a single byte from a file, the whole 4KB chunk containing the byte you asked for is read from disk and placed into the page cache. This makes sense because sustained disk throughput is pretty good and programs normally read more than just a few bytes from a file region. The page cache knows the position of each 4KB chunk within the file, depicted above as #0, #1, etc. Windows uses 256KB views analogous to pages in the Linux page cache.

Sadly, in a regular file read the kernel must copy the contents of the page cache into a user buffer, which not only takes cpu time and hurts the cpu caches, but also wastes physical memory with duplicate data. As per the diagram above, the scene.dat contents are stored twice, and each instance of the program would store the contents an additional time. We’ve mitigated the disk latency problem but failed miserably at everything else. Memory-mapped files are the way out of this madness:

When you use file mapping, the kernel maps your program’s virtual pages directly onto the page cache. This can deliver a significant performance boost: Windows System Programming reports run time improvements of 30% and up relative to regular file reads, while similar figures are reported for Linux and Solaris in Advanced Programming in the Unix Environment. You might also save large amounts of physical memory, depending on the nature of your application.

As always with performance, measurement is everything, but memory mapping earns its keep in a programmer’s toolbox. The API is pretty nice too, it allows you to access a file as bytes in memory and does not require your soul and code readability in exchange for its benefits. Mind your address space and experiment with mmap in Unix-like systems, CreateFileMapping in Windows, or the many wrappers available in high level languages. When you map a file its contents are not brought into memory all at once, but rather on demand via page faults. The fault handler maps your virtual pages onto the page cache after obtaining a page frame with the needed file contents. This involves disk I/O if the contents weren’t cached to begin with.

Now for a pop quiz. Imagine that the last instance of our render program exits. Would the pages storing scene.dat in the page cache be freed immediately? People often think so, but that would be a bad idea. When you think about it, it is very common for us to create a file in one program, exit, then use the file in a second program. The page cache must handle that case. When you think more about it, why should the kernel ever get rid of page cache contents? Remember that disk is 5 orders of magnitude slower than RAM, hence a page cache hit is a huge win. So long as there’s enough free physical memory, the cache should be kept full. It is therefore not dependent on a particular process, but rather it’s a system-wide resource. If you run render a week from now and scene.dat is still cached, bonus! This is why the kernel cache size climbs steadily until it hits a ceiling. It’s not because the OS is garbage and hogs your RAM, it’s actually good behavior because in a way free physical memory is a waste. Better use as much of the stuff for caching as possible.

Due to the page cache architecture, when a program calls write() bytes are simply copied to the page cache and the page is marked dirty. Disk I/O normally does not happen immediately, thus your program doesn’t block waiting for the disk. On the downside, if the computer crashes your writes will never make it, hence critical files like database transaction logs must be fsync()ed (though one must still worry about drive controller caches, oy!). Reads, on the other hand, normally block your program until the data is available. Kernels employ eager loading to mitigate this problem, an example of which is read ahead where the kernel preloads a few pages into the page cache in anticipation of your reads. You can help the kernel tune its eager loading behavior by providing hints on whether you plan to read a file sequentially or randomly (see madvise(), readahead(), Windows cache hints ). Linux does read-ahead for memory-mapped files, but I’m not sure about Windows. Finally, it’s possible to bypass the page cache using O_DIRECT in Linux or NO_BUFFERING in Windows, something database software often does.

A file mapping may be private or shared. This refers only to updates made to the contents in memory: in a private mapping the updates are not committed to disk or made visible to other processes, whereas in a shared mapping they are. Kernels use the copy on write mechanism, enabled by page table entries, to implement private mappings. In the example below, both render and another program called render3d (am I creative or what?) have mapped scene.dat privately. Render then writes to its virtual memory area that maps the file:

The read-only page table entries shown above do not mean the mapping is read only, they’re merely a kernel trick to share physical memory until the last possible moment. You can see how ‘private’ is a bit of a misnomer until you remember it only applies to updates. A consequence of this design is that a virtual page that maps a file privately sees changes done to the file by other programs as long as the page has only been read from. Once copy-on-write is done, changes by others are no longer seen. This behavior is not guaranteed by the kernel, but it’s what you get in x86 and makes sense from an API perspective. By contrast, a shared mapping is simply mapped onto the page cache and that’s it. Updates are visible to other processes and end up in the disk. Finally, if the mapping above were read-only, page faults would trigger a segmentation fault instead of copy on write.

Dynamically loaded libraries are brought into your program’s address space via file mapping. There’s nothing magical about it, it’s the same private file mapping available to you via regular APIs. Below is an example showing part of the address spaces from two running instances of the file-mapping render program, along with physical memory, to tie together many of the concepts we’ve seen.

This concludes our 3-part series on memory fundamentals. I hope the series was useful and provided you with a good mental model of these OS topics.

62 Comments

I hope the previous post explained virtual memory adequately, but I must admit I held back a much better explanation, which I first saw in Expert C Programming. It wasn't written by the book's author, Peter van der Linden, but rather by Jeff Berryman in 1972. Here goes:

The Thing King and the Paging Game

This note is a formal non-working paper of the Project MAC Computer Systems Research Division. It should be reproduced and distributed wherever levity is lacking, and may be referenced at your own risk in other publications.

Rules

1. Each player gets several million things.
2. Things are kept in crates that hold 4096 things each. Things in the same crate are called crate-mates.
3. Crates are stored either in the workshop or the warehouses. The workshop is almost always too small to hold all the crates.
4. There is only one workshop but there may be several warehouses. Everybody shares them.
5. Each thing has its own thing number.
6. What you do with a thing is to zark it. Everybody takes turns zarking.
7. You can only zark your things, not anybody else's.
8. Things can only be zarked when they are in the workshop.
9. Only the Thing King knows whether a thing is in the workshop or in a warehouse.
10. The longer a thing goes without being zarked, the grubbier it is said to become.
11. The way you get things is to ask the Thing King. He only gives out things by the crateful. This is to keep the royal overhead down.
12. The way you zark a thing is to give its thing number. If you give the number of a thing that happens to be in a workshop it gets zarked right away. If it is in a warehouse, the Thing King packs the crate containing your thing back into the workshop. If there is no room in the workshop, he first finds the grubbiest crate in the workshop, whether it be yours or somebody else's, and packs it off with all its crate-mates to a warehouse. In its place he puts the crate containing your thing. Your thing then gets zarked and you never know that it wasn't in the workshop all along.
13. Each player's stock of things have the same numbers as everybody else's. The Thing King always knows who owns what thing and whose turn it is, so you can't ever accidentally zark somebody else's thing even if it has the same thing number as one of yours.

Notes

1. Traditionally, the Thing King sits at a large, segmented table and is attended to by pages (the so-called "table pages") whose job it is to help the king remember where all the things are and who they belong to.
2. One consequence of Rule 13 is that everybody's thing numbers will be similar from game to game, regardless of the number of players.
3. The Thing King has a few things of his own, some of which move back and forth between workshop and warehouse just like anybody else's, but some of which are just too heavy to move out of the workshop.
4. With the given set of rules, oft-zarked things tend to get kept mostly in the workshop while little-zarked things stay mostly in a warehouse. This is efficient stock control.

Long Live the Thing King!

Update: Alex pointed out the difficulties of measuring grubbiness in a comment below.

14 Comments

After examining the virtual address layout of a process, we turn to the kernel and its mechanisms for managing user memory. Here is gonzo again:

Linux processes are implemented in the kernel as instances of task_struct, the process descriptor. The mm field in task_struct points to the memory descriptor, mm_struct, which is an executive summary of a program's memory. It stores the start and end of memory segments as shown above, the number of physical memory pages used by the process (rss stands for Resident Set Size), the amount of virtual address space used, and other tidbits. Within the memory descriptor we also find the two work horses for managing program memory: the set of virtual memory areas and the page tables. Gonzo's memory areas are shown below:

Each virtual memory area (VMA) is a contiguous range of virtual addresses; these areas never overlap. An instance of vm_area_struct fully describes a memory area, including its start and end addresses, flags to determine access rights and behaviors, and the vm_file field to specify which file is being mapped by the area, if any. A VMA that does not map a file is anonymous. Each memory segment above (e.g., heap, stack) corresponds to a single VMA, with the exception of the memory mapping segment. This is not a requirement, though it is usual in x86 machines. VMAs do not care which segment they are in.

A program's VMAs are stored in its memory descriptor both as a linked list in the mmap field, ordered by starting virtual address, and as a red-black tree rooted at the mm_rb field. The red-black tree allows the kernel to search quickly for the memory area covering a given virtual address. When you read file /proc/pid_of_process/maps, the kernel is simply going through the linked list of VMAs for the process and printing each one.

In Windows, the EPROCESS block is roughly a mix of task_struct and mm_struct. The Windows analog to a VMA is the Virtual Address Descriptor, or VAD; they are stored in an AVL tree. You know what the funniest thing about Windows and Linux is? It's the little differences.

The 4GB virtual address space is divided into pages. x86 processors in 32-bit mode support page sizes of 4KB, 2MB, and 4MB. Both Linux and Windows map the user portion of the virtual address space using 4KB pages. Bytes 0-4095 fall in page 0, bytes 4096-8191 fall in page 1, and so on. The size of a VMA must be a multiple of page size. Here's 3GB of user space in 4KB pages:

The processor consults page tables to translate a virtual address into a physical memory address. Each process has its own set of page tables; whenever a process switch occurs, page tables for user space are switched as well. Linux stores a pointer to a process' page tables in the pgd field of the memory descriptor. To each virtual page there corresponds one page table entry (PTE) in the page tables, which in regular x86 paging is a simple 4-byte record shown below:

Linux has functions to read and set each flag in a PTE. Bit P tells the processor whether the virtual page is present in physical memory. If clear (equal to 0), accessing the page triggers a page fault. Keep in mind that when this bit is zero, the kernel can do whatever it pleases with the remaining fields. The R/W flag stands for read/write; if clear, the page is read-only. Flag U/S stands for user/supervisor; if clear, then the page can only be accessed by the kernel. These flags are used to implement the read-only memory and protected kernel space we saw before.

Bits D and A are for dirty and accessed. A dirty page has had a write, while an accessed page has had a write or read. Both flags are sticky: the processor only sets them, they must be cleared by the kernel. Finally, the PTE stores the starting physical address that corresponds to this page, aligned to 4KB. This naive-looking field is the source of some pain, for it limits addressable physical memory to 4 GB. The other PTE fields are for another day, as is Physical Address Extension.

A virtual page is the unit of memory protection because all of its bytes share the U/S and R/W flags. However, the same physical memory could be mapped by different pages, possibly with different protection flags. Notice that execute permissions are nowhere to be seen in the PTE. This is why classic x86 paging allows code on the stack to be executed, making it easier to exploit stack buffer overflows (it's still possible to exploit non-executable stacks using return-to-libc and other techniques). This lack of a PTE no-execute flag illustrates a broader fact: permission flags in a VMA may or may not translate cleanly into hardware protection. The kernel does what it can, but ultimately the architecture limits what is possible.

Virtual memory doesn't store anything, it simply maps a program's address space onto the underlying physical memory, which is accessed by the processor as a large block called the physical address space. While memory operations on the bus are somewhat involved, we can ignore that here and assume that physical addresses range from zero to the top of available memory in one-byte increments. This physical address space is broken down by the kernel into page frames. The processor doesn't know or care about frames, yet they are crucial to the kernel because the page frame is the unit of physical memory management. Both Linux and Windows use 4KB page frames in 32-bit mode; here is an example of a machine with 2GB of RAM:

In Linux each page frame is tracked by a descriptor and several flags. Together these descriptors track the entire physical memory in the computer; the precise state of each page frame is always known. Physical memory is managed with the buddy memory allocation technique, hence a page frame is free if it's available for allocation via the buddy system. An allocated page frame might be anonymous, holding program data, or it might be in the page cache, holding data stored in a file or block device. There are other exotic page frame uses, but leave them alone for now. Windows has an analogous Page Frame Number (PFN) database to track physical memory.

Let's put together virtual memory areas, page table entries and page frames to understand how this all works. Below is an example of a user heap:

Blue rectangles represent pages in the VMA range, while arrows represent page table entries mapping pages onto page frames. Some virtual pages lack arrows; this means their corresponding PTEs have the Present flag clear. This could be because the pages have never been touched or because their contents have been swapped out. In either case access to these pages will lead to page faults, even though they are within the VMA. It may seem strange for the VMA and the page tables to disagree, yet this often happens.

A VMA is like a contract between your program and the kernel. You ask for something to be done (memory allocated, a file mapped, etc.), the kernel says "sure", and it creates or updates the appropriate VMA. But it does not actually honor the request right away, it waits until a page fault happens to do real work. The kernel is a lazy, deceitful sack of scum; this is the fundamental principle of virtual memory. It applies in most situations, some familiar and some surprising, but the rule is that VMAs record what has been agreed upon, while PTEs reflect what has actually been done by the lazy kernel. These two data structures together manage a program's memory; both play a role in resolving page faults, freeing memory, swapping memory out, and so on. Let's take the simple case of memory allocation:

When the program asks for more memory via the brk() system call, the kernel simply updates the heap VMA and calls it good. No page frames are actually allocated at this point and the new pages are not present in physical memory. Once the program tries to access the pages, the processor page faults and do_page_fault() is called. It searches for the VMA covering the faulted virtual address using find_vma(). If found, the permissions on the VMA are also checked against the attempted access (read or write). If there's no suitable VMA, no contract covers the attempted memory access and the process is punished by Segmentation Fault.

When a VMA is found the kernel must handle the fault by looking at the PTE contents and the type of VMA. In our case, the PTE shows the page is not present. In fact, our PTE is completely blank (all zeros), which in Linux means the virtual page has never been mapped. Since this is an anonymous VMA, we have a purely RAM affair that must be handled by do_anonymous_page(), which allocates a page frame and makes a PTE to map the faulted virtual page onto the freshly allocated frame.

Things could have been different. The PTE for a swapped out page, for example, has 0 in the Present flag but is not blank. Instead, it stores the swap location holding the page contents, which must be read from disk and loaded into a page frame by do_swap_page() in what is called a major fault.

This concludes the first half of our tour through the kernel's user memory management. In the next post, we'll throw files into the mix to build a complete picture of memory fundamentals, including consequences for performance.

124 Comments

Memory management is the heart of operating systems; it is crucial for both programming and system administration. In the next few posts I'll cover memory with an eye towards practical aspects, but without shying away from internals. While the concepts are generic, examples are mostly from Linux and Windows on 32-bit x86. This first post describes how programs are laid out in memory.

Each process in a multi-tasking OS runs in its own memory sandbox. This sandbox is the virtual address space, which in 32-bit mode is always a 4GB block of memory addresses. These virtual addresses are mapped to physical memory by page tables, which are maintained by the operating system kernel and consulted by the processor. Each process has its own set of page tables, but there is a catch. Once virtual addresses are enabled, they apply to all software running in the machine, including the kernel itself. Thus a portion of the virtual address space must be reserved to the kernel:

This does not mean the kernel uses that much physical memory, only that it has that portion of address space available to map whatever physical memory it wishes. Kernel space is flagged in the page tables as exclusive to privileged code (ring 2 or lower), hence a page fault is triggered if user-mode programs try to touch it. In Linux, kernel space is constantly present and maps the same physical memory in all processes. Kernel code and data are always addressable, ready to handle interrupts or system calls at any time. By contrast, the mapping for the user-mode portion of the address space changes whenever a process switch happens:

Blue regions represent virtual addresses that are mapped to physical memory, whereas white regions are unmapped. In the example above, Firefox has used far more of its virtual address space due to its legendary memory hunger. The distinct bands in the address space correspond to memory segments like the heap, stack, and so on. Keep in mind these segments are simply a range of memory addresses and have nothing to do with Intel-style segments. Anyway, here is the standard segment layout in a Linux process:

When computing was happy and safe and cuddly, the starting virtual addresses for the segments shown above were exactly the same for nearly every process in a machine. This made it easy to exploit security vulnerabilities remotely. An exploit often needs to reference absolute memory locations: an address on the stack, the address for a library function, etc. Remote attackers must choose this location blindly, counting on the fact that address spaces are all the same. When they are, people get pwned. Thus address space randomization has become popular. Linux randomizes the stack, memory mapping segment, and heap by adding offsets to their starting addresses. Unfortunately the 32-bit address space is pretty tight, leaving little room for randomization and hampering its effectiveness.

The topmost segment in the process address space is the stack, which stores local variables and function parameters in most programming languages. Calling a method or function pushes a new stack frame onto the stack. The stack frame is destroyed when the function returns. This simple design, possible because the data obeys strict LIFO order, means that no complex data structure is needed to track stack contents - a simple pointer to the top of the stack will do. Pushing and popping are thus very fast and deterministic. Also, the constant reuse of stack regions tends to keep active stack memory in the cpu caches, speeding up access. Each thread in a process gets its own stack.

It is possible to exhaust the area mapping the stack by pushing more data than it can fit. This triggers a page fault that is handled in Linux by expand_stack(), which in turn calls acct_stack_growth() to check whether it's appropriate to grow the stack. If the stack size is below RLIMIT_STACK (usually 8MB), then normally the stack grows and the program continues merrily, unaware of what just happened. This is the normal mechanism whereby stack size adjusts to demand. However, if the maximum stack size has been reached, we have a stack overflow and the program receives a Segmentation Fault. While the mapped stack area expands to meet demand, it does not shrink back when the stack gets smaller. Like the federal budget, it only expands.

Dynamic stack growth is the only situation in which access to an unmapped memory region, shown in white above, might be valid. Any other access to unmapped memory triggers a page fault that results in a Segmentation Fault. Some mapped areas are read-only, hence write attempts to these areas also lead to segfaults.

Below the stack, we have the memory mapping segment. Here the kernel maps contents of files directly to memory. Any application can ask for such a mapping via the Linux mmap() system call (implementation) or CreateFileMapping() / MapViewOfFile() in Windows. Memory mapping is a convenient and high-performance way to do file I/O, so it is used for loading dynamic libraries. It is also possible to create an anonymous memory mapping that does not correspond to any files, being used instead for program data. In Linux, if you request a large block of memory via malloc(), the C library will create such an anonymous mapping instead of using heap memory. 'Large' means larger than MMAP_THRESHOLD bytes, 128 kB by default and adjustable via mallopt().

Speaking of the heap, it comes next in our plunge into address space. The heap provides runtime memory allocation, like the stack, meant for data that must outlive the function doing the allocation, unlike the stack. Most languages provide heap management to programs. Satisfying memory requests is thus a joint affair between the language runtime and the kernel. In C, the interface to heap allocation is malloc() and friends, whereas in a garbage-collected language like C# the interface is the new keyword.

If there is enough space in the heap to satisfy a memory request, it can be handled by the language runtime without kernel involvement. Otherwise the heap is enlarged via the brk() system call (implementation) to make room for the requested block. Heap management is complex, requiring sophisticated algorithms that strive for speed and efficient memory usage in the face of our programs' chaotic allocation patterns. The time needed to service a heap request can vary substantially. Real-time systems have special-purpose allocators to deal with this problem. Heaps also become fragmented, shown below:

Finally, we get to the lowest segments of memory: BSS, data, and program text. Both BSS and data store contents for static (global) variables in C. The difference is that BSS stores the contents of uninitialized static variables, whose values are not set by the programmer in source code. The BSS memory area is anonymous: it does not map any file. If you say static int cntActiveUsers, the contents of cntActiveUsers live in the BSS.

The data segment, on the other hand, holds the contents for static variables initialized in source code. This memory area is not anonymous. It maps the part of the program's binary image that contains the initial static values given in source code. So if you say static int cntWorkerBees = 10, the contents of cntWorkerBees live in the data segment and start out as 10. Even though the data segment maps a file, it is a private memory mapping, which means that updates to memory are not reflected in the underlying file. This must be the case, otherwise assignments to global variables would change your on-disk binary image. Inconceivable!

The data example in the diagram is trickier because it uses a pointer. In that case, the contents of pointer gonzo - a 4-byte memory address - live in the data segment. The actual string it points to does not, however. The string lives in the text segment, which is read-only and stores all of your code in addition to tidbits like string literals. The text segment also maps your binary file in memory, but writes to this area earn your program a Segmentation Fault. This helps prevent pointer bugs, though not as effectively as avoiding C in the first place. Here's a diagram showing these segments and our example variables:

You can examine the memory areas in a Linux process by reading the file /proc/pid_of_process/maps. Keep in mind that a segment may contain many areas. For example, each memory mapped file normally has its own area in the mmap segment, and dynamic libraries have extra areas similar to BSS and data. The next post will clarify what 'area' really means. Also, sometimes people say "data segment" meaning all of data + bss + heap.

You can examine binary images using the nm and objdump commands to display symbols, their addresses, segments, and so on. Finally, the virtual address layout described above is the "flexible" layout in Linux, which has been the default for a few years. It assumes that we have a value for RLIMIT_STACK. When that's not the case, Linux reverts back to the "classic" layout shown below:

That's it for virtual address space layout. The next post discusses how the kernel keeps track of these memory areas. Coming up we'll look at memory mapping, how file reading and writing ties into all this and what memory usage figures mean.

189 Comments

When trying to understand complex systems, you can often learn a lot by stripping away abstractions and looking at their lowest levels. In that spirit we take a look at memory and I/O ports in their simplest and most fundamental level: the interface between the processor and bus. These details underlie higher level topics like thread synchronization and the need for the Core i7. Also, since I'm a programmer I ignore things EE people care about. Here's our friend the Core 2 again:

A Core 2 processor has 775 pins, about half of which only provide power and carry no data. Once you group the pins by functionality, the physical interface to the processor is surprisingly simple. The diagram shows the key pins involved in a memory or I/O port operation: address lines, data pins, and request pins. These operations take place in the context of a transaction on the front side bus. FSB transactions go through 5 phases: arbitration, request, snoop, response, and data. Throughout these phases, different roles are played by the components on the FSB, which are called agents. Normally the agents are all the processors plus the northbridge.

We only look at the request phase in this post, in which 2 packets are output by the request agent, who is usually a processor. Here are the juiciest bits of the first packet, output by the address and request pins:

The address lines output the starting physical memory address for the transaction. We have 33 bits but they are interpreted as bits 35-3 of an address in which bits 2-0 are zero. Hence we have a 36-bit address, aligned to 8 bytes, for a total of 64GB addressable physical memory. This has been the case since the Pentium Pro. The request pins specify what type of transaction is being initiated; in I/O requests the address pins specify an I/O port rather than a memory address. After the first packet is output, the same pins transmit a second packet in the subsequent bus clock cycle:

The attribute signals are interesting: they reflect the 5 types of memory caching behavior available in Intel processors. By putting this information on the FSB, the request agent lets other processors know how this transaction affects their caches, and how the memory controller (northbridge) should behave. The processor determines the type of a given memory region mainly by looking at page tables, which are maintained by the kernel.

Typically kernels treat all RAM memory as write-back, which yields the best performance. In write-back mode the unit of memory access is the cache line, 64 bytes in the Core 2. If a program reads a single byte in memory, the processor loads the whole cache line that contains that byte into the L2 and L1 caches. When a program writes to memory, the processor only modifies the line in the cache, but does not update main memory. Later, when it becomes necessary to post the modified line to the bus, the whole cache line is written at once. So most requests have 11 in their length field, for 64 bytes. Here's a read example in which the data is not in the caches:

Some of the physical memory range in an Intel computer is mapped to devices like hard drives and network cards instead of actual RAM memory. This allows drivers to communicate with their devices by writing to and reading from memory. The kernel marks these memory regions as uncacheable in the page tables. Accesses to uncacheable memory regions are reproduced in the bus exactly as requested by a program or driver. Hence it's possible to read or write single bytes, words, and so on. This is done via the byte enable mask in packet B above.

The primitives discussed here have many implications. For example:

1. Performance-sensitive applications should try to pack data that is accessed together into the same cache line. Once the cache line is loaded, further reads are much faster and extra RAM accesses are avoided.
2. Any memory access that falls within a single cache line is guaranteed to be atomic (assuming write-back memory). Such an access is serviced by the processor's L1 cache and the data is read or written all at once; it cannot be affected halfway by other processors or threads. In particular, 32-bit and 64-bit operations that don't cross cache line boundaries are atomic.
3. The front bus is shared by all agents, who must arbitrate for bus ownership before they can start a transaction. Moreover, all agents must listen to all transactions in order to maintain cache coherence. Thus bus contention becomes a severe problem as more cores and processors are added to Intel computers. The Core i7 solves this by having processors attached directly to memory and communicating in a point-to-point rather than broadcast fashion.

These are the highlights of physical memory requests; the bus will surface again later in connection with locking, multi-threading, and cache coherence. The first time I saw FSB packet descriptions I had a huge "ahhh!" moment so I hope someone out there gets the same benefit. In the next post we'll go back up the abstraction ladder to take a thorough look at virtual memory.

This post shows briefly how CPU caches are organized in modern Intel processors. Cache discussions often lack concrete examples, obfuscating the simple concepts involved. Or maybe my pretty little head is slow. At any rate, here's half the story on how a Core 2 L1 cache is accessed:

The unit of data in the cache is the line, which is just a contiguous chunk of bytes in memory. This cache uses 64-byte lines. The lines are stored in cache banks or ways, and each way has a dedicated directory to store its housekeeping information. You can imagine each way and its directory as columns in a spreadsheet, in which case the rows are the sets. Then each cell in the way column contains a cache line, tracked by the corresponding cell in the directory. This particular cache has 64 sets and 8 ways, hence 512 cells to store cache lines, which adds up to 32KB of space.

In this cache's view of the world, physical memory is divided into 4KB physical pages. Each page has 4KB / 64 bytes == 64 cache lines in it. When you look at a 4KB page, bytes 0 through 63 within that page are in the first cache line, bytes 64-127 in the second cache line, and so on. The pattern repeats for each page, so the 3rd line in page 0 is different than the 3rd line in page 1.

In a fully associative cache any line in memory can be stored in any of the cache cells. This makes storage flexible, but it becomes expensive to search for cells when accessing them. Since the L1 and L2 caches operate under tight constraints of power consumption, physical space, and speed, a fully associative cache is not a good trade off in most scenarios.

Instead, this cache is set associative, which means that a given line in memory can only be stored in one specific set (or row) shown above. So the first line of any physical page (bytes 0-63 within a page) must be stored in row 0, the second line in row 1, etc. Each row has 8 cells available to store the cache lines it is associated with, making this an 8-way associative set. When looking at a memory address, bits 11-6 determine the line number within the 4KB page and therefore the set to be used. For example, physical address 0x800010a0 has 000010 in those bits so it must be stored in set 2.

But we still have the problem of finding which cell in the row holds the data, if any. That's where the directory comes in. Each cached line is tagged by its corresponding directory cell; the tag is simply the number for the page where the line came from. The processor can address 64GB of physical RAM, so there are 64GB / 4KB == 2²⁴ of these pages and thus we need 24 bits for our tag. Our example physical address 0x800010a0 corresponds to page number 524,289. Here's the second half of the story:

Since we only need to look in one set of 8 ways, the tag matching is very fast; in fact, electrically all tags are compared simultaneously, which I tried to show with the arrows. If there's a valid cache line with a matching tag, we have a cache hit. Otherwise, the request is forwarded to the L2 cache, and failing that to main system memory. Intel builds large L2 caches by playing with the size and quantity of the ways, but the design is the same. For example, you could turn this into a 64KB cache by adding 8 more ways. Then increase the number of sets to 4096 and each way can store 256KB. These two modifications would deliver a 4MB L2 cache. In this scenario, you'd need 18 bits for the tags and 12 for the set index; the physical page size used by the cache is equal to its way size.

If a set fills up, then a cache line must be evicted before another one can be stored. To avoid this, performance-sensitive programs try to organize their data so that memory accesses are evenly spread among cache lines. For example, suppose a program has an array of 512-byte objects such that some objects are 4KB apart in memory. Fields in these objects fall into the same lines and compete for the same cache set. If the program frequently accesses a given field (e.g., the vtable by calling a virtual method), the set will likely fill up and the cache will start trashing as lines are repeatedly evicted and later reloaded. Our example L1 cache can only hold the vtables for 8 of these objects due to set size. This is the cost of the set associativity trade-off: we can get cache misses due to set conflicts even when overall cache usage is not heavy. However, due to the relative speeds in a computer, most apps don't need to worry about this anyway.

A memory access usually starts with a linear (virtual) address, so the L1 cache relies on the paging unit to obtain the physical page address used for the cache tags. By contrast, the set index comes from the least significant bits of the linear address and is used without translation (bits 11-6 in our example). Hence the L1 cache is physically tagged but virtually indexed, helping the CPU to parallelize lookup operations. Because the L1 way is never bigger than an MMU page, a given physical memory location is guaranteed to be associated with the same set even with virtual indexing. L2 caches, on the other hand, must be physically tagged and physically indexed because their way size can be bigger than MMU pages. But then again, by the time a request gets to the L2 cache the physical address was already resolved by the L1 cache, so it works out nicely.

Finally, a directory cell also stores the state of its corresponding cached line. A line in the L1 code cache is either Invalid or Shared (which means valid, really). In the L1 data cache and the L2 cache, a line can be in any of the 4 MESI states: Modified, Exclusive, Shared, or Invalid. Intel caches are inclusive: the contents of the L1 cache are duplicated in the L2 cache. These states will play a part in later posts about threading, locking, and that kind of stuff. Next time we'll look at the front side bus and how memory access really works. This is going to be memory week.

Update: Dave brought up direct-mapped caches in a comment below. They’re basically a special case of set-associative caches that have only one way. In the trade-off spectrum, they’re the opposite of fully associative caches: blazing fast access, lots of conflict misses.

26 Comments