The twisted road through right-to-left language support

In the beginning days of the Internet—way, way back, when dinosaurs roamed the earth and half of you who are reading this post were probably in diapers—the Internet assumed pretty much everything is left-to-right.

I remember building web pages in raw HTML that most of us would cringe at today. There were no sites, really—only a collection of static HTML pages that, more often than not, included horrendous tags such as <blink> and <marquee> and featured pages in which one-font-served-all and the backgrounds were tiled. Ah, the good ol' days.

Those days, Hebrew was, in fact, typed backwards. If I wanted to write the Hebrew word "שלום", which starts with the hebrew letter "ש", I would have to type it backwards, starting with the letter "ם", and produce "םולש"—because the letters would appear sequentially from left to right. This might be doable when typing one or two words, but if you had an entire paragraph or an entire article, it could get annoying fast.

There were several tools you could download in those ancient days that would take your text and flip it. 'Cause that's how we rolled back then.

Luckily, Unicode came in and defined directionality, and although Unicode still has problems, RTL users can at least type their language normally, rather than learn to write backwards. That helps.

Strong types

Strong types are character sets that have explicit directionality. Hebrew is right-to-left, always. English is left-to-right, always.** When I type in either of those character-sets, my characters would appear in sequence, one after the other, according to the directionality. This is how the word "Hello" appears from left to right, whereas the word "שלום" appears from right to left.

Strong types also set the directionality of the space they're in, meaning that if I inserted any characters that have weak or neutral directionality in the middle of the sentence you're reading now (and I have already done that), they will assume the direction of the strongly typed string—in this case, English. So, strong type isn't just about the character itself, but also its surroundings.

Weak types

Weak types are fun. These are sequences of characters that might have a direction, but it doesn't affect their surroundings, and may be adjusted based on their surrounding text. In this group are characters such as numbers, plus and minus signs, colon, comma, period, and other control characters.

According to the Unicode bidirectionality algorithm specifications, weak types resolve according to the previous characters.

Neutral types

Neutral types are the funnest. Neutral characters are character types that can be either right-to-left or left-to-right, so they completely depend on what string surrounds them. These include things such as new-line characters, tabs, and white-space.

According to the Unicode bidirectionality algorithm specifications, neutral types resolve their directionality according to the surrounding text.

Implicit level types: When what you type is not quite what you get

So we have strong types, weak types, and neutral types, but that's not where our directionality double-take ends. In fact, the real doozies are characters that are resolved differently (as in, they take literally different shapes) in either RTL or LTR.

Yes, you read that right: They actually literally and quite visibly look different when written inside an LTR string versus inside an RTL string.

The best examples for this are parentheses and (my personal best friend) the bracket. These symbols are, in fact, icons that represent direction already. The button on your keyboard that has "(" on it is not quite that, but rather a symbol of "open parentheses." In English (which is left-to-right) the symbol is naturally ( to open parentheses, and ) to close them. But in Hebrew and Arabic and the other RTL languages, the "open parentheses" symbol is the reverse ), because the string is right to left. So this symbol would appear on your screen either ( or ) depending where you typed it.

I know, right?

Mishmashing both ways

In general, if one uses only one direction in a document (specifically online), the problems are not as noticeable, because the strongly typed text surrounds all other weak and implicit-level character types, making them its own type by default.

The issues come up when we have to mix languages and directions, or use RTL language inside a block that is meant for LTR. This happens a lot online—if there is no explicit dir="rtl" anywhere in the HTML document, the document defaults to LTR directionality. The directionality of the page (either by using dir='rtl' or dir='ltr' or not using dir= attribute at all and relying on it's default fallback to 'LTR') is considered to explicitly set the directionality of the expected text. So, any characters of ambiguous directionality will take on the direction that was set by that attribute.

If, say, I try to type an RTL language inside a textbox in a page that has dir='ltr', I can run into a lot of annoying problems with punctuation, the positions of segments of the sentence, and mixing languages of a strong type. The same happens the other way around, if I try to type an LTR language (say, English) inside an RTL-set textbox.

It can get so confusing, that, quite often, as I try to figure out how to type LTR text into an RTL box and see how my text actually organizes itself, my state of mind is pretty much blown.

The good, the bad, and the ugly

So, obviously, the creation of Unicode was much superior to the reverse-typing (and the need to use multiple individual fonts) that existed before it. Browsers tend to follow the Unicode rules (though apps that do their own rendering sometimes don't, but that's a different issue.) And this Unicode directionality algorithm gives us a lot of really Good Things to work with when typing different directions, but it also has Bad Things, and occasionally, even really Ugly Things.

Good things

There are, indeed, a bunch of good things that happen due to Unicode's bidirectionality algorithm. As I've already mentioned, the fact RTL users can type their language normally (and not backwards) is already a good thing (and I know from experience because I used the system when it didn't have that nice feature.)

Other benefits of the bidirectionality algorithm is the fact we can use numbers (which are weakly typed LTR) inside RTL text. So, for instance, consider this text:

ניפגש ב09:35 בחוף הים

Literally, this means "we will meet at 09:35 at the beach." Notice, though, that even without any directionality fixes, the numbers 09 and 35 are left-to-right as they should be, because that's how numbers are read—but I didn't really need to manually reverse my typing when I wrote this sentence. The browser did it for me.

Here's a nice exercise, though. Select that sentence. When you do, you can see exactly what piece has what directionality. Which leads me to ...

Bad things

Selections

Selections are a major part of the problem of bidirectional text. As you can see from the example of the "good thing" (that I don't need to reverse typing), there is also a bad side, which is how to select my text. Selection can be logical or visual. This is also true of cursor movement, which I will go over in a second.

Visual selection is simply that—visual—which means that the selection treats the segment of text as if it's one continuous block, regardless of directions.

Logical selection means the text is divided to its bidirectional pieces. That means that if I start my selection at the beginning of an RTL text (at the right), and drag my mouse toward its end (to the left), the selection will split when I reach the number part, because the numbers are left-to-right.

This is, indeed, logical, because it goes from logical start to logical end, and because the text is bidirectional, those two pointers are different for each of the sections. This makes a lot of sense, but it can be confusing.

Cursor Movement

Similarly, the cursor can also move either logically or visually. This can be a little confusing, and sometimes this behavior is inconsistent across platforms. Most of the time, though, the movement is logical.

So, here's a quick test of where this behavior can become really weird. Consider the following sentence. It is inside a textbox. so you can select it and move your cursor within it properly.

Or the reverse:

(Hat tip to Amir Aharoni for this one)

Let's go over what goes on in that horrific textbox for a minute. First of all, part of the problem in the first textbox is that the textbox was forced RTL, and because most of the text in it was English, it broke in weird places. Here's the sentence when it is forced to be LTR:

Notice, though, that the textbox problem also happened just the same in the reverse case, where the box was LTR and the sentence was mostly RTL.

With a forced-RTL textbox (and majority of text strongly typed for LTR) the spaces took the directionality of the text they were surrounded by, which is LTR. Then we had a strongly-typed RTL word in Hebrew, which made the space inside it turn RTL, but the surrounding white space (the one between the RTL word and LTR sentence) was still affected by the surrounding text, which is LTR.

If you're still with me here, this may help drive the point home. Essentially, you had this:

The entire sentence structure was right-to-left, but the small English segment was left-to-right. Overall "chunk" direction was RTL. Each chunk had its own internal direction. When you read it, it looks all jumbled—because it is.

And that happened exactly the same (only in reverse) in the second textbox. With LTR instead of RTL, and vice versa.

I know. I... know.

Ugly things

Now we move to the ugly area, the things that are not just difficult behavior, but are also producing visually different results. Remember those weak typed and implicit-level types? That's where these come in, and they, I tell you, they have a blast confusing us thoroughly.

White spaces

White spaces are implicit-level types, which means they are defined by the text they live in. The spaces in the sentence you're reading right now are implicitly LTR, because they are inside an English text. The white spaces here: במשפט הזה יש רווחים ואלה מוגדרים ימין לשמאל Are implicitly RTL because they're inside Hebrew, even though the page itself is LTR.

This is good, but it also produces some weird results. Consider the situation in which I have a set of numbers inside a text. The numbers are separated by whitespace, and the whitespace is defined by the surrounding text. But numbers themselves are "weak" typed, which means they do not affect their own surroundings (even though they are internally LTR). The whitespaces would have to take their directionality from whatever words surround the entire segment of numbers.

This sounds weird? The behavior is even weirder. See this, for instance:

I purposefully encapsulated those numbers in an LTR text, and so the whitespaces that separate these are still LTR. What do you think would happen, though, if I replace those English words with Hebrew (RTL) ones? Well, this example is exactly the same sentence and sequence of numbers, in the same exact order, with the single difference that "Start" and "End" were replaced by their prospective Hebrew words.

The numbers are reversed! The numbers... are... Head spinning yet? This might be weird, but it makes sense; the spaces are now encapsulated in an RTL text, which means they are now RTL. The space in RTL sentences is right-to-left, so the grouping of numbers go from the right and to the left.

But I think your head isn't spinning fast enough just yet. What would happen if we added spaces inside the number grouping itself? I mean, the numbers are internally LTR, but the space is RTL, so we will add a space to break the group and ... and the group will go spinning?

Try it. Add spaces to the number groups below.

See it? SEEEEEEEEEE it?

Yeah. Exactly.

Parentheses and Brackets

As I discussed earlier in this post, brackets and parentheses are, in fact, representing "start-of" and "end-of," which means that depending on where they are inserted, they may appear on different directions on your screen.

So, if I press the button that has a nice little [ on it on my keyboard (below the { and near the P), I will get different results in LTR and RTL.

This means that this code:

LTR:
<span dir="ltr">[</span>
RTL:
<span dir="rtl">[</span>

Becomes this: LTR: [ RTL: [ Yes, I clicked the same button. Yes, I'm sure. You're welcome to go over the source.

More than being a weird thing, this effect makes it incredibly frustrating when, inside an RTL textbox, there's a need to add some html <tags>. And, yes, this happens in Wikipedia, and in the RTL Wikipedias too.

Try adding a <span style="font-size: 2em"> to some segment of the text below. Good luck, stay sane, and remember to breathe. If you feel especially adventurous, you could also try to insert some wikitext, like a link to a page "Somewhere" (English link) with a Hebrew caption.

Want to go even wilder? Add some English text after the Hebrew one, and try to set some <a href="https://opensource.com/something.html"> </a> starting from the Hebrew string, and ending at the English one.

Type it all, don't cheat and copy/paste. Try it for real. Go ahead, play. Experiment. Go RTL crazy.

Online text editors and Wikimedia's VisualEditor

Now that we've gone through a slew of horrible interesting challenges with working with Right-to-Left text online, we can see how those can affect the development work of an online text editor. In the Wikimedia Foundation, we've been working on VisualEditor—a WYSIWYG system for editing Wikipedia articles. Not only does it work with converting HTML to Wikipedia's "wikitext" syntax, but it also has to handle multiple languages in multiple directionalities, platforms, browsers and localization environments. Basically, we need to support all the cases we discussed above, and then some. How hard can that be?

As a text editor, VisualEditor expects users to type into it, and that they do. They also do that in multiple languages, and, more often than not, in mixed languages inside the same article. Mixing languages is extremely common, especially in Wikipedia, when there's a need to provide the original script of a word taken from another language, or a city name in its native script, etc.

But as we saw, typing can be tricky, especially when we mix directions. We have to make sure we allow the users to type while seeing the result they will get in the page logically. We also have to make sure that their typing makes sense, and that if there is a need to describe a specific span of text as a different direction, they can do that easily. We have to make sure their input is interpreted correctly, that RTL appears properly in the ContentEditable screen, and then renders properly in the article that is saved.

Also, as you can see from my above example with the [ character, there's a difference between the HTML code and the resulting rendering. That is, I typed [ but got ] and [ appeared in the code, but ] appeared in my resulting rendered markup. Which should happen inside VisualEditor? WYSIWYG is quite different when what you type is expected to be flipped.

These things aren't impossible to deal with, but they are quite challenging and they often require decision-making about what a user should expect. Most applications online (and offline) have problems dealing with LTR/RTL typing, making these strategic decisions even more complicated. The behavior needs to be designed according to what we think is the best way to do it, and not what the RTL users expect, because as you can see from the current behavior, RTL users usually expect horrendous behavior.

It's the good kind of challenge, though. The kind a lot of people care about finding a good way to fix.

But wait, there's more

There's a bunch of other issues with bidirectional text, some of which are problems that exist in published software and apps online and make RTL'er's lives rather annoying. I may write about that at some point, and share my RTL frustration. If you're interested in how those challenges translate to everyday life, you can also visit http://rtl.wtf and witness for yourself what RTL users experience online regularly.

In this article, I went over issues with RTL strings inside LTR boxes, problems with characters of ambiguous directionality, with selections and cursor movements and general "huh"isms. There are, of course, more RTL hardships, but this post was meant to serve as a sort of introduction to the main and most common bidirectionality issues.

I hope you've enjoyed it. At least, I hope you now understand what the programmers (and RTL users!) must deal with.

Sidebars:

Languages and scripts

In this article, I use the term "Language" to refer to English and Hebrew letters. In fact, I should be using the term "Script" to refer to the letters and characters themselves. The difference comes mostly from the fact that, although Hebrew and English are languages, they each use characters that may be used in other languages. For instance, English uses Latin script, and Hebrew script can be used in Yiddish language as well.

So, take into account that this is the case, and that the actual letters that are used and are LTR or RTL are really "script" and not quite the language, because the browser doesn't really care what words you literally type using these scripts.

For the sake of simplicity and to try and reduce confusion, however, I made a tactical decision to group it all up to the most familiar terminology of "Language." (Thanks MatmaRex for pointing out I should at least mention this difference.)

Useful links

• Unicode Bidirectional Algorithm: http://www.unicode.org/reports/tr9/
• Bidirectional Text Requirements for Visual Editor (by Amir Aharoni) https://www.mediawiki.org/wiki/Visual_editor/Bidirectional_text_requirements
• VisualEditor: Support for right-to-left / bidirectional content Tracking bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=33126
• Blast from the 90s #1: Hebrew/English web fonts for Windows 3.1
• Blast from the 90s #2: Tools to write and read Hebrew online