The thing I made today is called a ‘lexer’. A ‘lexer’ is a thing in the study of computers that reads in stuff and breaks it up into words (often called ‘tokens’). When we write, the words just have spaces in between them…though you might want to recognize other things as well if you are trying to get a computer to understand what you mean.
This ‘lexer’ is written in ‘Ruby’. I hope it is simple enough for you to follow along even without knowing that much about ‘Ruby’, or even knowing that much about telling computers what to do. I have put comments in what I wrote (after the # marks) to help you out.
The ‘lexer’ looks at the first letter it sees and decides what to do. If it is a real letter, it keeps reading until it has seen the entire word, then saves that word. If it is a “ or ‘, it reads ahead until it sees the same kind of letter, and calls everything between the two a single word. (It takes the letters at each end, too.) If it is another interesting letter, the word is just the single letter. And if it is anything else (like, say, a number), it tells you that it is not able to make a word.
I wrote a ‘lexer’ because ‘lexers’ are a big part of ‘compilers’, the things that take things written by humans and make new things that tell computers what to do. The ‘lexer’ goes first: it takes in something you wrote and breaks it up into words so that the rest of the ‘compiler’ can start trying to understand it. (You would not try to understand a book by reading it letter by letter, right?)
And ‘compilers’ are what I work on for my day job; I work on the one called ‘Clang’. ‘Clang’ works with the ‘C’, ‘C’++, and ‘Objective_C’ kinds of computer talk. My job, for the most part, is to tell the ‘compiler’ to look for things you say that sound bad, and tell you that you probably did it wrong. It is making the ‘compiler’ understand what the things mean, rather than just what they do. I love it.
Inspired by the examples on the site, I tried describing an existing work of fiction, which some of you may recognize:
In this story, people live in a different world—actually, ten and two different worlds. A long time ago, the people made computer-people to do things for them. The computer-people were not happy, and fought with the normal people. The normal people did better, so the computer-people left for many years. The story starts when the computer-people come back.
At the start of the story, we learn that the computer-people can look like normal people now. Some of the computer-people have been hiding on the world. They pretended to be friends with normal people to learn how they fight. Once they learned, they were able to beat the normal people very quickly on all of their worlds. All of the cities were broken, or are now just gone. The fastest people got on space-cars to run away to space.
Everyone is scared. There are only about five ten hundreds of people left. Everyone who decided things is dead, so a person who decides things about teachers has to be the one who leads everyone now. There is only one space-car with guns, so all of the other ones have to stay close to be safe.
The story talks about the people on the big gun-car. One of them is the one who can tell all the people on the car what to do. Another is the man who helps the first man. The gun-car also has lots of little cars that can fly around to help it. The people who fly these cars are a woman who is angry a lot, a woman who likes the man who fixes the cars, and the son of the man who leads the people on the big gun-car. The man who fixes the cars and a woman who helps him are also part of the story.
Last there is a man who knows a lot, who says that he can tell if a person is actually a computer-person in hiding. The man who leads the big fight-car and the woman who leads all the people decide that he should make his thing that finds computer-people. But there’s a problem: he is always imagining a tall woman he met back on his world. We don’t know if she is real or not, but that woman was a computer-person!
And last I wrote this poem:
I enjoy words that say what they mean,
and words that are built up from pieces I know.
I sometimes enjoy using those not so simple
For pretty they are on their own.
If always I needed to use
the words that we use every day
I’d feel like losing my voice or my hearing
My brain wouldn’t know what to say.
(I got to use “brain”? Really?)
Even now I find myself thinking
Of all that may be in one’s voice
All of the words that people can speak
That to me and mine are just noise.
But even those stronger ten hundred are right
When they can together tell stories tonight.
…which just proves that grammar can get complicated even when the words are simple.
Shoutout to fellow NaCreSoMo participant Evelyn’s Sound Huntress as well, who’s living in a world with a different limitation. Thus ends the first week of NaCreSoMo 2014!