ダニエルの冒険

Exploring Ginza at Twilight.

Seen in Loft, Lazona Kawasaki.

In Japan they love using charts in the mornings. They assemble the charts while describing a piece of news. The most interesting news is always about famous couple’s relationship troubles.

“Iris” was on the news yesterday.

Here we see her beside her “TOY BOY” with 不倫 (“adultery”) in a heart in the middle, along with ４０歳差 (“40 year age gap”).

And here we see that she’s married with 3 children to Peter Robinson.

The presenter reminds us that they have been married for 40 years (結婚４０周年) and that they are an “ideal couple” (理想の夫婦).

Finally we are presented with the “secrets” (秘) recently discovered: a confession of unfaithfulness March last year, and a suicide attempt.

I finally got a phone sorted. I wanted to use my Irish iPhone but the mobile companies here weren’t having any of that so I ended up buying a new one. The girl in the SoftBank shop was incredibly helpful and after a lot of various obstacles owing to my non-Japaneseness, I finally ended up with a new iPhone. It’s a 16GB 3GS, like my Irish one, but I got it for free. Normally you either pay for the phone up front, or pay for it bit by bit over 12 or 24 months. In their current “iPhone for everybody” campaign, you pay for it every month over 24 months (1920 yen/mo I think), but they also give you a 1920 yen discount every month, so it ends up being free. However, I paid for the iPhone up front and will get that money back through the discount over the next two years.

I ran into quite a bit of bureaucracy when trying to get it sorted. For example, if you don’t have 90 days on your visa you can’t enter the contract, and I had 87 when signing up because I had been here for 3 days already and the tourist visa is 90 days. This means that if I had arrived and immediately done my alien registration, then hurried along to the phone shop, I could have done it without any trouble. In the end I needed Satomi’s dad to put his name down on my contract to cover them in case I skip the country.

It’s a relief to have constant internet access back, and since then I have filled up my schedule until Monday! SoftBank gave me about 4 or 5 different passwords for different things when I signed up and I’m having trouble figuring out which ones go with which services, but soon I should have it all figured out.

The journey to Japan started off really well on the 7th when I unexpectedly met my Japanese teacher on the train. I had wanted to meet with her before leaving but I never had the chance, so I was really delighted to have the chance to catch up before I left. When I arrived at the airport things suddenly got a lot worse: the flight to London was cancelled, and then the later flight was delayed. I ended up missing my connection and sleeping in a corridor in Heathrow, which is just as awful as it sounds but at least I made an interesting friend who kept me company. Bob was on his way to Thailand, heading back to keep his farm running and to look after his go-go bar in Bangkok. He is a jiujitsu grand-master, has professional bodyguard experience, and is full of amazing stories. I think Satomi’s quite sick of hearing about Bob, though.

I finally arrived in Narita, to much warmer weather than Ireland/London—about 10°C with a little bit of sun.

Then it was off to Akihabara where I picked up a dual SATA to USB adapter for my hard drives (36€… it was about 115€ in Maplin in Limerick), introduced Satomi to the world of maid cafés, and had some dinner.

Yesterday we went to Kawasaki Daishi for Hatsumoude (初詣: the first shrine visit of the year). There were dozens of stalls selling various hot food and sweets prepared right in front of you. Everything you could want was right there, including sugar-glazed strawberries and apples, chocolate-covered bananas, octopus balls, Japanese “pancakes” (okonomiyaki.. not at all like a pancake really) in Osaka and Hiroshima style, hot dogs, candy floss, and lots of things that would take a lot of prose to explain but is very delicious!

They even had Fried Pikachu!

After passing through all the stalls, everyone was corralled into a narrow residential street and waited to enter the shrine.

Everyone was very orderly and quiet so even waiting there for about 30 minutes didn’t feel stressful or tiring. There were lots of nice decorations to look at around the shrine too.

Finally we got to the entrance of the shrine and got into a second queue.

The shrine itself was very beautiful.

Fun was had by all, especially me and my tanuki friend.

I released a couple of new apps last night: Ultra JLPT and dessid.

Ultra JLPT

Homepage | App Store

Ultra JLPT is the Japanese language trainer/tester I blogged about writing recently. There are five different apps for sale: one for each level of the JLPT test (1, 2, 3, 4) and then one simply called “Ultra JLPT” which contains vocabulary for all levels.

Ultra JLPT is totally focused on JLPT. For example, if you are studying JLPT4 there is much vocabulary that uses kanji that’s not on the syllabus. Ultra JLPT only displays these words in Hiragana, but of course there is an option to display all kanji characters in JLPT4 words, even if they are not required for the exam.

Logs are recorded and can be used to track progress, vocabulary can be refined by its type (only display kanji words, or only katakana words, or only those two, etc), and tests and be enabled/disabled in any combination, so if you want to focus on kanji reading that’s no problem.

dessid

Homepage | App Store

There have been many occasions where I’ve gone to a friend’s house and they’ve told me they have a wireless network, but they have to go searching for the password and when they do find it, it’s a long string of hex characters. This app discovers the password for many Eircom wireless networks so the password can just be put in your pasteboard and pasted in when the iPhone/iPod touch asks you for the password.

The password is also displayed on the screen so if you have some other device you need to connect, the password can be taken down from the iPhone/iPod touch screen, saving you from having to look up the password on the router.

JLPT level 4 only has about 100 kanji and I wanted to check a database of vocabulary to see which entries are readable by people with JLPT4 and which need me to convert to a phonetic reading.

I first thought I’d do my standard regex check for hiragana and katakana, then compare the rest of the characters against an array of JLPT4 kanji.

I then realised I might as well just use a regex for the whole thing, and this monster was born:

^[ぁ-ゖ~ァ-ヶー一七万三上下中九二五人今休会何先入八六円出分前北十千午半南友口古右名四国土外多大天女子学安小少山川左年店後手新日時書月木本来東校母毎気水火父生男白百目社空立耳聞花行西見言話語読買足車週道金長間雨電食飲駅高魚]+$

Breaking it down:

• ぁ-ゖ: Hiragana
• ~: Handle prefixes/suffixes in the dictionary (such as “~枚” or “第~”)
• ァ-ヶー: Katakana
• The rest is the entire JLPT4 kanji list, in no particular order.

This year is the first year that the JLPT (Japanese Language Proficiency Test) will be run in Ireland. I’ve submitted my application form and now just have to wait until December. I searched the App Store for some apps that might help me brush up on my kanji and vocabulary for level 2, but I couldn’t find anything that worked the way I wanted it to. That left me with a great opportunity to make my own!

I’m using Core Data to store all the vocabulary. This allows me search through all the vocabulary somewhat quickly by specifying appropriate NSPredicates. An NSPredicate might look something like so

(isKanji == YES OR isKanjiHiragana == YES) AND level == 'JLPT 2'

This will give every JLPT level 2 word with a kanji character in it.

It has three different types of test, and seven tests including variations. The three tests are

• Reading input: you must type in the correct reading.
• Choose correct answer: you must select the correct answer from a list of potential answers.
• Hide answer: the answer is hidden under a button. This one is pretty much a flash-card.

These three become seven when you see that you can take a Choose Correct Answer test and ask the user to select the correct meaning of a word, select the correct reading of a word, or select the correct word for a particular meaning. This extends to other tests and we end up with seven. Three classes exist, but there are seven objects with various different instance variables determining how they behave.

There are various categories of vocabulary: Katakana words, Hirgana words, Kanji words, Kanji+Hiragana words, prefixes, suffixes, miscellaneous (mix) and untestable (do not have reading or meaning available).

There are four JLPT levels (for now), and each of these levels has the categories above.

Every time a particular test has to generate a question, it has to find a word in the database suitable for the particular test. For example, if it’s Reading Input, there’s no point in testing a katakana word because it will be phonetic to begin with. It only makes sense to pick words with kanji in them somewhere. This means choosing a random managed object (i.e., a random row) that matches a particular NSPredicate from Core Data.

It turns out that this is slow. When I’d load a test, the screen would freeze up for a couple of seconds until it had sifted through the 5k~ rows of the database with a complex predicate. There were a few things I did to get this going faster.

Cache query results for each test

The first place I ran to was to cache the array of possible questions/answers for each test in the test object. When a test queries Core Data for a list of objects matching a predicate, save that list for next time so it won’t have to do a search again. A test’s predicate will not change through-out its life so caching everything with no expiry defined is safe to do.

This meant that the second time a particular search ran, it was fast. The first time, however, it was just as slow as before. And with seven different tests, that meant the user would have to experience seven slow-downs.

Load all data into a shared array and search manually

Next I decided to try to centralise the slow-downs to one place. I loaded everything in the database into an array at startup, which takes a couple of seconds but no longer than it took for each individual test.

When a test would need to find a value matching its predicate, it would pull a random one from the shared array and check it against the predicate. It would keep doing this until it found one. Except, sometimes it didn’t ever find one. Or it would take a really really long time before it came across one that would work! If it was never going to find one, there was no way that it could know that… so this was starting to look like a dead end.

Preload all data

Trying to centralise the slow-down without using a big, dumb shared array of vocabulary meant that each test would just fetch its own set of vocabulary matching its predicate at application start-up. Of course, this took many times longer than the previous method… but I had a plan…

Use a disk cache

The plan was to use a disk cache so that the hit wasn’t repeated across application launches. I implemented this, adding NSCoding to my Core Data object, but it was slow. Really slow. Writing or reading the file to disk took many times longer than querying the database to generate that array for me dynamically.

This was going to require a bit more intelligence…

Shared memory cache

Now knowing that querying the database directly was faster than loading query results saved to disk, I went about creating a memory cache shared across the whole app. This is basically an NSMutableDictionary with some fancy-pants methods around it. When the database is queried with a particular predicate, that predicate is stored as a key in the dictionary and the results from the database as the key’s object.

This meant that if you ran queries (or “fetch requests”, if you will… still not 100% comfortable with Core Data terminology) from different objects in the application but with the same predicates, the first would take time but the others would return immediately.

This meant that out of my seven tests, only five had to query the database and the other two could just pull from the cache.

Eliminate redundancy in level checking

The predicates I was generating at first were a total mess. To check for all vocabulary in JLPT 1, it would look something like this:

level == 'JLPT 1' AND (
  isKatakana == YES OR
  isHiragana == YES OR
  isKanji == YES OR
  isKanjiHiragana == YES OR
  isPrefix == YES OR
  isSuffix == YES OR
  isMiscellaneous == YES OR
  isUntestable == YES
)

Out of all those properties beginning with “is”, every single object will have at least one set. Therefore, when the user has not disabled any category (i.e., checking for all vocabulary as above), we might as well just use

level == 'JLPT 1'

This meant that to check for everything in the database, rather than having that big mess above multiplied by 4 (as there are four levels), the following can be used instead:

level == 'JLPT 1' OR level == 'JLPT 2' OR level == 'JLPT 3' OR level == 'JLPT 4'

Right? Right… but wait a minute! When the user has not disabled any level the above can be simplified to:

TRUEPREDICATE

So when I was building a predicate to select everything in the database, it went from this

(level == 'JLPT 1' AND ( isKatakana == YES OR isHiragana == YES OR isKanji == YES OR isKanjiHiragana == YES OR isPrefix == YES OR isSuffix == YES OR isMiscellaneous == YES OR isUntestable == YES )) OR (level == 'JLPT 2' AND ( isKatakana == YES OR isHiragana == YES OR isKanji == YES OR isKanjiHiragana == YES OR isPrefix == YES OR isSuffix == YES OR isMiscellaneous == YES OR isUntestable == YES )) OR (level == 'JLPT 3' AND ( isKatakana == YES OR isHiragana == YES OR isKanji == YES OR isKanjiHiragana == YES OR isPrefix == YES OR isSuffix == YES OR isMiscellaneous == YES OR isUntestable == YES )) OR (level == 'JLPT 4' AND ( isKatakana == YES OR isHiragana == YES OR isKanji == YES OR isKanjiHiragana == YES OR isPrefix == YES OR isSuffix == YES OR isMiscellaneous == YES OR isUntestable == YES ))

to this

TRUEPREDICATE

This gave a ~300% speed increase. If a user deselects one category from each level, the speedup will be totally lost. However, I do not expect users to do this too much. The default settings have everything on, and I expect that users may wish to disable a full category (possibly the extremes?—too easy and/or too hard?).

Order operands by property name in logic operators

As I said earlier, by caching the query results against its predicate, I only had to run five of the seven queries. However, I noticed that a couple of these looked the same, except with the order of “sub-predicates” reversed. What I mean is that I saw

TEXT != nil AND reading != nil

together with

reading != nil AND TEXT != nil

These are obviously the same thing, logically, but as strings they are completely different. There is no way to prove predicates are logically equivalent on the iPhone so to solve this, I made sure that when I construct these predicates with logic operators such as AND and OR, I order the operands based on the property name’s alphabetic order. What that means is that either of the above will become

reading != nil AND TEXT != nil

This is because “reading” < “text”.

This reduced the number of queries from five down to three!

Skip != nil if the property is not optional

Finally, I noticed that many of my predicates contained

TEXT != nil

The text property is not “optional”. That’s the same as having a “not null” column in the normal database world. So why bother checking if it’s nil if it’s never going to be nil?

NSEntityDescription can provide an NSDictionary of NSPropertyDescriptions, which let you know if any property isOptional or not. I used this to eliminate these redundant tests.

And this brought the number of unique queries down from three to just two!!

Conclusions

By simplifying predicates to logically equivalent but more sensible versions where possible, I reduced loading time to a third with default settings.

By shuffling around operand ordering and removing redundant comparisons, less obviously logically-equivalent predicates were discovered: for the seven tests there is only actually two predicates required.

By using a shared memory cache there is the overhead of a few seconds at application startup, but this means that queries with equal (not equivalent) predicates are only run once. It’s up to the rest of the application to make sure that equivalent predicates end up being equal as much as possible.

Quite a productive few days in Dublin! In no particular order.. at all..

Got an IBM.. er.. Lenovo Thinkpad from IBM and presented my FYP at their Dublin location. Thanks, IBM! It was a great event!

Got shouted at by a PC World employee for saying that I didn’t need Norton and would be using AVG. I was told it was “crap” and I “think I’m safe but probably have trojans on my PC”. I know he definitely knew what he was talking about because he told me he “used to be an engineer”.

Downloaded iTunes 9! Thanks, Steve!

Got Muse’s new album! Thanks, Matt!

Visited my grandma and a few relatives. Had a nice stir fry. Thanks, Stan!

Never got lost! Thanks Google+Apple!

Had a very interesting lunch! Thanks, Gavin!

PC World machine didn’t work in the end. No thanks to you, PC World