iTunes library revisit

I wonder if many will remember the small task I gave myself half a year ago? When I commenced that “project”, I projected 1.5 years for the entire project to be completed. Well, it is done; I had finished tagging/correcting/organizing/updating/revamping/re-ripping my music library ahead of schedule. It was frustrating at times, almost futile but I persevered. Today, I present my new iTunes library:

Go on, expand this thumbnail 😉

It took a lot of effort & perseverance, not to mention time. The irony of it all is now I literally have to search for non-western songs using the appropriate characters, i.e. I can’t easily locate my Japanese or Korean titles any more. Despite all the hassle, I’m pretty pleased with the end-result. The end of this task is succeeded by another frightening one: synchronizing my iPod to this new library. Frightening, as I have a lot of application-specific data on my iPod & I’m afraid sync-ing to a new library will wipe the iPod clean. Nonetheless, it had to be done (otherwise I clean up my library for?). So with much trepidation (& adequate backup of critical materials wherever possible), I proceed to connect my iPod to the new library for the first time. I shall share a few of my observations for anyone who likes to tinker with iTunes and iPod synchronization.

 

The Observations

I was perplexed why the new iTunes library didn't immediately try to overwrite my iPod with "emptiness" upon connection; then I realized I'd this option "unchecked" all this while. In fact, I'd advise everyone to leave this option unchecked to safe-guard against any unintended synchronization.

 

Connecting my iPod Touch to the new library does not initiate a sync automatically as I’d the “Prevent iDevice from syncing automatically” option unchecked all this while.

EDIT: I strongly advise to leave that option unchecked to prevent any “accidental” syncing. Do note however, leaving this option unchecked does not mean setting your iPod into “Manual management” mode; unchecking the option simply means a sync will not start automatically.

[ I was hoping the apps that are currently already on my iPod will not be wiped off, and then re-sync-ed afterwards. This is because once these apps are wiped, their application-specific data will be gone too. To prevent this, with my iPod connected, I right-clicked on its entry in iTunes “source list” & select “Transfer purchases“. Then I realized uploading 100+ ipa files from my iPod to my new library ain’t as speedy as I thought (even though it was done over an USB connection). I interrupted & just dragged all my 177 ipa files from my old library’s “Mobile Applications” folder, literally onto the “Apps” section (again in the iTunes “source list“). This imports those apps just fine & extremely speedily (since there’s no longer any uploading from the iPod involved; it’s simply copying the ipa files from my old library folder to my new library folder). So what will happen to the set of apps already on my iPod (associated & uploaded previously with my old library), since I didn’t “transfer any purchases”? Read on…

Amazingly, even though I’m using iTunes with a new library that hasn’t even sync-ed with my iPod once, it seems to be able to detect the sub-set of apps that I previously indicated (via my old library) I wished to sync to my iPod. Example: suppose with my old library, I’ve selected 10 apps out of the full 177 apps that I wish to sync onto my iPod Touch. Even though I haven’t sync my iPod to this new library once, it “knows” which 10 apps were previously picked to exist on the iPod. & the good news? When I select to “Sync apps” between the new library and my iPod, it does not wipe and then re-sync those apps that are already existing on the iPod. Which means all data preserved! Yeah! ]

NOTE: please read the 2 paragraphs in [ ] as a set, lest you miss out anything & your (applications) data get wiped. & I wish to iterate: the above experience relates to Mobile Applications only.

UPDATE: per my knowledge gained from iLounge forums, Apps (& Photos and PIM data) only will not have their “Sync checkboxes” checked by default when you connect an iPod to a new iTunes library. Hence the reason why the existing apps on my iPod are left untouched until I explicitly chosed to “Sync Apps” from the new library. Even then, when iTunes detects the apps that I wish to sync onto the iPod, were already on the iPod, it will not wipe them then re-upload, essentially preserving any application-specific data.

Source (look for the lines, “You’ll find that the settings for Apps, Podcasts, Photos and Info are left alone (unchecked) on the new computer, and that data will remain in place unless you specifically enable those options separately …”, & “That’s exactly it.  The warning uses the language “Erase and Sync” as that is the perception of the average user. The fact that existing content is left alone is really just to save time…“)

Another thing I learnt is that iTunes treat each type “certain groups” of content as an individual library between itself and the iPod. Example: after great success with the Apps syncing, the next thing to do will be to wipe all the existing music (with their outdated tags) off my iPod and re-upload songs from my new library onto the iPod. Naturally, since my iPod hasn’t sync music with this new library before, iTunes warned me in the rather cryptic manner, “All existing music on the iPod will be replaced with music from this iTunes library“. Yup, that’s one of the reason/warning why some folks realized sync-ing their iPod/iPhone to a new library causes all the existing songs on their iDevice to be wiped. But I’m more concerned about whether only the music on my iPod will be wiped and replaced, or will all existing data (which includes the apps that I’ve luckily preserved earlier on, my mail account settings, my notes, etc) be treated as part of one overarching “library” and be annihilated by this library replacement sync? Well, it turns out (as I’d pointed out earlier): music library is disparate from apps library, disparate from podcast library, disparate from photo library, etc. So while my music is refreshed/replaced, my apps (along with their data) are preserved. Again! Sweet!

Another very useful piece of information which I learnt from iLounge forums, is that sync decisions (unless you are doing Manual Management) made at the device’s tabs, are perpetuated at the content-group-level. Example: if you choose to sync Music with the library, your iPod music, movies, TV shows, ringtones, (audio & e-) books will be synced with the current library as well, but the sync settings that is in place for all other content types will be left “as is”/untouched. This is very important. As for which type of content are grouped together, basically it’s:

• Music, Movies, TV Shows, Books, Ringtones
• Apps
• Podcasts / iTunes U
• Photos
• Info — Contacts, Calendars, Bookmarks, Notes (unless you are syncing your PIM information via non-iTunes channels, e.g. MobileMe, Exchange, etc)

Source (look for the line, “If you want to look at them as “libraries” then yes. It’s broken down like this:“)

Each of these tabs (representing each type of media that can be synced to an iPod Touch/iPhone) were unchecked initially when I just connect my iPod to the new library. Checking any one of them will bring up the warning prompt, "This iPod is synced with another library. Selecting erase and sync will replace all the contents of this iPod with the contents of this library". Any "erase and sync" ala a 'replace', is local to the content group. As for which content are grouped together, it's mentioned in the paragraph above.

 

In case I wasn’t clear, please allow me to elaborate. In the scenario illustrated above, if I “check” the “Sync Apps” checkbox, an “erase and sync” warning will appear, & if I approve it, all the apps on the iPod (that are incoherent with my new library) will be replaced with the apps from my library later when I press the “Sync” button at the bottom-right of the iTunes application. Apps that are coherent with my new library but already exist on the iPod, will not be “re-loaded. Likewise, if I want to replace all podcasts on the iPod with the set of podcasts from my (new) iTunes library, I will check “Sync Podcasts” & select “erase and sync”. Remember 3 things, however:

1. The “replacement” (or erasing, for that matter) will not happen instantaneously when you click “Erase and sync“. This is because iTunes doesn’t know exactly what new set of media you wish to replace the iPod’s set with. Only after you select what playlists, apps, etc. you wish to put on the iPod and then click “Sync“, will the actual (erasing and) replacement commence. This point is erroneous, as per this source (look for the line, “This prompt appears only when“)
2. “Erase and sync” approvals are localized to each media type the content groupings except for “Music” and “Movies“. Any decisions you configured for music will be acted upon with the “Movies” segment as well. I think this is because Music and videos are treated as one library. UPDATE: please refer to content-grouping a few paragraphs above.
3. If you have authorized your current library & there are purchased content on your iPod, besides “Erase and sync“, there’ll be a “Transfer purchases” button which allow you to transfer purchased content from your iPod back to your current library. However, I’m not sure whether all purchased content (which means apps, songs, TV shows, etc) will be uploaded, or you will have to do it content-type by content-type. UPDATE: yup, I’m still unsure whether all types will be transferred back at once.

This is the exact sub-menu where you will import your OPML file. I'm surprised this isn't elaborated on Apple's iTunes site. Perhaps Mac veterans are acclimatised to dragging-&-dropping stuff from anywhere to everywhere.

Talking about podcast. I managed to import all podcast subscriptions into the new library without any hassle at all. I’d exported them as an OPML file previously, which is the export file type you want if you wish to “save” your subscriptions. Note though: podcasts refresh settings, how many episodes to retain, etc are controlled by your iTunes preferences and not part of this OPML file (I’d checked my OPML file for clues).

Last but not least (for this section), if you are the type of reader that desires even further details, breakdowns, elaborations, scenario explanations, you are in luck: please enjoy the back-&-forth between 1 of the administrators of iLounge forums and me. Trust me, it’s a solid read.

Source (huge thanks to jhollington of iLounge forums, for his/her relentless explanation)

Anything new (& not widely elaborated by tech blogs) with iTunes 10?

For the new “Album list” view, some folks are confused when album artwork for their albums with lesser than 5 songs do not show up in this view. To force the album artwork in this view to show up no matter how “complete” one’s album is, please tweak the setting as shown below:

 

Just in case you are wondering why some album artwork are displayed while others are hidden.

 

I also noticed with iTunes 10 (if you have been tracking this bug & it has been fixed by version 9.2 of iTunes, please notify me), the mobile applications (ipa files) will not be forced by iTunes to reside on the same drive as where the iTunes library database files are stored. This bug will be very familiar to those who store their media content in an external hard disk but have their database files retained on local hard disk. Previously, try as they might (to manually shift those ipa files to their external volume), the moment they update their mobile applications through iTunes, an (updated) copy of the ipa file will be stored locally. Now, even my ipa files are stored externally.

Smart Playlists

 

This light-coloured column on the left side of iTunes is commonly referred to as the "Source list". Near the bottom, one will observe the "feeder/component" smart playlists are further encapsulated into folders for cleaner organization.

 

As with my iTunes library, my approach to music appreciation on my iPod is revamped. At this moment, I’m using Smart Playlists to re-iterate through my song collection. Unlike a regular (more meaningfully referred to as a “static”) playlist, smart playlists generate their resultant list of songs dynamically using “conditions” the user set. I have been trying to create the ultimate smart playlist based on the best of what these sites can offer.

The tricky bit about conjuring one single ultimate from all these, is to ensure the criteria that each list is based on, wouldn't have semantic errors when all are combined.

 

Before I delve into the conditions, my ratings scheme are as follows:

• ★★★★★ – very rare will a song be issued this, unless it is superb, or holds special meaning to me.
• ★ star – my “fail” rating for a song; this song must explicitly piss me off to be awarded this.
• ★★ to ★★★★: so songs that “pass” can only be awarded these ratings. ★★ means a song is just bearable, 3 stars means its average while a ★★★★ title will deem it “quite good”. Again, it is very, very hard for a song to get ★★★★★

And here are the [smart] playlists:

UPDATE (11 Nov 2010): Apparently the way iTunes evaluate nested smart playlists, is counter-intuitive to conventional (boolean) logic. I realized something was going on when my master spl isn’t really updating/rotating/refreshing songs in its list. Posted my spl logic onto iLounge & 1 of the veterans were quick to point out iTunes’ flawed “logic” in this specific aspect of spl. The discussion & in-depth explanation can be viewed at this iLounge thread from this point onwards. Anyway, I’ve since corrected my master spl, as well as updated a few of my component spls to better reflect conventional “design”. In addition, the exclusion of “latest_additions” spl content (to prevent double weightage) has been surfaced from all rated component spls to the master spl level to remove “bad form/design” at the component spl-level. Updated versions of the respective spls are displayed directly below their old verions.

 

Short-list music that I have not rated. Purpose is to encourage me to rate my music, even songs that I'd played but didn't bother to rate.

Collects the music I acquired within the last 1 month gets uploaded, and yet not bore me too quickly by ensuring the same song do not get played twice within 3 days.

This is OUTDATED && INCORRECT. Allow me to give a 2nd listen to poorly-rated (less than 4 stars) that I actually like enough to not skip them more than 10 times. To elaborate, a 3- or 2-star song that I keep skipping will not be given any second chances.

 

The previous "rating < ★★★" condition isn't proper design, as it will inadvertently include unrated songs, which are already captured in another spl. The updated rating criteria implies that I'd explicitly rated these songs before. In addition, a 2 month cooling period ensures these "barely-there" tunes get a proper, unbiased re-evaluation.

This is OUTDATED && INCORRECT. Surface any music that I heard less than 10 times or haven't hear in half a year

 

"rarely_heard" spl has to be corrected to circumvent iTunes' flawed logic,

This is OUTDATED && INCORRECT. Super nice music that has been rated and not in latest_additions (if I leave out this part, a newly-added song that has been rated, may have double its weightage). Only songs that haven't been heard in a week can be played again. Besides this 5-star list, I also have 2-star to 4-star lists. Those other 3 lists are similar except songs in 2-star cannot repeat until 2 months later; songs in 3-star list cannot repeat till 1 month later; songs in 4-star list cannot repeat till 2 weeks later. In other words, more highly-rated songs are allowed to repeat more frequently (which makes sense, to me).

 

As mentioned in the "UPDATE (11 Nov 2010)", the exclusion of the "latest_additions" spl is surfaced to the master spl level as it is illogical (or more appropriately, "bad form"?) to have the exclusion at this level. This exclusion is propagated to all "rated" spls. When someone looks at a n★-list, the list should contain all the n★ songs. You may be eager to point out that if that's the case, shouldn't the "Last Played" criteria be surfaced to a higher-level list as well. Well, that is a future decision.

This middle-tier playlist is obsolete. Groups 2* to 5* songs into one sub-list for easier "integration" into master list later on

This is OUTDATED && INCORRECT. If I rate a song 1 star AND also skipped it more than 10 times, obviously I detest it.

 

Not only is this tweaked to account for iTunes’ flawed nested spl logic, but I have also updated the logic to be an “OR” instead of an “AND”. This is because I found that I very rarely 1-★ a song, which means too few songs fall into this list, hence the “OR” logic will enforce a more stringent level of quality in my final list.

This is OUTDATED && INCORRECT. Sync random picks of songs (up to 7GB) from the earlier sub-lists. Explicitly leave out detestable songs (those 1*)

 

 

As you can see, to make iTunes' flawed nested spl logic evaluate the way conventional wisdom dictates, one has to go about a very lengthy workaround. In addition, I learnt that the "Playlist IS/IS NOT" criteria can be directly applied to a folder of playlists; rendering my "rated_already" 2nd-tier playlist redundent.

Ultimately, this one single master list (sync_to_iDevice) is the only list of music I sync to my iPod. To put it succinctly, it will greatly encourage rating of songs while churning through the rated ones, & keep the hits fresh by not repeating them too often based on how much I like them. However, there’s a bug with this system that I’m unable to solve yet. For those who want to figure it out, don’t peek at the solution:

There is no guarantee that at any time, there will be songs from all those sub-lists synced to my iPod. E.g. it is possible (though very unlikely) that at any moment, there may be no songs from my latest additions on my iPod, etc.

 

Smart playlists are also useful for weeding out songs with incomplete tags, e.g. songs with missing artwork

 

However, I’m unable to use Smart playlists to isolate songs with missing lyrics (& other more advanced tags not recognized as SPL criteria). Then how? How will I know which songs are missing lyrics? Well, it’s nice to have a super powerful mp3 tagger that allows regex/boolean expressions to filter by all sorts of tags/criteria on-the-fly: the extremely powerful (in other words, verify the tagging options very carefully) MP3Tag.

 

For "proper/advanced" tagging purposes, can't go wrong with MP3Tag. Just need to refresh your tags within iTunes after you update/correct them externally.