Skip to Content
Skip to Table of Contents

← Previous Article Next Article →

ATPM 9.01
January 2003

Columns

Segments

How To

Extras

Reviews

Download ATPM 9.01

Choose a format:

The Personal Computing Paradigm

by Michael Tsai, mtsai@atpm.com

E-Mail Archiving with Eudora and Mail.app

Why Archive

Back in November, Ken Gruberman wrote about how to slim down your Outlook Express or Entourage mail database. As the databases get large, the software slows down, the files become harder to back up, and the potential for data loss increases. One option, of course, is to delete mail older than a specified age. Another is to keep only a small amount of old messages that you think you may need in the future. My philosophy is that it’s not worth the time and chance for mistakes to pick and choose which mail to save; I just save everything. By moving, or archiving, old messages outside the program’s mail database, you can make your mail client zippy again while still keeping the old messages available if you need to search them some day. This article will teach you how to archive old mail compactly and in a way that you will be able to find old messages when you need them.

E-Mail Clients

Emailer, Entourage, Mailsmith, Outlook Express 5, and PowerMail are all clients that can benefit from e-mail archiving. All but Mailsmith store mail in a single database file. This means that they slow down when you have a lot of mail and that the database file is a single point of failure if you ever run into disk trouble. A single large file is hard to back up, and yet it’s all the more reason to backup frequently. If you use one of these clients you should definitely think about e-mail archiving.

Mailsmith, unlike the others, uses one database file for each mailbox. This makes it less susceptible to the above problems, but there are several reasons to keep its mail store from growing too large. First, the program slows down as you add more mailboxes, even if you aren’t viewing them. Second, Mailsmith sometimes modifies mailbox files that you aren’t using, so your incremental backup software will waste time and space backing them up even if they haven’t really changed. Third, its database files use about five times more disk space than other clients’.

Archival Methods

Entourage Email Archive and other AppleScript-based solutions purport to solve the above problems by saving your e-mail messages to one of the following:

  • One text file (in various formats) per message.
  • One text file (in various formats) per mailbox.
  • A FileMaker Pro database.

A lot of people like to do this, but I don’t find any of these solutions acceptable, because:

  • Archiving tens or hundreds of thousands of messages to individual text files will slim down your mail database but slow down your file system.
  • It’s not very convenient to browse messages stored in text files.
  • FileMaker databases can’t store more than about 64K per field, so long messages will be truncated. Also, FileMaker databases, in my experience, are slow and unreliable when they get to be very large.
  • With the above methods it’s easy to lose track of attachment files and which messages they were attached to.
  • The tools for searching text files and FileMaker databases are not optimized for searching e-mails.

I propose that it’s better to archive old mail into another e-mail client. Eudora and Mail.app are both available for free and both have many advantages when dealing with vast quantities of mail. There are many reasons why you might prefer to use another program for your day to day mail. However, the criteria that make a program good for downloading, reading, and composing messages are for the most part quite different from those that make a program good for storing and searching large quantities of messages. Eudora and Mail.app have much nicer interfaces than any FileMaker database I’ve seen. BBEdit may be good at searching folders of text files, but it can’t compete with programs that were designed for searching mail.

Importing Mail

Both Eudora and Mail.app have Import features, for bringing in messages from your primary e-mail client. Most other clients can export in mbox format, which both Eudora and Mail.app know how to read. The mbox format is standard and compact. It preserves attachments, but you will lose client-specific metadata such as message colors and the markers that show whether you’ve replied to or forwarded a message. Emailer doesn’t have a built-in mbox export feature, but Robert Shapiro has written an AppleScript to do the job. Entourage’s mbox export feature is hidden; you can drag a mail folder to the Finder to save it as an mbox file.

Where the Mail is Stored

Of course you will want to back up your e-mail archive, and to do that you’ll need to know where it’s stored. The current version of Eudora stores its data in the Eudora Folder inside your Documents folder. The actual messages are stored in the Mail Folder inside the Eudora Folder. Mail folders in Eudora correspond to folders inside the Mail Folder. If you like, you can replace the Mail Folder or any of the folders therein with an alias to another folder. In this way, you can store portions of your mail archive in separate places. For instance, I keep a separate folder outside Documents for mailbox files that I don’t want synchronized with my iBook (for lack of disk space).

Mail.app stores its data in the Mail folder of your Library folder. The actual messages are stored in the Mailbox folder inside the Mail folder. As with Eudora, mail folders in Mail.app correspond to Finder folders inside the Mailboxes folder. The alias trick also works with Mail.app except that rather than creating an alias you have to create a symbolic link (basically a Unix-style alias). The easiest way I’ve found to do this is to use Path Finder, but you can also do it using the ln -s command in Terminal.

How the Mail is Stored

Both Eudora and Mail.app store messages in the standard mbox format. This has the advantages of being compact and human-readable. That is, even if an mbox file gets corrupted you’ll still be able to read the intact parts by opening the file in BBEdit. Since the mbox format is standard, you’ll surely be able to find programs that can read it after your current Mac has been retired.

Eudora and Mail.app also store auxiliary information for each mailbox file, such as colors for the messages, the sort order, which messages you’ve replied to, and a “table of contents” so that they can display the list of messages without having to load the whole mbox file. What’s really neat is that both store this auxiliary information separate from the message data.

Eudora can either store it in the resource fork of the mbox file or in a separate, adjacent file. I recommend the latter, which you can enable by clicking “Use old-style ‘.toc’ files” in the Miscellaneous Settings. Using separate .toc files means that if you change the sort order of a mailbox or color one of the messages, Eudora won’t have to modify the mbox file that contains the messages themselves. Since this (much larger) file hasn’t been modified, your backup software won’t waste time or space backing it up again. Plus, if you want to save some disk space you can easily delete the .toc file without losing any essential information.

Mail.app stores each mailbox in a file package whose name ends with “.mbox.” A file package acts like a file, but it’s really a folder. You can see what’s inside by control-clicking in the Finder and choosing Show Package Contents. Inside the package is an Info.plist file, which holds the mailbox’s sort order, a “mbox” file that stores the message data (in mbox format), a table_of_contents file (much like a Eudora .toc file), and several mbox.SKindex files that store the indexing data that Mail.app uses to make searching faster. You can delete the index and table of contents files to save disk space, and they will automatically be re-created as needed.

Eudora and Mail.app scan their folders (Mail Folder and Mailboxes, respectively) to determine which mailboxes are available and how they are organized into subfolders. You can quit the mail program and re-arrange the mailboxes and sub-folders, and the changes will be reflected in the mail program. (Don’t do this if you have filter/rules set up, but if you use Eudora or Mail.app for archival only, this typically won’t be a problem.) You can move a mailbox file out of Mail Folder or Mailboxes if you want to store it elsewhere. To “re-attach” it, just drag it back into the appropriate folder and it will show up when you re-launch the program. (Additionally, with Eudora you can double-click a mailbox file outside of Mail Folder to open it directly.) If you’d rather save space without moving anything out of Mail Folder or Mailboxes, you can compress select mailbox files that you seldom use. Making them available again is as simple as decompressing the files and re-launching the mail program.

Reading Mail

Whether you prefer Eudora or Mail.app for viewing old mail is a matter of taste. Here are a few criteria that I considered. Mail.app lets you set separate fonts for the message list and the message contents (I recommend Osaka 9 and ProFont 9), while Eudora makes you use the same font for both. Both support two-pane browsing. Mail.app supports three-pane browsing (via the Mailboxes drawer), while Eudora has a separate Mailboxes window and a Mailbox menu. Both show the standard columns in mail list windows.

In Eudora, you can reverse the sort direction by option-clicking on a column header. In Mail.app, simply click the column header again (like in the Finder). Eudora lets you sort by multiple columns at once. For instance, to view messages grouped by subject and sorted (within each subject) by date, you can click on the Subject column and then shift-click on the Date column. To can option-click on part of a message in a message list to select all the messages that are similar to the part you clicked on. For instance, option-clicking in the Who column (not on the column header) will select all the messages sent by the person whose name you clicked on and group them together. Mail.app doesn’t support these fancy message list tricks, but it does have the option to color-code messages that are in the same thread as the selected message.

Eudora lets you label important messages in various colors. Mail.app only lets you mark them as flagged or unflagged, although you can use the color panel to temporarily color messages.

Eudora is much faster than Mail.app at mail browsing tasks such as switching between different mailboxes. It hardly seems to slow down at all as individual mailboxes grow larger, though there is a limit of about 32760 messages per mailbox. With Mail.app, on the other hand, there can be a long delay when you switch from one mailbox to another as the program loads the message list. You can reduce the delay by keeping fewer messages in each mailbox and by leaving mailboxes sorted by Number or Status, to reduce the time it takes Mail.app to sort the message list. You can click the Stop button when Mail.app starts “Updating color for messages.” Also, if you know that you will want to view the same mailbox again, you can leave its window open and make a new Viewer window for viewing other mailboxes; that way, you can avoid the delay as Mail.app reloads the first mailbox. Although Mail.app is much slower than Eudora in absolute terms, its performance is improving with each release and it has the advantage of being very well threaded. You can browse a mailbox, search in another window, and transfer messages from one mailbox to another, all while Mail.app is indexing yet another mailbox. If your Mac has multiple processors, Mail.app can take advantage of them.

Searching with Eudora

Eudora has a sophisticated search feature for finding messages. You can select one or more mailboxes to restrict the search to only those mailboxes, and you can perform more than one search at a time. The search window itself lets you specify multiple criteria. Each criterion can match against a field such as Body, From, or Date, and you can search for words, phrases, and regular expressions. You can require that Eudora find messages that match all the criteria, or ones that match any one criterion. The search options are almost as powerful as those in Mailsmith, and searching is fast even though Eudora doesn’t rely on content indexes.

pcp-eudora-search

Searching with Mail.app

Mail.app’s search features are not as powerful as Eudora’s, but they are more powerful than they appear at first glance. Like Eudora, you can select one or more mailboxes to restrict the search to those mailboxes, and you can perform more than once search at once by opening multiple viewer windows. The pop-up menu at the left of the search box lets you specify the type of search as well as whether to search the selected mailboxes or all mailboxes.

pcp-mail-search

An Entire Message search looks in the message headers and the message body. You type some words into the search box and Mail.app shows the matching messages, ordered by relevance. To get the most out of Entire Message searches, you need to know a little bit about how they work.

You are essentially searching by word. Mail.app will only find matches that begin at the start of a word. For instance, if you search for “str” you will find messages containing “structure” and “street” but not “astronomy.” An exception is that if a word contains capital letters; searching for “str” would find “MyStreet.” As you can see, search terms are case-insensitive.

Mail.app does not consider punctuation to be part of words. Searching for “example@domain.com” is equivalent to typing “example domain com.” Since punctuation is ignored, you can’t search for technical terms like “<h1>” or “$/.”

In iTunes, entering several words separated by spaces finds only those songs that match all the words. In Mail.app, the opposite is the case; the search will find messages that match any of the words. The message with the highest relevance will not necessarily contain all the words; instead, it might have high relevance because it contains many occurrences of one of the words.

If you want to find messages that match all the words, you can separate them with “and.” For instance, you could search for “example and domain and com.” Note that this will not restrict the results to messages that contain the three words in that order; they can appear anywhere in the message so long as all are present. There is no way to do a phrase search, i.e. find messages that contain a sequence of words like “Mary had a little lamb.” In addition to “and,” you can connect words with “or,” and you can group them with parentheses. Searching for “screen and (iMac or iBook)” would find messages that contain the word “screen” as well as either “iMac” or “iBook” (or both).

Sometimes Mail.app will get confused and an Entire Message search won’t find messages that it should. In this case, you can often fix the problem by rebuilding the index file that Mail.app uses for Entire Message searches. The Rebuild Mailbox command does not do this. Instead, you should make a new mailbox and move all the messages to it, or open the mailbox’s file package in the Finder and delete the mbox.SKIndex files.

Besides Entire Message searches, Mail.app offers To, From, and Subject searches. These searches work differently from Entire Message searches. They are not word-based, “and” and “or” have no special meaning, and spaces and punctuation are not ignored. This means that you can search for phrases.

Advanced Searching with Mail.app

Mail.app’s search features are quick and easy to use, but they are not as powerful as you might wish. Here are some workarounds for doing advanced searches in Mail.app.

If you know the mailbox that contains the message you are looking for, it may be easiest to import that mailbox into an e-mail client that has a better search feature. This is particularly easy with Mailsmith, as you can simply drag Mail.app’s .mbox file into the Mailsmith mailbox list.

An intriguing option, if you don’t mind pre-release software, is Steven Frank’s Emila. Unfortunately, the current version of Emila must import all your Mail.app mailboxes at once. This takes a long time and uses a lot of memory and disk space.

The Mail.app that comes with Mac OS X 10.2 has much improved rules that support multiple criteria. You can mimic many of Eudora’s complex searches by creating a rule that flags messages that match the criteria of your search. You can then select the messages you want to search, use the Apply Rules to Selection command, and then sort by the Flags column to see which messages matched. Of course, you will want to first disable any other rules that you have.

pcp-mail-rule

Mail.app’s rules don’t have a Date criterion, but you can restrict matches to a particular date range by writing an AppleScript that unflags messages outside the desired date range. (Or, if there aren’t many matches, you could sort by Date and find the messages you want by inspection.)

Eudora or Mail.app?

Both Eudora and Mail.app can handle large amounts of mail and store it compactly. Eudora is much faster and provides more powerful searching options. Mail.app’s iTunes-like search interface is easier to use for quick searches, and if you have trouble remembering the message you’re searching for you may find its relevance-ranked searches helpful. In my experience, Eudora sometimes parsed imported mbox files incorrectly, making it impossible to view (or search for) parts of certain messages. Also, it sometimes didn’t let me extract the attachments from imported messages. Most people probably won’t experience these troubles, but they caused me to switch to Mail.app. Overall, I think Eudora is better suited for e-mail archiving, but you can’t go wrong with either program.

Also in This Series

Reader Comments (13)

Mike Cohen · January 3, 2003 - 21:08 EST #1
I've used Eudora Pro for many years, but since I began using OS X, I quit using Eudora because I really hate the way windowing works under OS X. After using Entourage for a while and having a database corruption scare, I went to Mail.app in Jaguar and have all of my mail servers configured as IMAP to maintain mail on the server. I also use server-based folders extensively to organize my e-mail.

When I accumulate too much mail, I download it into Eudora for archiving. I find Eudora is, by far, the best & fastest at handling a large amount of eemail and can search faster than anything else.
Lee Bennett (ATPM Staff) · January 3, 2003 - 23:44 EST #2
Mike - allow me to call your attention to a little details since this once personally affected me. Make sure your host that is holding all your IMAP e-mail and folders has a solid backup policy in effect. A former host of mine had a hard drive crash within their e-mail server and it was at this time I discovered (to my profound dismay) that there were only weekly backups, and it had been six days since the last run!! Weekly!!?? The messages in my inbox seldom resemble the messages that were in there a week ago!

Officially, I changed hosts for one that had more storage for the same money, plus gave me PHP access so I can someday start blogging. But that backup policy was a major UNofficial reason!

Demand daily backups or, at the very worst, every other day, or find a new host.
Tom E. M. · January 4, 2003 - 21:11 EST #3
Very interesting article of great interest, since I'm archiving about 5,000-10,000 messages per year. It makes me feel better about my recent decision to dump Entourage in favor of Mail.app, but what about all my years of old Entourage or Outlook e-mail which I have peeled off to backup once per year? I don't think I want to import it all into a single disk area, but clearly I do want to get it into mbox format.

Perhaps a followup article could deal with long term archiving of huge amounts of e-mail from Mail.app and Eudora. My mail pile gets to several hundred MB per year. I can keep the individual folders from getting too large by making more folders, but eventually there are too many folders to deal with. I can move older folders into groupings of folders or out to archive disks and then just bring them back as needed, apparently. Perhaps I'll be able to do this without arbitrary annual boundaries now

Any thoughts on long term archival strategies would be appreciated in a future article.
Michael Tsai (ATPM Staff) · January 4, 2003 - 22:40 EST #4
Tom E.M.: If you have the disk space, I see no reason not to import it all into a single area. I've put 9 GB of mbox files in Eudora, and then in Mail.app, and didn't have any trouble. Like I said above, Mail.app takes a while to open large mailboxes (say, more than 5,000 messages), but once open the speed is fine.

How your organize your mail is up to you. Aside from the mailbox size issue, it really makes no difference as far as Eudora or Mail.app is concerned. I archive my mail by topic. If there are too many messages in one topic for a single mailbox, I just make a second one and divide the messages according to age. When there are too many folders, I start nesting them to reduce the clutter.

Depending on your backup strategy, it might be easier to organize first by year and then by topic.

If you have specific questions, let me know, but I think the take-home message is that both of these apps are flexible enough to fit in with a variety of backup and archival strategies.
Angus · February 12, 2003 - 08:36 EST #5
I'm struggling with exactly this topic. I've been using Mail.app for a year with IMAP servers and it's gotten really, really slow.

I've archived my e-mail the old way. I had BSD Mail through a telnet session to my mail server write an archive file and now I've got those archives on my Mac.

I've installed Entourage, but I dont like it's tinsel and badly thought out user interface. I've also been using Mozilla since I have a PC and wanted a unified IMAP experience.

I haven't used Eudora for a long time (like 1993) so I'll give it a try again.
Tom E.M. · February 19, 2003 - 09:25 EST #6
I've done some archiving of old Entourage mail by dragging folders to the desktop to create Mbox files, with some success. I'm doing this first on my personal OS 9 Power Mac 6500, so I'm using Eudora for now. One problem cropped up immediately in that Eudora would not recognize the Mbox files unless I changed their Type and Creator to those Eudora uses. This may be less of a problem with Mail.app on OS X, but I haven't gotten that far yet.

The really nice upside is that folderfulls of Mboxes can be moved around and backed up at will or scanned with a text editor, if desired, in addition to being aliased into the appropriate folders for Eudora or Mail.app. So this really is an excellent archival method and a great illustration of the power of standard formats. Thanks so much for the article!

Tom E.M.
Another Angus · August 6, 2003 - 18:45 EST #7
Eudora is great. Its mbox format works for legacy Eudora mail, for the current Eudora versions, and likely for future implementations.

It's free in Sponsored mode, runs fast on any platform, is multiplatform, and you can import/export e-mails either way.

And, it syncs with Palm.

Also, when all hell breaks loose, you can human-read the data to retrieve that important bit of text or that lost e-mail address.

What more can a guy (or gal) want? :-)

PS - oh, and it's not owned by M$. Hopefully never!
Steve Dorst · July 14, 2005 - 18:57 EST #8
I was trying to transition from Eudora 6.2.1 to Mail 1.3.11. All mailboxes have imported fine, but every time I import the "Out" box (consisting of 18,000+ emails), they lose the "From," "Date Received," and "Time." It thinks that I am sending all the emails today!

Do you know how I can remedy this, since my Out box is a helpful resource to know what my brain's been doing the past four years! Thanks in advance! -Steve
Michael Tsai (ATPM Staff) · July 14, 2005 - 19:03 EST #9
You could try importing with another mail program, and then export from it to Mail.
Mikael Byström · August 16, 2005 - 13:10 EST #10
You say "Emailer doesn’t have a built-in mbox export feature, but Robert Shapiro has written an AppleScript to do the job. "

The scripts you link to only contain the Export to Eudora script and some export digest scripts, not one export mbox script as far as I can tell.
Michael Tsai (ATPM Staff) · August 16, 2005 - 13:27 EST #11
I think the Eudora one is the one you want.
Mikael Byström · October 26, 2005 - 20:47 EST #12
No, I specifically don't want Eudora format as that destroys international chars.....
Michael Tsai (ATPM Staff) · March 28, 2011 - 14:15 EST #13
With Apple Mail no longer using mbox format and Eudora having problems with international characters and attachments, none of the existing applications met my needs. So I developed my own e-mail archiving application called EagleFiler. It imports from most mail programs by selecting the desired messages and pressing a hotkey, stores mail in mbox format, and supports fast indexed searches (including attachment contents) and smart folders.

Add A Comment





 E-mail me new comments on this article