Wednesday, July 31, 2013

Zekira update, and looking for an advisor on cloud database security

We’ve been making a ton of progress on Zekira. I want to update you on what’s happening, and also ask if you can think of a potential advisor for us on cloud database security.


Zekira Progress

The feedback you gave us on our last beta clustered in two areas:

     --Although performance and stability were better, we were still seeing a troubling level of bugs and slowdowns.

     --Many people felt the app would be much more compelling if they were able to access their index quickly while on the go (via smartphones and tablets).

After chewing over the feedback, we decided the best way to address both themes was to switch our core database to Apache Solr.  It’s faster, more stable, supported by an active open source community, and ideally suited to host data in the cloud.

Fortunately, because of the way Zekira was written we could swap databases without starting over. There was still a lot to do, but the end result is a faster, more stable desktop application that can store your index online, where you can access it from mobile devices.

In our next beta, we’ll be shipping desktop applications for Windows and Mac OS as well as a client for Android devices.  Obviously, iOS will be a future target as well.

I don’t want to promise a release date for the upcoming beta because there are still a couple of issues we need to resolve. Which brings me to my next topic...


Do you know of a potential advisor on cloud database security?
Since we’re going to be hosting peoples’ information online, we want to be extremely thorough with our security architecture. We’re doing this work very carefully, but to be extra safe we’d like to also find an advisor who’s experienced in cloud database security. We’ll do the programming, but we’d like to get advice from someone who can help us make the right decisions and avoid making stupid mistakes. It should not be a big commitment of time.

In exchange we’d make this person an official company advisor, with a stock grant and of course the opportunity to get Zekira for free.

We’re working our own contacts on this issue, but I also wanted to post the question here. Many of the people following Zekira are in the tech industry. If that includes you, can you suggest anyone who might be interested in advising us?

If you have any ideas, please send me a message at michael(dot)mace(at)zekira(dot)com.

Sunday, April 28, 2013

Zekira progress report


I'm overdue to give you a progress update on Zekira. I apologize for the long silence; we've been sending updates to our beta testers, but I should have posted them here as well.

We distributed Zekira beta 8 in March. If you're on the beta list and didn't receive the notification, please ping me and we'll get you the information on it. The feedback so far on the beta, plus our own performance tests, showed us that we needed to make some improvements to the internal database in the app. We were getting occasional errors in searches, and performance sometimes wasn’t up to your (and our) expectations.

Those changes require some careful restructuring of Zekira’s internal database. While we have the database opened up, we’re also making changes that will let us take the app mobile more quickly. We’re making good progress on all of that work, but it’s the sort of thing you can’t rush.

The new version we’re working on won’t just be more stable; it will also do a much better job of ranking search results by relevance. We hope that will dramatically improve the quality of the results it gives you. We’re also adding some other interesting features that I’ll discuss as we complete them.

Thanks for your interest, and as always you can drop me a note if you have any questions.

Tuesday, November 20, 2012

Update - Adding data from the cloud: What are your top priorities?

We're making rapid progress on Zekira in two areas: fixing the remaining bugs that our testers found in beta five, and adding more data sources.  In this note I'll update you on our status, and I also have some questions about which cloud information you'd like added to your index.

Let's start with the cloud data.  The current beta of Zekira indexes information stored on your computer, but we've always planned to reach out to the cloud to index the information you have there as well. We've started implementing that in the next Zekira beta.  Twitter is coming first, followed closely by gMail.  If all goes well, the next beta of Zekira will index your Tweets and gMail messages, and perhaps some other cloud information as well.

Here's a little peek at what Twitter integration looks like.  Tweets are included in your message index.  You can click to find addresses and hashtags the same as you would any other word. 

   

We'd like your help in prioritizing our cloud features and the way they work; there are three questions below.  We'd be very grateful if you could read through the questions and share your thoughts.  You can post a comment here, or send an e-mail to contactus (at) zekira (dot) com

1. Here's a list of web information sources we can add to Zekira.  Which ones are most important to you?
   Twitter
   Facebook
   gMail
   Google calendar/contacts
   Google Docs
   IMAP e-mail
   Skype
   LinkedIn
   Other web e-mail (please specify)
   Something else?

Some web services generate an enormous volume of information. Please help us think about what you'd want to index from those sources.  Do you want everything, or will that be overwhelming?  Specifically...

2. If you're interested in Facebook integration, which types of information do you most want added to your index?  Private messages?  Status updates?  Wall posts?  Comments?  Photos?  Chat sessions? Do you want to index only the things you send, or also things you receive from friends?

3. If you're interested in Twitter integration, which types of messages would you like indexed?  Direct messages?  Tweets that mention you?  All tweets you send?  All tweets from everyone you follow, or tweets only from specific accounts?

Thanks in advance for your feedback.


Status update.  We're working in parallel on two things: the web services integration I mentioned above, and fixing issues that people identified in the current beta.  We're no longer hearing many concerns about the speed of indexing; if you're testing Zekira and still have worries about that, let me know.  For now we're focusing on improving the speed at which Zekira loads (a big index can take a long time to start up) and fixing a variety of bugs and UI tweaks that our testers identified.  Probably the most common comment we've received is that people are confused by the Connections slider, so we're revising that.

We deeply appreciate your support; you're helping to make Zekira a better product.

Monday, September 24, 2012

Hidden in Plain Sight: Zekira and the Battle for Facebook

A long and complicated essay was posted last week by Aaron Greenspan, one of the people who claim that Mark Zuckerberg stole the idea for Facebook from him (link).  Most of the essay is focused on Greenspan's vast reservoir of dislike for Zuckerberg.  I didn't read the whole thing; it's about as entertaining as listening to me complain about the stuff Microsoft did to Apple in the 1990s.

But what did stand out to me was the start of the essay, where Greenspan reports that just this month he found copies of IM conversations he had with Zuckerberg when Facebook was conceived in 2004.  Greenspan writes:

"It turns out that my Other folder contained some of the most important legal evidence regarding the origins of Facebook, Inc. that I had been trying to find for years, but without much success. I had already scoured the folders I created for the Student Entrepreneurship Council and come up with a number of the AOL Instant Messenger conversations that I had with Mark, but not all of them, and not the most revealing ones. I had also checked my "Aaron" folder, which had another sub-folder for Instant Messenger conversations, as well as the large hierarchy of folders containing all of Think Computer Corporation's data, which had even more conversations, but none with Mark. The conversations I remembered having, but could not recall the exact details of, were nowhere to be found, at least until September 12, 2012."

The files stayed lost for eight years because they were buried in a flood of other documents Greenspan had saved.  Most people who save documents end up with this problem.  You can't know what you'll need in the future, so you save everything.  But the more information you save, the more difficult it is to find anything when you need it.  It sounds like Greenspan was unusually rigorous in the way he organized his files, but no amount of careful organization can help you find something if you don't remember where you put it.

This is one of the problems that a context manager should solve.  In addition to helping you recall the context around current events (meetings you're going to, people you know, etc), it should help you go back in time and recall your context around past issues.  I think that if Greenspan had been using Zekira, he could have found those files in under a minute.  And who knows, maybe the legal wars over Facebook would have turned out differently.

I'm not saying you'll end up in a legal fight over the origins of a leading web property, but you never know.  What I can promise is that the more successful you get, the more likely it is that you'll desperately need to find some files or messages you saved, and Zekira will be able to help you.

Announcing Zekira Beta Five

I'm happy to report that the fifth beta version of Zekira has now been completed.  If you sponsored us on Indiegogo at the $35 level or higher, you should have already received an e-mail with downloading instructions for the beta.  If you didn't receive your e-mail, please send us a message right away.

We took a lot of time on this beta because we wanted to thoroughly optimize the performance of Zekira while it indexes your information.  That was the #1 request from testers.  Our engineers did a huge amount of difficult low-level work on the innards of the app, running enormous in-house tests to identify bottlenecks, and carefully tuning the code.  Mike spent much of the summer as Mr. Tester so the engineers could focus on the hard work.  The whole process took longer than we wanted, but the result is a much snappier indexing process in our tests, and we look forward to hearing what the beta testers think.

We also simplified the UI, added support for more file types, and of course squished a lot of bugs.  A partial summary of the changes is below.

I've learned over the years not to try to predict a specific release date while waiting for beta feedback.  We originally hoped to ship in summer, but all I'm comfortable committing to at the moment is that we'll ship version 1.0 this year.  Thanks for your patience, and we'll update you as we progress.


Changes in the latest beta

Support for more document types.  We added support for Excel files, Outlook messages saved outside of a PST (.msg files), an additional form of web file (.mht format), and RSS feeds stored by your e-mail system.  The index also now lists every document Zekira finds, even ones that it doesn't know how to read.  Those documents are indexed by name, date, and folder location.

We eliminated the Context list.  Many of you told us that the Context list, displayed on the right side of the main window, was confusing and cluttered up the screen.  Since you can now search for any word or date by clicking on it, the list is redundant.  So we eliminated it. 

If you want to reduce the size of your index, or speed up the indexing process, you can specify a cutoff date for how far back in time you want information indexed.  Items older than that date will be ignored.

We made a huge number of bug fixes.  One highlight: Zekira now handles Outlook PST files more robustly.


These changes are in addition to the new features we added in beta four:

Everything's clickable.  When you're looking at a document, meeting record, or other item in Zekira, you can click on any word, name, or date in it to do an instant search for it.  This makes it much easier to follow the connections between items.

Narrow or broaden a search with a single click.  If you click on an item like a name, meeting, or document to see other things related to it, you'll sometimes find that you get either far too many matches, or too few.  You can use a new control, the Connections slider, to adjust your search.  Slide it to the left to show only the closest matches; slide it to the right to do a broader search.

Filter by folder.  We've added a new filter that lets you look for items that were saved in a particular folder or set of folders. 

We added a Query window.  This is a small window that you can leave open on your desktop to get info on your next meeting, and do quick queries on a word or name.

We simplified the setup process so you don't have to fuss with choosing folders to index or look for the location of your Outlook or AppleMail files.  


Add up all the changes, and this is a very different app from the one we previewed at the beginning of summer.  That means we need to go back and re-create the preview videos on the Zekira website.  It's a nice problem to have -- it means we're making progress.

Wednesday, August 8, 2012

Zekira Update: Can You Spare a PST?


We're continuing to make very good progress on Zekira, squashing bugs found by our beta testers and adding features you asked for.  We've now implemented the features I talked about in our last update, including click-anywhere searching and filtering by folder.  I think they make Zekira easier to use and more powerful. 

We're also making steady progress on increasing the speed at which we index your files.  Although the performance has improved a lot, it's not yet all the way to where we want it to be.  So this is our number one goal, and we're holding off on the next beta release until we meet it.  There are still a lot of things in indexing that we can tune, so I'm very confident that we'll hit our goal.

Meanwhile, we're learning some interesting things about file compatibility on personal computers.  Although there are many commercial and open source software projects that claim to be able to read various file types, in practice many of them are riddled with bugs.   Even something that you'd expect to be simple and well-defined, like reading a PDF file, can be a minefield of incompatibilities. 

Apparently many of the disk indexing products out there handle this problem by just skipping the files that they have trouble reading, and then not telling you about it.  This makes them look compatible, but you may not be able to find what you need, when you need it.  Unfortunately, there will always be cases of damaged files that can't be read and must be skipped, but in the Zekira project we're investing heavily in fixing as many incompatibilities as we can, and I think we'll have a very high level of compatibility as a result.

As part of this compatibility testing, we'd like to create a collection of Outlook .PST files that we can use in testing.  We have many of our own, but the more we can get, the better.  We know many people are not comfortable sharing .PST files because they contain personal information, and we respect that.  We've set up a procedure under which your file can be accessed by only one of us (Steven Glass), and he will make sure that the file never gets to anyone else.

If you have a .PST file or files that you're willing to share, please contact me at michael.mace@zekira.com .  I'll put you in touch with Steven so you can send the files to him directly.

Thanks for your support.

Thursday, June 21, 2012

Zekira Update: Successful Fundraising and New Features

Thanks again to all of you who contributed to our crowdfunding campaign.  We beat our goal, which is not an easy thing for a productivity app considering that crowdfunding is usually used for games and hardware accessories.  But most importantly, we found some really enthusiastic beta users who have already provided us with great feedback and helped us prioritize what we’ll improve first.


Features just added and available soon

In response to your requests, we’ve made a number of improvements.  We are running them though a test cycle now and will be posting Beta 4 with these changes in the next week or so.  We've made it much easier to quickly have Zekira do various sorts of sophisticated searches.  The changes include:

Folder-based filtering. We added a fifth filter to Zekira to help you when you remember that the thing you're looking for came from a specific place on your hard drive or a mail folder.



In addition to filtering the search by folder, you can now right-click on an item to search for everything else that was stored in the same folder as that item.  This is helpful when you suspect that something relevant is stored in the same place as an item you've found.

As in all the Zekira filters, you can combine this with other filters, such as date and time, keywords, etc.

Click anything to search. When you click on an item, we put information about the item and the item’s contents in the blue area at the top of the results window.  The main body of text scrolls so you can quickly see it all without opening the original.  Sometimes you'll see a name word that you'd like to use to start a new search, or add to the current one.


You can now do that immediately by clicking on the word.  Everything in the items displayed is clickable.  Exactly what happens will depend on what you click, and what other keys are held down when you click:

If you click on a word in the body of the item, Zekira initiates a new search for that word. 

If you shift click on a word, Zekira adds the word to the keyword filter with an OR. 

If you option click, we’ll add it with an AND.

Clicking on the metadata (any text outside the scrolling box) will also initiate new searches.  Shift clicking will add the search to the current collection of filters.

Click on... Action Action with shift
Title Initiates new search using word clicked on Adds word to keyword filter panel (joined by OR)
Author, Sender or Meeting Organizer Initiates new search for person Adds person to people filter
Date Initiates new search using date clicked Adds date clicked to date filter
Path Initiates new search for everything in the folder that contains the item Adds folder to the folder filter


Improved Connections slider.  The connections slider now has 11 settings.  The higher the setting, the more items Zekira will find.  The lower the setting, the fewer items Zekira will find. 
Easily Unapply a Filter.  You can now remove a filter from the current search by clicking its breadcrumb (the dark gray icon at the top of the window).  This makes it easier to try alternative scenarios.  The forward and back buttons let you quickly move back and forth between different choices.

Easily Unapply a Filter.  You can now remove a filter from the current search by clicking its breadcrumb (the dark gray icon at the top of the window).  This makes it easier to try alternative scenarios.  The forward and back buttons let you quickly move back and forth between different choices.



Features we're working on

Indexing performance.  Not surprisingly, many of you have some really large archives of information.  Our initial beta releases could not index it fast enough.  We found that we could handle about 40,000 items overnight but that’s just a drop in the bucket for some users.  We made some improvements in the Beta 2 and Beta 3 release, but these are not enough.  This is our number one priority and will remain so until we meet our performance goals.

PST challenges.  While we’ve been successful indexing the context of most PST files, a number of you have files that we’ve not been able to index.  Most of these are older PST files from older versions of outlook.  We found and fixed a couple of errors that stopped the indexing process, but there are still some files we can’t read.  We have more homework to do here as we try to figure out what’s going wrong.  In the meantime, as a work-around, you may try opening troublesome PST files in Outlook and copying their contents to a newly created PST.  We have yet to find a PST file that could not be fixed with this process. 

This problem is difficult to diagnose because we don't have any of these un-indexable files in our hands so that we can pick them apart.  If you have troublesome PST files that you're are willing to share, we’ll be glad to add them to our testing matrix and see if we can figure out how to index them.  We will take special processes to protect your personal info.  Send us a note at support@zekira.com to let us know if you can help.

European date formats.  In the spirit of getting the first product to market quickly, we took a few short-cuts around internationalization.  As such, our beta releases only support US date formats so far.  We’ve added support for European date formats to our “to do” list.

Indexing all files.  We got some feedback about files whose contents we do not yet read.  Currently we ignore those files, but several of you asked us to at least include the file in the index with its name and other metadata.  We have that working on a prototype now and will be releasing it in an upcoming beta.

Mac behavior.  We heard that you don’t like the “windows-style” menus in a window, when running Zekira on a Mac.  This was another one of those time to market trade-offs.  We’ve put it on our “to do” list.

Mac data types.  Our initial indexing support on the Mac includes Mac Mail the built-in calendar and the built-in contacts.  You’ve asked us to also support
•    Outlook on the Mac
•    Entourage on the Mac
•    Shared calendars on the Mac
These are all on our “to do” list.

Indexing your cloud data.   In the cloud era, you are more likely to have information stored at multiple locations and in multiple services.  We want Zekira to be the go to place for accessing all that information.  We’re looking to index your Google docs, Gmail, Google Calendar, Google Contacts, Google Drive, LinkedIn Account, Twitter and more.


More feedback welcome
The context engine is a new concept.  The Zekira team has spent a lot of time brainstorming ways it can and will be used.  But we don’t for a minute believe we’ve thought of everything.  We know that as more people use Zekira, they’ll think of other ways they want to try to get back to something they remember.  If you wish Zekira would let you do something it isn’t already doing, let us know.  We want your feedback.  You are helping to make Zekira better.