Jay Freeman Comments on Why Cydia Reloads Data

Quote (Jay Freeman):
Cydia, unlike the App Store (or almost any similar service), stores the entire package catalog locally. This is why it can (quite quickly, in fact) render an insanely long table with all packages in it that you can just fling your way through, while the App Store shows you 25 packages at a time with a slow "load more" button.

However, this means that it actually has to keep that many packages locally, and has to keep their records up to date. From a "cold start", downloading the package catalog is something like three megabytes, compressed (totalled from BigBoss, ModMyi, and ZodTTD).

Downloading those large files, often at around the same time other people are downloading them, from centralized and not terribly well organized hosting, really sucks. This is made even worse by them often not following my recommendations on "diff indices" (causing you to have to redownload the entire catalogs randomly).

However, honestly, even that isn't a serious problem unless everyone is coordinating their downloads due to a jailbreak or something: the key issue is that the reason we have this distributed package catalog infrastructure is because people can and do use entirely-random totally-third-party repositories.

While BigBoss, ModMyi, etc. are setup with diff-indices, almost no other repositories in the ecosystem are. If you install another large repository, you are probably downloading that entire catalog every time it refreshes (which might be almost a megabyte of data, even compressed), from some likely sketchy/small server.

If that server is offline, it is an even more serious problem: TCP sucks over mobile protocols, and it is really difficult to just claim "ok, that server is down" rather than "that server is slow" (difficult to impossible even in the best of cases). This can make that step take nigh-unto insanely long as it is trying to painstakingly download these files.

One possible solution to this is offline index updates, but there has been serious pushback on those in the past (as many users overseas, and even now in the US, have limited and even overly expensive data plans; it is important to not use data when they don't consider it "safe").

However, those cause the other problem that almost all users of Cydia simply don't realize that it is an offline app: seriously none of the services people compare it to work like that (the closest I can think of is the much-more-rare PreWare, the Cydia-a-like for WebOS).

This means that when users read about something online that is "now available", they open Cydia and if it isn't available they tweet, e-mail, call, comment... whatever they can figure out to do with a phrase similar to "why is it not in my Cydia".

Users are honestly more willing to believe they don't have the right source installed (as to them there are so many and they think they should have all of them on their phone... there are users out there with over 50 sources installed... I can't imagine what their refresh time must be like) than that they need to refresh some weird offline database.

Unfortunately, moving "fully online" (where each interaction with the app takes 2 seconds of network time and you can only see 25 packages at a time; aka, exactly like the App Store, which I find due to this to be frustratingly and painfully slow in comparison to a loaded copy of Cydia) isn't an option. :(

I ran some statistics a few days ago, and it is in fact the vast minority of people who stick only to the default repositories. The fun of Cydia simply seems to be installing and finding interesting repositories, some of which are run by two people in a garage, and some of which are run by large companies (often with a single package in them).

So, we are left in a place where we are constantly looking at ways of producing hybrid models, or at least figuring out ways to improve the performance of people who use the recommended subset of repositories. Certainly, I hate it when default repositories go offline temporarily, and am working with them to improve (or even outright replace) their hosting.

In the meantime, though: Cydia is an offline database containing tens of thousands of entries that has the user-requirement of being able to search and interact with those tens of thousands of entries in sufficiently "real-time" that users can rapidly and smoothly scroll through all of the packages in the entire catalog, filtered for a category or sorted by some Unicode locale.

Given that set of constraints, I'd say that Cydia is doing a "bang-up job", considering that the slowest step (and certainly the primary one that users complain about on any device less than a couple years old) really and honestly is "downloading that catalog from a distributed and entirely unrelated-to-each-other set of sources ran by random users on the Internet, most of whom have no experience with hosting or operations". ;P