[Shotwell] Enhansment to shotwell

Fri Apr 23 17:31:23 PDT 2010

Shotwell's not designed for that kind of scalability (yet!)  I'm more
concerned with 10,000 photo collections.

Right now I would want to design this correctly and cleanly, then do
performance testing to determine if there are bottlenecks and correct
them.  As I mentioned, we could design it so each Tag maintains its
photos and its childrens' photos, adding and removing them
dynamically.

-- Jim

On Fri, Apr 23, 2010 at 4:33 PM, Bengt Thuree <bengt at thuree.com> wrote:
> Will the performance be ok with a photo collection of 50,000 - 1,000,000
> Photos or above?
>
> Bengt
>
> On Fri, 2010-04-23 at 11:54 -0700, Jim Nelson wrote:
>> So far, this sounds like steps in the right direction.  Some notes:
>>
>> Our general strategy in Shotwell is to load the entire database in one
>> swoop at startup and maintain all database rows in memory.  Once the
>> application is started, database access is limited to updates and
>> deletes.  (There are exceptions, particularly in Event.)  Generally
>> it's faster to do searching in memory than via the database.  It's a
>> trade-off, but one we take for performance reasons.
>>
>> If you read the architecture overview on Data Structures, I talk about
>> the strategy I use for signalling (i.e. listeners, or observers as I
>> call them).  We definitely want each Tag to signal when it's children
>> and parent is altered.  I would also recommend having these signals
>> reflected by TagSourceCollection, that way an observer can register
>> once than register for every Tag.  For example, LibraryWindow could
>> register once and receive notifications on all Tags being moved
>> around, updating the Sidebar accordingly.
>>
>> Also note that we've invested a lot of code time in fully synchronized
>> data structures for the "big" objects and their collections: Tag,
>> Photo, Event, etc.  Since each Tag maintains a ViewCollection of Photo
>> objects, it's easy to recurse up and down the hierarchy gathering a
>> HashSet of Photos.  It could be as simple as this:
>>
>> void get_photos_and_children_photos(Gee.HashSet<LibraryPhoto> photos) {
>>     photos.add_all(get_view().get_all());
>>     foreach (Tag child in children)
>>         child.get_photos_and_children_photos(photos);
>> }
>>
>> Since I don't expect Tag hierarchies to be crazy-deep, this is
>> probably sound.  (There can be no loops in the tree, of course.
>> You'll need to protect against that.)
>>
>> Alternately, each Tag could monitor additions and deletions of its
>> children and maintain a separate ViewCollection of its Photos and its
>> descendants' photos.  I'm not convinced that there's a big win here,
>> but it's possible with the architecture in place.
>>
>> In short, I would discourage looking to the database for anything more
>> than data persistence and organization.  My feeling right now is, this
>> can all be done in memory faster and and more efficiently than via
>> SQL.
>>
>> -- Jim
>>
>> On Fri, Apr 23, 2010 at 5:20 AM, Ruslan A. Bondar <fsat at proton-sss.ru> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > Hello, Adam.
>> >
>> > I think the code change will contain following steps:
>> > 1. Add field "ParentTagId" to TagTable
>> > 2. Change the TagTable class to handle this field.
>> > 3. Change the find_parent_marker and add_tag_page methods in LibraryWindow to provide recursive tree creation for tag tree.
>> > 4. Change drag'n'drop handler to allow moving tags between branches.
>> >
>> > And something more, such as listeners of tag tree alteration etc. I think these changes will not affect any other functionality. And no major changes in design are required.
>> >
>> > I think it would be better to store only photos "owned" by this tag, but on load build the full list, containing photos from this and all child tags. And for now I see several ways:
>> > 1. Create helper table to speedup loading, but this will slow down tag reassigning (Tag moving between branches). This table may contain only 2 fields: TagId and AllTagPhotos.
>> > 2. Walk over the table recursively and build the list via concatenation, this will be slightly slower.
>> >
>> > These are the common thoughts. I'll dig the source deeper during the weekend.
>> >
>> > On Thu, 22 Apr 2010 19:25:51 -0700
>> > Adam Dingle <adam at yorba.org> wrote:
>> >
>> >> Ruslan,
>> >>
>> >> thanks for your email.  As you probably know, we're also interested
>> >> in hierarchical tagging.
>> >>
>> >> I think at this point we'd like to implement a single-parent system,
>> >> in which each tag can have multiple children but only one parent.
>> >> This is ticket 1401.  After that, we might consider extending the
>> >> system to support multiple parents in some way.
>> >>
>> >> A single-parent tag system in Shotwell would work like this.  In the
>> >> sidebar, the user should be able to drag any tag A onto any other tag
>> >> B; then A becomes a child of B.  If the user drags any tag onto the
>> >> Tags heading, then it becomes a top-level tag.
>> >>
>> >> If the user selects any tag, then Shotwell should display all photos
>> >> which belong to the tag or any of its children.  The set of photos is
>> >> sorted according to the View->Sort Photos settings.  If a photo
>> >> belongs to multiple child tags, it is displayed only once in the
>> >> sorted set.
>> >>
>> >> All tag names must be globally unique.  For example, I can't create a
>> >> tag named "washington" under a parent "states", and another tag also
>> >> named "washington" under a parent tag "cities".
>> >>
>> >> You said you might like to participate in coding this feature - that
>> >> would be great.  If the description above sounds reasonable to you,
>> >> as a next step maybe you could look at the Shotwell code and send a
>> >> brief design proposal to this mailing list.   In your proposal,
>> >> please describe the database schema changes you intend, plus any new
>> >> major classes introduced (if any) and any major changes to existing
>> >> classes. We'd be happy to comment on your implementation plan.
>> >> Thanks!
>> >>
>> >> adam
>> >>
>> >> Ruslan A. Bondar wrote:
>> >> > -----BEGIN PGP SIGNED MESSAGE-----
>> >> > Hash: SHA1
>> >> >
>> >> > Hello all.
>> >> >
>> >> > I have an idea, maybe similar to http://trac.yorba.org/ticket/1401,
>> >> > but I'll tell in more details.
>> >> >
>> >> > It would be great to allow organizing tags into graph.
>> >> > So one tag may have multiple parent and multiple children tags. And
>> >> > each of them may be assigned to the photo. It will be more flexible
>> >> > to organize your photos. As an owner of large, enough, photo album
>> >> > (~100 GB of photos) I can say - This feature is VERY useful.
>> >> >
>> >> > Also I'd like to participate in coding of this feature.
>> >> > -----BEGIN PGP SIGNATURE-----
>> >> > Version: GnuPG v2.0.14 (GNU/Linux)
>> >> >
>> >> > iEYEARECAAYFAkvNx+QACgkQRadsTlsSykPASgCcDBrgeobgyNweH2CLTaFt1xZu
>> >> > gqUAn3jfVCjMrDPuwQJQWvh3KBYqdq7q
>> >> > =4KsN
>> >> > -----END PGP SIGNATURE-----
>> >> > _______________________________________________
>> >> > Shotwell mailing list
>> >> > Shotwell at lists.yorba.org
>> >> > http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
>> >> >
>> >> >
>> >>
>> >
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v2.0.14 (GNU/Linux)
>> >
>> > iEYEARECAAYFAkvRkIYACgkQRadsTlsSykNC4gCeJ6XCVc9RzUwluOwj/lo960xg
>> > G30AnAvyDgsu+DK9raFT2LUZFOtf3APZ
>> > =LAuc
>> > -----END PGP SIGNATURE-----
>> > _______________________________________________
>> > Shotwell mailing list
>> > Shotwell at lists.yorba.org
>> > http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
>> >
>> _______________________________________________
>> Shotwell mailing list
>> Shotwell at lists.yorba.org
>> http://lists.yorba.org/cgi-bin/mailman/listinfo/shotwell
>>
>
>
>