Lazy loading virtual pages#131

Currently we can feed Kirby “virtual pages” by overriding the children() function of a model: there we pack all data and content into an array and pass that to the Pages::factory() function. With that collection, the Kirby API (and therefore also the panel) can then work with “our virtual data”.

The problem arises when we have a huge dataset. This could come from an external API, a database, the fibonacci sequence, but really also a normal Kirby page with many children. When Kirby starts filtering and slicing the data, all data has already to be loaded (which could be a problem).

Delaying the loading of the data to the moment when they are actually used (like, in the panel or a template), would allow us to integrate data from external, big, sources and theoretically edit them in the panel.

One way to create such lazy loadig collections would be to separate out the actual filtering and slicing methods from the Collection classes. The collection classes would only be reponsible for “remembering” those filters, without actually executing them against a dataset. Another class (or closure) would then need to “interpret” those filters and translate them to what is adequate for the datasource (like SQL, Firestore, REST queries, etc…), but only when something wants to access the data. Most importantly, the implementation for those “other classes” can (and should) be left as an exercise for the community.
Even without databases, this could make certain situations with “stock Kirby” faster. Think about pages with lots of children: $page->children()->limit(5) doesn’t need to load all children (actually I don’t know if it currently does).

Lukas Kleinschmidt has created the resolve plugin to solve these kind of issues. I have no idea if this works with virtual pages though.

See: https://github.com/lukaskleinschmidt/kirby-resolve

I think you already can realize that. You could use something else, instead of overwriting the children method. For example, choose another method-name, create a Page object or reuse the current page-object and use this as the parent: in your blueprint, and then create Subpages on that object. We use this to create collections with 1,000 or more subpages and don’t have any speed problems so far.

@Bart Vandeputte The resolve plugin solves the problem of the router searching through all pages for the slug. However it does not address the problem of having to load everything when using the collections in a foreach loop, for example.

@Ao
Loading a 1000 pages from an ssd drive is probably not problematic, but if you have something like a firestore collection of 500000 documents (where you pay for each read) you definitely don’t want to fetch all those docs only to show the “newest 10” in the panel.
So regardless of from where I access the collection, if I write:

$mypages->first()

it should load 0 docs.

If I write

echo $mypages->first()->title()

it should load 1 doc, not all 500000

If I write

echo $mypages->filter('myvar', '==', 'uniqueval')->first()->name()

it should pass that filter, which could look something like this:

{ 
  filter: [
    [ 'myvar', '==', 'uniqueval']
  ],
  limit: 1,
  offset: 0
}

to a data handler (which, for example, was passed to the Pages::factory function). Then that handler would probably do a query and return a single doc in a collection.

At the moment, only to “have” the $mypages variable I need to load all 500000 docs