Re: [kDev] Revocation, reification, logging and audit tree...

[Daniel Harris <daniel@xxxxxxxxxxxxx>]

Hi Kirit and All,

Our data could be comprised of anything from freeform to structured. 
So, we need a solution that can cope with both these extremes and 
anywhere in between. Seeing that RDF type triples can get really fine 
grained how do we represent them in our database? A triple store? So, 
are you saying we could have a triple store for unstructured data and 
metadata structured tables side by side?

What happens if the data starts off being unstructured and moves 
towards being more structured as people add more data, perhaps using 
the same terminology to describe things? Or vice versa, the data 
becomes more unstructured as people make more cross relationships 
between elements of structured data. We just don't know what's going to 
happen with any given data or set of users.

Another way I look at your recommendation is to think of it as 
optimisation. Optimisation of queries. So, you're saying set up a load 
of tables based on the structure of the metadata/data and I'm saying 
that these metadata structures could change as time progresses. But 
what we're really talking about here is how we query the data. We want 
to optimise the query. But the queries may change from day to day and 
from user to user. So, we need to cope with new, unforeseen query 
structures as well as expected query structures.

Our database, in response to an incoming query (external or internal to 
the database), could dynamically create the tables necessary to fulfil 
the query - to optimise performance. We would then cache the tables 
(the structure and the data). The next time the same query came in we'd 
either use the same table or refresh the data in it using the same 
table structure. Apart from the initial table creation this would give 
us better performance, wouldn't it?

And, for good measure, when importing XML, we'll create their 
respective structured tables. And also allow users to pre-define tables 
based on hunches about incoming queries. So, the table cache would hold 
these and dynamically created tables and have rules about time to live 
and first in first out, etc.

I can't believe any of this is original thought. Someone must have 
looked into this. Any ideas who? Has it been implemented? Does it make 
sense? Are there any alternatives?

In order to "think this through" I'm using methods that I'm familiar 
with, such as open discussion, requirement descriptions and simple 
demonstrations. But not much in the way of learnt database theory, I'm 
afraid. The big question is: how would you think this through? What 
methods would you use? Could you do a simple demonstration?

Cheers Daniel

Joke: What advice did people give to the person who invented the wheel? 
;-)

On 26 Oct 2003, at 12:33, Kirit Saelensminde wrote:
> You're not really proposing to store a huge amount of disparate data 
> patterns in just a few tables are you? If you want a database 
> structure like that then maybe putting it into an SQL database isn't 
> what you really want - the performance is going to suck big time the 
> moment you put a few million records in there.