[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [kDev] Revocation, reification, logging and audit tree...

Hi Kirit and All,

Our data could be comprised of anything from freeform to structured. So, we need a solution that can cope with both these extremes and anywhere in between. Seeing that RDF type triples can get really fine grained how do we represent them in our database? A triple store? So, are you saying we could have a triple store for unstructured data and metadata structured tables side by side?

What happens if the data starts off being unstructured and moves towards being more structured as people add more data, perhaps using the same terminology to describe things? Or vice versa, the data becomes more unstructured as people make more cross relationships between elements of structured data. We just don't know what's going to happen with any given data or set of users.

Another way I look at your recommendation is to think of it as optimisation. Optimisation of queries. So, you're saying set up a load of tables based on the structure of the metadata/data and I'm saying that these metadata structures could change as time progresses. But what we're really talking about here is how we query the data. We want to optimise the query. But the queries may change from day to day and from user to user. So, we need to cope with new, unforeseen query structures as well as expected query structures.

Our database, in response to an incoming query (external or internal to the database), could dynamically create the tables necessary to fulfil the query - to optimise performance. We would then cache the tables (the structure and the data). The next time the same query came in we'd either use the same table or refresh the data in it using the same table structure. Apart from the initial table creation this would give us better performance, wouldn't it?

And, for good measure, when importing XML, we'll create their respective structured tables. And also allow users to pre-define tables based on hunches about incoming queries. So, the table cache would hold these and dynamically created tables and have rules about time to live and first in first out, etc.

I can't believe any of this is original thought. Someone must have looked into this. Any ideas who? Has it been implemented? Does it make sense? Are there any alternatives?

In order to "think this through" I'm using methods that I'm familiar with, such as open discussion, requirement descriptions and simple demonstrations. But not much in the way of learnt database theory, I'm afraid. The big question is: how would you think this through? What methods would you use? Could you do a simple demonstration?

Cheers Daniel

Joke: What advice did people give to the person who invented the wheel? ;-)

On 26 Oct 2003, at 12:33, Kirit Saelensminde wrote:
You're not really proposing to store a huge amount of disparate data patterns in just a few tables are you? If you want a database structure like that then maybe putting it into an SQL database isn't what you really want - the performance is going to suck big time the moment you put a few million records in there.