RE: [kDev] Beginning development

[Daniel Harris <daniel@xxxxxxxxxxxxx>]

Hi Scott and All,

At 11:00 15/07/2002 +0100, Scott Switzer wrote:
>Overall Development Plan:  Does a development plan exist?  What about
>functionality specs, and architecture documents?  What is the status of
>Kendra development?  Where can I find this information?  (Kendra.org.uk
>does not have this information)

If it's not on the website then we don't have it yet. All these things 
would be great. I always feel a bit out of my depth here as I've never 
built these types of plans, specs or docs in a formal way. I'm not sure 
what they should contain. I *do* understand that we need these things.

It's difficult to isolate bits of Kendra when talking about development 
status as it's all interlinked and as a coding developer it's important to 
know about what marketing developers are up too (and vice versa). Since the 
launch in mid 2000 we've really been building up an understanding about 
what the Kendra project needs to be in order to fulfill it's goals. The 
goals are set but the methods/tools are not. Given the end goals we need to 
gather requirements from content owners, consumers and service providers to 
find out what they do now and how they'd like to do it better in the 
future. One way to attract people into the project and coax out their 
requirements is to provide them with a little something in return (rather 
than a grandiose promise of a future system to cure all content ills and 
nothing in the short term). The trials provide us with that marketing and 
also allow us to experiment/model with ways for the system to work.

Gathering requirements is key here and the process needs to be easy and 
scale-able. That's where a large part of my thoughts have been going:
[kFW] Needed: Collaborative systems for driving consensus...
http://www.kendra.org.uk/lists/archive/k-framework/msg00024.html
and that's a large reason why I want to set up Kendra Foundation so I can 
put into practice some of the tools we need to get in these requirements. 
My thoughts are that this is an ongoing process though some disagree and I 
welcome that discussion. You should read:
Re: [kGen] Kendra Status?
http://www.kendra.org.uk/lists/archive/k-general/msg00109.html
and the ongoing discussion:
[kGen] Iterative approach
http://www.kendra.org.uk/lists/archive/k-general/msg00110.html

>Development web site:  If development is going to be open source, should
>we develop a PHP web site, and hook it into SourceForge (or has this
>been done already?)  We can leverage the source code control, forums,
>and bandwidth resources of SourceForge this way.  (and, perhaps, attract
>some developers)

Development is open source see:
Re: [kDev] Licences for Kendra stuff...
http://www.kendra.org.uk/lists/archive/k-developers/msg00016.html

SourceForge would be cool or having the functionality would be cool. My 
main concern here is who's going to manage that process? Can they make a 
commitment to stick with the project or find a suitable replacement if they 
leave? If something goes wrong will *I* have to personally clean it up? 
When it comes to "source code control" - I know nothing. In my old age I'm 
getting cautious about putting myself in uncomfortable positions to ensure 
that this project moves forward.

As I've said, it's not fair to ask for this kind of consistent commitment 
from developers not getting paid to work on Kendra (either by their own 
company or whoever). That's another reason why I want to have Kendra 
Foundation so that I can hire at least 1 techy to manage and be responsible 
for the stuff you are talking about.

That's not to say that we can't do those things that your suggesting right 
now. However, while I'm solely *responsible* for this project relying on 
favours for critical functions needs to be kept to an absolute minimum. For 
example, if this wesbite had to be updated via a source code control system 
hosted by SourceForge and I could only modify the wesbite (which I 
continuously do) via SourceForge and we didn't have a contract with 
SourceForge to maintain their service then that's a potential problem. You 
may think this is rather over the top but I call it future damage 
limitation - I can then sleep and night! ;-)

My comments to your "Specific Functionality" list are purely my own and 
don't represent the total of what I think we should be doing in Kendra. 
It's about discussing these issues and finding people's requirement so I 
chip in here as having an interest in all but being an expert in none.

>Generalised Redirect System:  There are two companies that I know of
>which map IP address to physical location, with a >99% accuracy rate:
>Digital Envoy, and another business that starts with Q.  We used this
>technology to determine where requests were coming from, for Digital
>Rights Management, and eCommerce (e.g.  if we only had rights for a
>video in the UK, for example).  We looked into implementing a mini-CDN,
>where this technology would determine whether to stream from our
>European Server, our US Server, or our Asian Server.

So far, we've been talking more about network nearness rather than physical 
location. Network nearness would enable us to provide the file from the 
*best* point in the network taking into account server loads and network 
congestion which may not be the physically nearest server. Physical 
location could help with that too but is definitely needed for rights 
issues but then users would be registered (?) and so, if we belive what 
they tell us, then they'll give us their location.

>Metadata:  I have implemented systems which store customised metadata,
>based on file type.  Depending on the type of metadata, it can either be
>saved inside the metadata file (e.g. RAM file for Real, or ASX file for
>Microsoft), or put directly into the file.  If the metadata is going to
>live directly inside the file, I have found that this is best done on
>the fly, so that the metadata is always up to date.

Sounds good to me.

>Streaming Server:  Is Kendra going to build their own Streaming
>(MMS/RTSP/HTTP) server, or use existing streaming servers?  I have found
>limitations with existing streaming servers, especially in a distributed
>environment.  Specifically, they are not auto-updatable, so that your
>network is not really homogeneous (read: dependable).  Also, it is very
>difficult to do any real-time integration, so that files can be built
>and streamed at the same time (important for DRM and up-to-date
>metadata).  In addition, if you want to implement another protocol (e.g.
>HTTP for downloading files as well), a lot of integration work needs to
>be done.

Kendra is all about making things work together by providing intelligent 
glue. That glue is not only technological but psychological too. There are 
loads of people hours being poured into building streaming servers. If we 
built our own would that best help content owners, consumers and service 
providers? We'd just be another fish in the pond. If we built plugins to 
enable *all* existing streaming servers to operate in a better way - more 
aligned with Kendra's goal - then we'd be feeding the fish and hopefully 
they'd follow the trail of food as we coaxed them to better waters - more 
aligned with Kendra's goal.

I'm not saying we shouldn't build our own streaming server. It may be 
beneficial to do so as a guideline for other streaming server builders out 
there. However, building a complete working content distribution system in 
isolation to other people building complete working content distribution 
systems would just mean we're in competition with them and by definition 
Kendra is not in competition with anyone. - unless they're a megalomaniac 
and want to own everything in which case we just ignore them and they'll 
disappear. I love simple logic! ;-)

>Squid:  I have had success using the open-source Squid proxy server as a
>cache for files on a CDN.  It may be worth using in place of a
>home-grown LRU cache.

Eeeerrr... Cool!

>Integration:  Depending on the high level goals of this project,
>defining which products to integrate with at the start of the project is
>key (I have been burned by this before).  Is the goal of Kendra to be as
>horizontal as possible, or are there products which can be targeted for
>integration first?  Some initial thoughts that come to mind are Digital
>Asset Management systems on the corporate side, and P2P systems on the
>consumer side.

By horizontal I'm taking you to mean work across the field. In which case, 
yes, Kendra is horizontal. It has to relevant to everyone in the content 
value chain (from content owners to consumers and back again). Just did a 
search for "content value chain" on Google - interesting:
http://www.google.com/search?as_epq=content+value+chain&as_sitesearch=www.kendra.org.uk

You are right, we should make a stab at defining which products to 
integrate with first. Which ones will give us the most coverage with the 
community we want to get to at that point. New products will come along. We 
have to remain flexible. Glue that glues all!

>Searching for Content:  Are there any ideas on how to find content on
>the Kendra system?  Perhaps using some sort of searching interface, or
>partnering with a searching portal would be useful.  I am sure that
>there are quite a few sites which allow searching for audio and video
>files.  Is part of the Kendra initiative to sponsor an 'open directory'
>for content?

I'm glad you mentioned interface. My whole thinking on this is to get the 
interfaces sorted first along with the business models and then we start to 
fix some technological glue. Again that's not to say we don't start 
building models (coding software) of what we want the Kendra system to be 
before we have everyone's requirements - let's call these 'trials' - a 
non-threatening word. The trials will pull more people into the project as 
they start to become useful and in turn they'll inform us of their 
requirements and then we'll refine the software and protocols and so on 
until it works.

What you call 'open directory' I've described as 'content space' (go on, do 
a Google search on Kendra). I've likened it to having for content what we 
have for webpages. Any content viewer/browser being able to search/view 
content (movies, music, text or any digital content) on a distributed 
network (Internet) with a common naming system - or, at least, a common set 
of documented naming systems - to cater for "I'm not using your naming 
system" scenarios.

Realising that I've actually said a lot of this before and much of it is up 
on the Kendra website in some form or other. Perhaps putting a search 
engine interface on the site would help...

Cheers Daniel