Hi Anton,
Over the weekend I saw that you used an example of data mining in the financial
environment to argue your point of view concerning insertion/deletion
of tree entries.
Well let's persue that a little bit more. So data is coming from providers
like Reuters, Bloomberg. CantorFitz, GL ...... The list can be extended quite
a bit. Now the first thing to wonder about is why so many. Well many
will provide tick data on the same commodity/equity/option but unfortunately
not agree/ have data missing.
That is the real-time aspect of it. In addition you want to store historical
data which will come from again other vendors.
If you want to build the "perfect" database for commodity x, you will most
likely use statistical tests to decide which feed is right and store its
entry in your "perfect" database (who is right is time dependent).
To make a long story short:
1) You will have to store all the incoming information as is, you do not
want to alter the "raw" data
2) Your data used for simulation / prediction is a combination of the
different sources. You might change the selection algorithm over time.
3) Records from vendors changing previous data constitute only a small
part of your data. So why not have besides your usual tree another
tree containing "corrections". Each time you load data from the tree
you check if corrections are available. If the number gets too large
you will create a new tree incorporating them.
I have seen you raise again the point of manpower on the ROOT project but in the
same breath you mention inefficient management of these large-scale experiments.
So what do you want ?
It is only natural that the ROOT team listens closely to the wishes of the
physics community. An easy way to put your mark on the project is to open
a can with programmers at Fortis-bank and implement your vision. Because
let's face it at the moment you and I have a free ride using the ROOT team.
Eddy Offermann
Renaissance Technologies Corp.
Route 25A, East Setauket NY 11733
e-mail: eddy@rentec.com
http://www.rentec.com
This archive was generated by hypermail 2b29 : Tue Jan 01 2002 - 17:50:38 MET