Log of /trunk/net/net/inc/TWebFile.h
Parent Directory
Revision
48397 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Jan 23 17:00:50 2013 UTC (2 years ago) by
rdm
File length: 4674 byte(s)
Diff to
previous 40671
From Fabio Hernandez:
The enclosed patch includes a new class and some modifications to existing
classes. The new class TS3WebFile is an extension of TWebFile and belongs
to the net module. I deliberately did not modify the existing TAS3File.
I think that the name TS3WebFile reflects better the fact that this solution
is intended to be generic to several S3 servers and not limited to Amazon's,
in addition to the fact that it actually extends the capabilities of TWebFile.
Compared to the current support of S3 in ROOT (basically the class TAS3File),
the modifications in this patch include the improvements below:
- add support for using HTTPS : you can use different schemas for specifying
the underlying transport protocol to use "s3:", "s3http:", "s3https:"
["s3" uses HTTPS]. The current schema, namely "as3:", is supported for
backwards compatibility.
- extend support for other S3 service providers that do not offer the
virtual hosting functionality (currently only Amazon offers this)
- support the possibility of specifying user credentials on a per-file
basis or for all S3 files via environment variables
- honor the "NOPROXY" option when specified in the constructor
- exploit the capability of the S3 file server to provide partial
content responses to multi-range HTTP requests
Here are some examples of usages from the end user perspective:
TFile* f = TFile::Open("s3://s3.amazonaws.com/mybucket/path/to/my/file", "AUTH=<accessKey>:<secretKey> NOPROXY")
TFile* f = TFile::Open("s3://s3.amazonaws.com/mybucket/path/to/my/file") // Uses environmental variables for retrieving credentials
Limitations:
- we cannot efficiently detect that a S3 server is able to respond to
multi-range HTTP GET requests. Some servers, such as Amazon's, respond
to such kind of requests with the whole file contents. Other servers,
such as Huawei's, respond with the exact partial content requested.
Therefore, I added the possibility of configuring the behavior via the
ROOT configuration file: the identity of the servers known to correctly
support multi-range requests is configurable. If the server is known to
support this feature, ROOT will send multi-range requests, otherwise it
will issue multiple single-range GET requests, which is also the default
behavior.
- currently the virtual host syntax
"s3://mybucket.s3.amazonaws.com/path/to/my/file"
is not supported but can be added if this is considered useful
Revision
40671 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 23 12:18:37 2011 UTC (3 years, 5 months ago) by
rdm
File length: 4605 byte(s)
Diff to
previous 34712
From Marcelo Sousa:
add two new classes to read ROOT files from the Amazon S3 and
Google Storage clouds. The access to these two services is via HTTP
requests and hence these two classes derive from TWebFile.
Revision
34712 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Aug 3 09:27:19 2010 UTC (4 years, 5 months ago) by
rdm
File length: 4443 byte(s)
Diff to
previous 34444
totally rewrote redirection code and handle temporary and permanent redirection.
Can now read ATLAS dCache files via HttpDoor.
Revision
34444 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Fri Jul 16 02:17:39 2010 UTC (4 years, 6 months ago) by
pcanal
File length: 4302 byte(s)
Diff to
previous 34385
Avoid doing seeks on the physical file reprensation when it is not absolutely necessary
(i.e. it is not need if we are going to pick the data from our cache).
Introduce a SetOffset to change fOffset (i.e. the logical position in the file) without doing a lseek.
Introduce ReadBuffer(char * buf, Long64_t pos, Int_t len) to avoid the lseek when the data is in the cache.
See <http://savannah.cern.ch/bugs/?69845>
Revision
30365 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Sep 23 09:02:58 2009 UTC (5 years, 4 months ago) by
rdm
File length: 4151 byte(s)
Diff to
previous 29777
Optimizations in TWebFile. Improves performance especially for TTree::Map()
with about 35%. The following was done:
- cache large part for the the request strings (esp avoid recalculating
the auth base64 encoding).
- drastically optimize the reading of the response headers.
Revision
29777 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Thu Aug 13 15:34:46 2009 UTC (5 years, 5 months ago) by
rdm
File length: 3585 byte(s)
Diff to
previous 28645
implement TWebSystem. We can use TSystem::AccessPathName() and GetPathInfo()
to check if a web file exsits and to get its size. Directory browsing is not
implemented yet.
Revision
27461 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon Feb 16 15:58:10 2009 UTC (5 years, 11 months ago) by
rdm
File length: 2838 byte(s)
Diff to
previous 23091
implement proxy support in TWebFile. The proxy URL can be sprcified either
via TWebFile::SetProxy() or via the shell variable http_proxy, as is being
used by wget, e.g.:
export http_proxy=http://pcsalo.cern.ch:3128
To bypass the proxy, the TWebFile ctor (or via TFile::Open()) supports the
option "NOPROXY".
Revision
23091 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Wed Apr 9 15:04:27 2008 UTC (6 years, 9 months ago) by
rdm
File length: 2492 byte(s)
Copied from:
trunk/net/inc/TWebFile.h revision 23090
Diff to
previous 21736
moving the follwing directories to "net":
alien, auth, glite, globusauth, krb5auth, ldap, monalisa, net, netx,
rootd, rpdutils, srputils, xrootd
Revision
15126 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Mon May 22 11:13:33 2006 UTC (8 years, 8 months ago) by
brun
Original Path:
trunk/net/inc/TWebFile.h
File length: 1906 byte(s)
Diff to
previous 14740
Add new class TFilePrefetch.
This class is a class under user control to specify a list of blocks
to be read in one single transaction.
This speeds up considerably the ROOt I/O in case one reads a remote file
and the network latency is high.
The class speeds up also remore I/O on a local area cluster.
Additional developments will follow shortly to take advantage
of the multi-block capability introduced by this class.
The new class is currently used only in TFile and TWebFile.
In TWebFile, the redundant data member fOffset has been removed. It is
already a member of the base class TFile.
Revision
14740 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Apr 18 14:23:20 2006 UTC (8 years, 9 months ago) by
rdm
Original Path:
trunk/net/inc/TWebFile.h
File length: 1948 byte(s)
Diff to
previous 7810
From Gerri:
move non-persistent member fUrl that was in all derived classes to
TFile. It is accessable via the new method GetEndpointUrl() and will
return in the case of xrootd files the url of the node really having
the file (and not the generic redirector host id).
Also added code to better handle the case of many concurrent async
opens (keep a list of async open handlers).
Revision
7810 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 30 13:16:51 2003 UTC (11 years ago) by
brun
Original Path:
trunk/net/inc/TWebFile.h
File length: 2003 byte(s)
Diff to
previous 5776
Mega patch to add support for large files (bigger than 2 GBytes)
----------------------------------------------------------------
==========>NEW VERSION 4.00/00 <===========
Because this new version has a considerable number of changes,
and new important additions by Philippe are scheduled for the coming days
we are starting a new major version 4.
-Support for large files
-Automatic schema evolution for foreign classes
-New data type Double32_t
Large files are currently tested only under Linux with gcc3.2.
Support for other systems will be gradually added in the coming days.
By default under Linux, files are created with the option LARGEFILE.
Note that when creating a Tree, the default maximum size for a Tree
is set to 1.9 GBytes. You can change this default value by calling
TTree::SetMaxTreeSize(Long64_t maxsize)
The implementation of this new feature has required changes in many places.
The files produced by this new version can still be read by old ROOT versions
if their size if smaller than 2 GBytes. And obviously, the new version
can digest old ROOT files.
WARNING: note the following backward incompatibility:
-TSystem::GetPathInfo has a new signature: (also TUnixSystem, TWinNTSystem, etc)
old: GetPathInfo(const char *path, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: GetPathInfo(const char *path, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TFile::SysSeek and TFile::SysStat have a new signature (also TNetFile, TWebFile, etc)
old: Int_t SysSeek(Int_t fd, Long_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long_t *size, Long_t *flags, Long_t *modtime);
new: Long64_t SysSeek(Int_t fd, Long64_t offset, Int_t whence);
Int_t SysStat(Int_t fd, Long_t *id, Long64_t *size, Long_t *flags, Long_t *modtime);
-TTree::SetMaxTreeSize has a new signature:
old: void TTree::SetMaxTreeSize(Int_t maxsize)
new: void TTree::SetMaxTreeSize(Long64_t maxsize)
All references to Seek_t have been replaced by Long64_t.
Revision
5776 -
(
view)
(
download)
(
as text)
(
annotate)
-
[select for diffs]
Modified
Tue Dec 10 02:19:46 2002 UTC (12 years, 1 month ago) by
rdm
Original Path:
trunk/net/inc/TWebFile.h
File length: 1996 byte(s)
Diff to
previous 5555
new method TFile::ReOpen(Option_t *mode). Reopens a file with a different
access mode, like from READ to UPDATE or from NEW, CREATE, RECREATE, UPDATE
to READ. Thus the mode argument can be either "READ" or "UPDATE". The
method returns 0 in case of success, 1 in case mode did not change and
-1 in case of failure.
This form allows you to request diffs between any two revisions of this file.
For each of the two "sides" of the diff,
enter a numeric revision.
