Log of /trunk/net/net/src/TS3WebFile.cxx

---

Revision 48687 - (view) (download) (as text) (annotate) - [select for diffs]
Modified Mon Feb 25 15:14:20 2013 UTC (22 months, 4 weeks ago) by rdm
File length: 17158 byte(s)
Diff to previous 48397

From Fabio Hernandez:
The class TS3WebFile was modified to support also reading files hosted by
Google using the S3 protocol. Its TFile plugin was also modified to reflect
this. The class TS3HTTPRequest is new.

---

Revision 48397 - (view) (download) (as text) (annotate) - [select for diffs]
Added Wed Jan 23 17:00:50 2013 UTC (2 years ago) by rdm
File length: 15299 byte(s)

From Fabio Hernandez:
The enclosed patch includes a new class and some modifications to existing
classes. The new class TS3WebFile is an extension of TWebFile and belongs
to the net module. I deliberately did not modify the existing TAS3File.
I think that the name TS3WebFile reflects better the fact that this solution
is intended to be generic to several S3 servers and not limited to Amazon's,
in addition to the fact that it actually extends the capabilities of TWebFile.

Compared to the current support of S3 in ROOT (basically the class TAS3File),
the modifications in this patch include the improvements below:

 - add support for using HTTPS : you can use different schemas for specifying
   the underlying transport protocol to use "s3:", "s3http:", "s3https:" 
   ["s3" uses HTTPS]. The current schema, namely "as3:", is supported for
   backwards compatibility.
 - extend support for other S3 service providers that do not offer the
   virtual hosting functionality (currently only Amazon offers this)
 - support the possibility of specifying user credentials on a per-file
   basis or for all S3 files via environment variables
 - honor the "NOPROXY" option when specified in the constructor
 - exploit the capability of the S3 file server to provide partial
   content responses to multi-range HTTP requests

Here are some examples of usages from the end user perspective:

   TFile* f = TFile::Open("s3://s3.amazonaws.com/mybucket/path/to/my/file", "AUTH=<accessKey>:<secretKey> NOPROXY")
   TFile* f = TFile::Open("s3://s3.amazonaws.com/mybucket/path/to/my/file")   // Uses environmental variables for retrieving credentials

Limitations:

 - we cannot efficiently detect that a S3 server is able to respond to
   multi-range HTTP GET requests. Some servers, such as Amazon's, respond
   to such kind of requests with the whole file contents. Other servers,
   such as Huawei's, respond with the exact partial content requested.
   Therefore, I added the possibility of configuring the behavior via the
   ROOT configuration file: the identity of the servers known to correctly
   support multi-range requests is configurable. If the server is known to
   support this feature, ROOT will send multi-range requests, otherwise it
   will issue multiple single-range GET requests, which is also the default
   behavior. 
 - currently the virtual host syntax
      "s3://mybucket.s3.amazonaws.com/path/to/my/file"
   is not supported but can be added if this is considered useful