Data from internet sources

[PeterF]

Dear Tom,

some central banks like the Fed or the Deutsche Bundesbank provide time series, which could be downloaded as a CSV file. Also Yahoo Finance provides the possibility to download data in the CSV file format. Of course, it is possible to download the data, store the file at the hard drive and then read the data into RATS. However, this is rather cumbersome if it had to be done for a larger number of financial instruments every day. Thus, my question is whether it is possible to open a URL with the Open Data "URL" command and then read the data with a Read command directly into RATS without storing it at the pc?

Best regards

Peter

[TomDoan]

Not at this point. That's on the wish list for a future release.

[PeterF]

As the professional version of RATS already allows to retrieve data directly from the FRED database over the internet, it should not be too difficult to implement the basic features. However, as the structures of various CSV data files available over the internet are different, it would be sufficient to give the user the flexibility to write the necessary code for retrieving the data from a particular site.

[PeterF]

I found a way to load data from some sources directly from the website into RATS.

As Tom noted above, it is not possible to use the Open Data "Filename" command to load data directly from a URL. However, a small trick is doing the job. A short-time ago, I moved a data file into a sub-folder and the command OPEN DATA in a RPF file opened the Windows form for opening a file. Remembering that applications like Excel can read the data from a URL location, if the URL is inserted into the filename field. Thus, I split the open command in two steps. In the first step, I executed only the OPEN DATA command with providing the filename (URL) in the command line. After the Open File window popped up, I inserted the URL manually (copy & paste) into the filename field and pressed the open button in the second step. Then I continued as usual with a DATA command to read the data into the series.

I tested this procedure for macroeconomic data (new orders) for Germany from the website of the Deutsche Bundesbank and for the Euroarea from the European Central bank. It worked perfectly. A small code example is provided. Of course, it would be great if RATS would accept a URL as file location in the "filename" argument of the Open Data command directly and thus making the second manual step obsolete.

Code: Select all

CALENDAR(M) 1991:1
* Import German New Manufacturing Order Index directly from the Deutsche Bundesbank

open(immediate) data
* after executing the command above, the window for selecting file to open pops-up, insert the URL below in the file name field
* http://www.bundesbank.de/cae/servlet/StatisticDownload?tsId=BBDE1.M.DE.Y.BAA1.A2P000000.G.C.I10.A&its_csvFormat=en&its_fileFormat=csv&mode=its
data(format=cdf,nolabels,org=col,skiplines=5,left=2,right=2) 1991:1 * neword

table

* now import the New Order time series for the eurozone from the European Central Bank (ECB) server
open(immediate) data
* after executing the command above, the window for selecting file to open pops-up, insert the URL below in the file name field
* http://sdw.ecb.europa.eu/export.do?SERIES_KEY=132.STS.M.I7.Y.ORDT.NSC002.3.000&periodSortOrder=ASC&node=2120800&exportType=csv
data(format=cdf,nolabels,org=col,skiplines=5,left=2,right=2) 1995:1 * ezneword

table

GRAPH(STYLE=LINE) 2
# EZNEWORD
# Neword

This procedure should work also with other internet sites, like yahoo finance or Quandl.com, which provide data for download in a file format that RATS can read.

Best regards
PeterF

[TomDoan]

Thank you for your detective work. I was somewhat confused about how that managed to work when just open data "http://..." doesn't. After running it through the debugger, it appears that entering the URL in the file dialog causes Windows to download a temporary file, and then returns not the URL that you entered but the name of the temporary file.

[PeterF]

Tom,

i did not use a debugger but made a similar experience. I used the Data(other formats) wizard and inserted the "http://..." URL as file name. RATS opens the file like it were stored on a hard drive. The wizard continues with the next window to get more information about the file format and date settings. The code written by the wizard contains Open Data "path and name of temporary internet file".

Reading the data directly from a website makes data management a lot easier, especially for data which is subject to frequent revisions like many macroeconomic data. And as a user, I am not limited to data available at FRED.

best regards
PeterF