Asp Forum - Parsing CSV file into a database

Nick Hoyle

2/16/2009 2:25:00 PM

How would i go about parsing a csv from the web eg yahoo finance stock
data into my database table called stocks?

I would like this process to happen each time a user logs into there
account so it will update the table with the latest data possible from
the csv file.
--
Posted via http://www.ruby-....

9 Answers

Jesús Gabriel y Galán

2/16/2009 3:42:00 PM

On Mon, Feb 16, 2009 at 3:25 PM, Nick Hoyle <niho28@tiscali.co.uk> wrote:
> How would i go about parsing a csv from the web eg yahoo finance stock
> data into my database table called stocks?

If you have a url to download the csv, you can read it with something
like open-uri.
Then use a CSV library like FasterCSV for csv parsing, and then I
would use a ORM
(Sequel, ActiveRecord or Datamapper come to mind) to create the rows
in your table.

Hope this helps,

Jesus.

Nick Hoyle

2/16/2009 3:58:00 PM

ok thanks thats great, thanks for the quick response. Have you any
sample code or examples that you could possible provide me?

Thanks for your time

Nick

JesÃºs Gabriel y GalÃ¡n wrote:
> On Mon, Feb 16, 2009 at 3:25 PM, Nick Hoyle <niho28@tiscali.co.uk>
> wrote:
>> How would i go about parsing a csv from the web eg yahoo finance stock
>> data into my database table called stocks?
>
> If you have a url to download the csv, you can read it with something
> like open-uri.
> Then use a CSV library like FasterCSV for csv parsing, and then I
> would use a ORM
> (Sequel, ActiveRecord or Datamapper come to mind) to create the rows
> in your table.
>
> Hope this helps,
>
> Jesus.

--
Posted via http://www.ruby-....

James Gray

2/16/2009 4:08:00 PM

On Feb 16, 2009, at 9:57 AM, Nick Hoyle wrote:

> ok thanks thats great, thanks for the quick response. Have you any
> sample code or examples that you could possible provide me?

It's probably as simple as:

require "open-uri"
require "rubygems"
require "faster_csv"
open("url goes here") do |csv|
FCSV.new(csv).each do |row|
# load row into database here...
end
end

Give it a shot and come back with specific questions when you get stuck.

James Edward Gray II

Nick Hoyle

2/16/2009 4:39:00 PM

Ok great! will do

Nick

James Gray wrote:
> On Feb 16, 2009, at 9:57 AM, Nick Hoyle wrote:
>
>> ok thanks thats great, thanks for the quick response. Have you any
>> sample code or examples that you could possible provide me?
>
> It's probably as simple as:
>
> require "open-uri"
> require "rubygems"
> require "faster_csv"
> open("url goes here") do |csv|
> FCSV.new(csv).each do |row|
> # load row into database here...
> end
> end
>
> Give it a shot and come back with specific questions when you get stuck.
>
> James Edward Gray II

--
Posted via http://www.ruby-....

Bosko Ivanisevic

2/16/2009 7:37:00 PM

On Feb 16, 5:39 pm, Nick Hoyle <nih...@tiscali.co.uk> wrote:
> Ok great! will do
>
> Nick
>
>
>
> James Gray wrote:
> > On Feb 16, 2009, at 9:57 AM, Nick Hoyle wrote:
>
> >> ok thanks thats great, thanks for the quick response. Have you any
> >> sample code or examples that you could possible provide me?
>
> > It's probably as simple as:
>
> > require "open-uri"
> > require "rubygems"
> > require "faster_csv"
> > open("url goes here") do |csv|
> > FCSV.new(csv).each do |row|
> > # load row into database here...
> > end
> > end
>
> > Give it a shot and come back with specific questions when you get stuck.
>
> > James Edward Gray II
>
> --
> Posted viahttp://www.ruby-....

Here is a sample for downloading historical data from Yahoo:

def get_historical_data(symbol, startDateString, endDateString)
startDate = Date.parse(startDateString)
endDate = Date.parse(endDateString)
query = "/table.csv?s=#{symbol}&g=d" +
"&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
{startDate.year}" +
"&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
{endDate.year.to_s}"
Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
res = http.get(query)
res.body
}
end

7stud --

2/16/2009 8:35:00 PM

Bosko Ivanisevic wrote:
> On Feb 16, 5:39ï¿½pm, Nick Hoyle <nih...@tiscali.co.uk> wrote:
>> >> sample code or examples that you could possible provide me?
>> > ï¿½ ï¿½end
>>
>> > Give it a shot and come back with specific questions when you get stuck.
>>
>> > James Edward Gray II
>>
>> --
>> Posted viahttp://www.ruby-....
>
> Here is a sample for downloading historical data from Yahoo:
>
> def get_historical_data(symbol, startDateString, endDateString)
> startDate = Date.parse(startDateString)
> endDate = Date.parse(endDateString)
> query = "/table.csv?s=#{symbol}&g=d" +
> "&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
> {startDate.year}" +
> "&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
> {endDate.year.to_s}"
> Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
> res = http.get(query)
> res.body
> }
> end

require 'date'
require 'net/http'

def get_historical_data(symbol, startDateString, endDateString)
startDate = Date.parse(startDateString)
endDate = Date.parse(endDateString)
query = "/table.csv?s=#{symbol}&g=d" +
"&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
{startDate.year}" +
"&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
{endDate.year.to_s}"
Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
res = http.get(query)
res.body
}
end

p get_historical_data("AAPL", "12/28/2008", "1/7/2009")

--output:--
/usr/lib/ruby/1.8/net/http.rb:1556:in `read_status_line': wrong status
line: "Date,Open,High,Low,Close,Volume,Adj Close" (Net::HTTPBadResponse)
from /usr/lib/ruby/1.8/net/http.rb:1538:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:833:in `request'
from /usr/lib/ruby/1.8/net/http.rb:615:in `get'
from r1test.rb:13:in `get_historical_data'
from r1test.rb:12:in `start'
from /usr/lib/ruby/1.8/net/http.rb:324:in `start'
from r1test.rb:12:in `get_historical_data'
from r1test.rb:18

--
Posted via http://www.ruby-....

Bosko Ivanisevic

2/17/2009 7:48:00 AM

On Feb 16, 9:34 pm, 7stud -- <bbxx789_0...@yahoo.com> wrote:
> Bosko Ivanisevic wrote:
> > On Feb 16, 5:39 pm, Nick Hoyle <nih...@tiscali.co.uk> wrote:
> >> >> sample code or examples that you could possible provide me?
> >> > end
>
> >> > Give it a shot and come back with specific questions when you get stuck.
>
> >> > James Edward Gray II
>
> >> --
> >> Posted viahttp://www.ruby-....
>
> > Here is a sample for downloading historical data from Yahoo:
>
> > def get_historical_data(symbol, startDateString, endDateString)
> > startDate = Date.parse(startDateString)
> > endDate = Date.parse(endDateString)
> > query = "/table.csv?s=#{symbol}&g=d" +
> > "&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
> > {startDate.year}" +
> > "&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
> > {endDate.year.to_s}"
> > Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
> > res = http.get(query)
> > res.body
> > }
> > end
>
> require 'date'
> require 'net/http'
>
> def get_historical_data(symbol, startDateString, endDateString)
> startDate = Date.parse(startDateString)
> endDate = Date.parse(endDateString)
> query = "/table.csv?s=#{symbol}&g=d" +
> "&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
> {startDate.year}" +
> "&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
> {endDate.year.to_s}"
> Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
> res = http.get(query)
> res.body
> }
> end
>
> p get_historical_data("AAPL", "12/28/2008", "1/7/2009")
>
> --output:--
> /usr/lib/ruby/1.8/net/http.rb:1556:in `read_status_line': wrong status
> line: "Date,Open,High,Low,Close,Volume,Adj Close" (Net::HTTPBadResponse)
> from /usr/lib/ruby/1.8/net/http.rb:1538:in `read_new'
> from /usr/lib/ruby/1.8/net/http.rb:833:in `request'
> from /usr/lib/ruby/1.8/net/http.rb:615:in `get'
> from r1test.rb:13:in `get_historical_data'
> from r1test.rb:12:in `start'
> from /usr/lib/ruby/1.8/net/http.rb:324:in `start'
> from r1test.rb:12:in `get_historical_data'
> from r1test.rb:18
>
> --
> Posted viahttp://www.ruby-....

Probably cut-paste caused error. Here is my slightly rewritten script
and output:

require 'net/http'
require 'date'

def get_historical_data(symbol, startDateString, endDateString)
startDate = Date.parse(startDateString)
endDate = Date.parse(endDateString)
query = "/table.csv?s=#{symbol}&g=d"
query.concat("&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
{startDate.year}")
query.concat("&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
{endDate.year.to_s}")
Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
res = http.get(query)
res.body
}
end

puts get_historical_data("MSFT", "12/28/2008", "1/7/2009")

Date,Open,High,Low,Close,Volume,Adj Close
2009-01-07,20.19,20.29,19.48,19.51,72709900,19.51
2009-01-06,20.75,21.00,20.61,20.76,58083400,20.76
2009-01-05,20.20,20.67,20.06,20.52,61475200,20.52
2009-01-02,19.53,20.40,19.37,20.33,50084000,20.33
2008-12-31,19.31,19.68,19.27,19.44,46419000,19.44
2008-12-30,19.01,19.49,19.00,19.34,43224100,19.34
2008-12-29,19.15,19.21,18.64,18.96,58512800,18.96

7stud --

2/17/2009 1:15:00 PM

Bosko Ivanisevic wrote:
> Probably cut-paste caused error. Here is my slightly rewritten script
> and output:
>

I get the same error:

require 'net/http'
require 'date'

def get_historical_data(symbol, startDateString, endDateString)
startDate = Date.parse(startDateString)
endDate = Date.parse(endDateString)
query = "/table.csv?s=#{symbol}&g=d"
query.concat("&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
{startDate.year}")
query.concat("&d=#{endDate.month-1}&e=#{endDate.mday}&f=#
{endDate.year.to_s}")
Net::HTTP.start("itable.finance.yahoo.com", 80) { |http|
res = http.get(query)
res.body
}
end

puts get_historical_data("MSFT", "12/28/2008", "1/7/2009")

--output:--

$ ruby r1test.rb
/usr/lib/ruby/1.8/net/http.rb:1556:in `read_status_line': wrong status
line: "Date,Open,High,Low,Close,Volume,Adj Close" (Net::HTTPBadResponse)
from /usr/lib/ruby/1.8/net/http.rb:1538:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:833:in `request'
from /usr/lib/ruby/1.8/net/http.rb:615:in `get'
from r1test.rb:13:in `get_historical_data'
from r1test.rb:12:in `start'
from /usr/lib/ruby/1.8/net/http.rb:324:in `start'
from r1test.rb:12:in `get_historical_data'
from r1test.rb:18

--
Posted via http://www.ruby-....

Siep Korteling

2/17/2009 1:35:00 PM

7stud -- wrote:
> Bosko Ivanisevic wrote:
>> Probably cut-paste caused error. Here is my slightly rewritten script
>> and output:
>>
>
> I get the same error:
>

I hope this resolves the issue:
These 2 lines should be on one line in your editor:
query.concat("&a=#{startDate.month-1}&b=#{startDate.mday}&c=#
{startDate.year}")

And the same is true for the next "query.concat etcetera" lines.

hth,

Siep
--
Posted via http://www.ruby-....

comp.lang.ruby

Parsing CSV file into a database

Nick Hoyle

Jesús Gabriel y Galán

Nick Hoyle

James Gray

Nick Hoyle

Bosko Ivanisevic

7stud --

Bosko Ivanisevic

7stud --

Siep Korteling

x Login to ForumsZone