LB Booster
Programming >> Liberty BASIC language >> LB 4.5.0 INPUTCSV 'broken'
http://lbb.conforums.com/index.cgi?board=lblang&action=display&num=1439732563
LB 4.5.0 INPUTCSV 'broken'
Post by Richard Russell on Aug 16th, 2015, 1:42pm

I tried the new INPUTCSV statement in LB 4.5.0 with the test data from the Wikipedia page (listed below) and it doesn't work! Specifically, it seems to get confused when a record contains one or more embedded quote characters (which in a CSV file are doubled-up from " to "").

The whole point of 'quoted' CSV records is that they can contain any characters, including commas, quotes and newlines. It looks to me as though INPUTCSV copes only with the case when commas are included, but that doesn't allow for arbitrary strings.

I am tempted to release a version 3.01 of LBB with the INPUTCSV statement correctly implemented. What do you think?

Richard.

CSV Code:
Year,Make,Model,Description,Price
1997,Ford,E350,"ac, abs, moon",3000.00
1999,Chevy,"Venture ""Extended Edition""","",4900.00
1999,Chevy,"Venture ""Extended Edition, Very Large""",,5000.00
1996,Jeep,Grand Cherokee,"MUST SELL!
air, moon roof, loaded",4799.00
Re: LB 4.5.0 INPUTCSV 'broken'
Post by Richard Russell on Aug 17th, 2015, 08:33am

on Aug 16th, 2015, 1:42pm, Richard Russell wrote:
I am tempted to release a version 3.01 of LBB with the INPUTCSV statement correctly implemented. What do you think?

INPUTCSV is the only new statement in LB 4.5.0 (the other keyword additions are all functions and can easily be supported in LBB by means of a library). It's potentially a useful feature, or at least it would be if it wasn't seriously broken. sad

So in the interests of compatibility, and to give LB users an implementation of INPUTCSV that actually works, I could quite easily add it to LBB. But it would be helpful to know whether anybody else thinks it's worthwhile. undecided

Richard.
Re: LB 4.5.0 INPUTCSV 'broken'
Post by tsh73 on Aug 17th, 2015, 11:07am

Sure working tool is better then no tool.
CSV is pretty much standard exchange format. Last time I mucked around it it was phone book transfer from old phone to new one.

Is where a possibility to specify a delimiter? Here in Russia Excel reads/writes CSV delimited with semicolon ( ; ).
Re: LB 4.5.0 INPUTCSV 'broken'
Post by Richard Russell on Aug 17th, 2015, 1:13pm

on Aug 17th, 2015, 11:07am, tsh73 wrote:
Is there a possibility to specify a delimiter?

Yes, I would like to do that. The use of an alternative delimiter (usually a semicolon) is common in countries which use the 'decimal comma' (Germany for example) and it is what Excel does automatically when running on a PC which is so configured.

I will need to devise a suitable syntax. I was wondering about:

Code:
    inputcsv #file, a, b, c, d, ... ' Comma delimiter
    inputcsv ";", #file, a, b, c, d, ... ' Specified delimiter

Sadly Liberty BASIC provides very poor support for 'internationalisation'. This used to be a common problem with American software, and in the case of LB I'm afraid it still is.

I've tried my best to rectify this in LBB.

Richard.
Re: LB 4.5.0 INPUTCSV 'broken'
Post by net2014 on Aug 17th, 2015, 2:13pm

I agree, I use csv files for transfer between different machines, but my functions seem to be much to slow so a faster fully functioning INPUTCSV would be much appreciated. Any additions/improvements are always most welcome.

Thanks for your time and effort in maintaining LBBooster. smiley
Re: LB 4.5.0 INPUTCSV 'broken'
Post by Richard Russell on Aug 19th, 2015, 12:46pm

on Aug 17th, 2015, 1:13pm, Richard Russell wrote:
I was wondering about:
Code:
    inputcsv #file, a, b, c, d, ... ' Comma delimiter
    inputcsv ";", #file, a, b, c, d, ... ' Specified delimiter

I don't think anybody commented on this proposal. If I receive no objections I will assume it is acceptable; it could of course be used to read a Tab Separated Value (TSV) file:

Code:
    inputcsv chr$(9), #file, a, b, c, d, ... ' TSV file

I also propose to make INPUTCSV accept files with 'Unix-style' line terminations (i.e. LF rather than CRLF). This is a little non-standard for a Windows application but many CSV samples I have found are of that type, and it could significantly simplify using LBB to read CSV files generated on different platforms.

Here are some sources of 'reference' CSV files I have found so far, which I will use to validate INPUTCSV:

https://en.wikipedia.org/wiki/Comma-separated_values
http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm
https://github.com/maxogden/csv-spectrum

Richard.
Re: LB 4.5.0 INPUTCSV 'broken'
Post by tsh73 on Aug 19th, 2015, 6:01pm

Quote:
If I receive no objections I will assume it is acceptable

it looks good so far.
Quote:
I also propose to make INPUTCSV accept files with 'Unix-style' line terminations (i.e. LF rather than CRLF).

I see no problem if it will work on both.
Re: LB 4.5.0 INPUTCSV 'broken'
Post by pierscintilla on Aug 31st, 2015, 08:45am

Every your observation is like a lighthouse!
It would be a very useful function.

Thank you for existing!!

Hi Pier
Re: LB 4.5.0 INPUTCSV 'broken'
Post by Richard Russell on Aug 31st, 2015, 11:17am

on Aug 31st, 2015, 08:45am, pierscintilla wrote:
It would be a very useful function.

LBB v3.01, with the INPUTCSV statement, was released ten days ago! rolleyes

Does anybody know whether Carl has reacted to my report that INPUTCSV in LB 4.5.0 is broken (I posted it on the LB Bug Tracker wiki)? It seems very odd to me that he didn't check it with even the test data in the Wikipedia article, let alone the more comprehensive CSV example data sets that can be readily found on the web.

Richard.
Re: LB 4.5.0 INPUTCSV 'broken'
Post by Mystic on Sep 10th, 2015, 9:45pm

This was a great add Richard! I only wish I would have saw it before I spent yesterday going through a CSV file manually pulling comma's from the data.

Was only 1250 lines and search worked great for it. I had to pull the weeds before my program would properly read in all of the lines so I could manipulate them.

Live and learn. I look forward to using that feature in the future. Looks like I may be converting another file soon.

Many thanks for your continued awesomeness!