$title CSV2GDX - Testing the autoRow and autoCol Option (CSV2GDX5,SEQ=750)

$onText
At first, the autoRow and autoCol option are tested particular. Afterwards,
autoCol and autoRow are tested in combination with the useHeader resp. index
option.

Contributor Jan-Erik Justkowiak, March 2018
$offText


* Testing the autoRow option
$onEcho > data.csv
x1,x2
565.0,575.0
25.0,185.0
345.0,750.0
945.0,685.0
845.0,655.0
$offEcho

$onEcho > testAutoRow.gms
* 1. This should fail, since a second dimension must be specified
$log
$log 1. Without autoRow
$call csv2gdx data.csv id=d useHeader=Y values=1,2 trace=0
$ifE errorLevel=0 $abort Error expected: CSV2GDX should have failed!
$log Expected Error: Duplicate records/keys!


* 2. Using the autoRow option, the missing dimension with unique UELs should
*    be generated automatically
$log
$log 2. With autoRow
$call csv2gdx data.csv id=d useheader=Y values=1,2 autoRow=rowPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log

* Check the results
Set dim1 / rowPrefix1*rowPrefix5 /, dim2 / x1*x2 /;
Parameter d
/
rowPrefix1.x1  565.0
rowPrefix1.x2  575.0
rowPrefix2.x1   25.0
rowPrefix2.x2  185.0
rowPrefix3.x1  345.0
rowPrefix3.x2  750.0
rowPrefix4.x1  945.0
rowPrefix4.x2  685.0
rowPrefix5.x1  845.0
rowPrefix5.x2  655.0
/;

$gdxOut temp.gdx
$unLoad dim1 dim2 d
$gdxOut
$offEcho

$call gams testAutoRow.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoRow!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoRow!
*-------------------------------------------------------------------------------



* Testing the autoCol option
$onEcho > data.csv
1,2,3,1,2,3,1,2,3,1,2,3
$offEcho

$onEcho > testAutoCol.gms
* 3. This should fail, since a second dimension must be specified
$log
$log 3. Without autoCol
$call csv2gdx data.csv id=d useHeader=N values=1..lastCol colCount=12 trace=0
$ifE errorLevel=0 $abort Error expected: CSV2GDX should have failed!
$log Expected Error: Multiple value columns require "autoCol=xyz" or "useHeader=Y"!
$log


* 4. Using the autoCol option, the missing dimension with unique UELs should
*    be generated automatically
$log
$log 4. With autoCol
$call csv2gdx data.csv id=d useHeader=N values=1..lastCol colCount=12 autoCol=colPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log

* Check the results
Set dim1 / colPrefix1*colPrefix12 /;
Parameter d
/
colPrefix1  1
colPrefix2  2
colPrefix3  3
colPrefix4  1
colPrefix5  2
colPrefix6  3
colPrefix7  1
colPrefix8  2
colPrefix9  3
colPrefix10 1
colPrefix11 2
colPrefix12 3
/;
$gdxOut temp.gdx
$unLoad dim1 d
$gdxOut
$offEcho

$call gams testAutoCol.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoCol!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoCol!
*-------------------------------------------------------------------------------



* Testing the autoRow and autoCol option in combination
$onEcho > data.csv
1,2,3,4
5,6,7,8
9,10,11,12
13,14,15,16
$offEcho

$onEcho > testAutoRowAutoCol.gms
* 5. This should fail, since there are no dimensions specified
$log
$log 5. Without autoCol and without autoRow
$call csv2gdx data.csv id=d useHeader=N values=1..lastCol colCount=4 trace=0
$ifE errorLevel=0 $abort Error expected: CSV2GDX should have failed!
$log Expected Error: Multiple value columns require "autoCol=xyz" or "useHeader=Y"!
$log


* 6. Using the autoCol and autoRow option, the missing dimensions with unique
*    UELs should be generated automatically
$log
$log 6. With autoCol and autoRow
$call csv2gdx data.csv id=d useHeader=N values=1..lastCol colCount=4 autoCol=colPrefix autoRow=rowPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log

* Check the results
Set dim1 / rowPrefix1*rowPrefix4 /, dim2 / colPrefix1*colPrefix4 /;
Parameter d
/
rowPrefix1.colPrefix1  1
rowPrefix1.colPrefix2  2
rowPrefix1.colPrefix3  3
rowPrefix1.colPrefix4  4
rowPrefix2.colPrefix1  5
rowPrefix2.colPrefix2  6
rowPrefix2.colPrefix3  7
rowPrefix2.colPrefix4  8
rowPrefix3.colPrefix1  9
rowPrefix3.colPrefix2 10
rowPrefix3.colPrefix3 11
rowPrefix3.colPrefix4 12
rowPrefix4.colPrefix1 13
rowPrefix4.colPrefix2 14
rowPrefix4.colPrefix3 15
rowPrefix4.colPrefix4 16
/;
$gdxOut temp.gdx
$unLoad dim1 dim2 d
$gdxOut
$offEcho

$call gams testAutoRowAutoCol.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoRowAutoCol!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoRowAutoCol!
*-------------------------------------------------------------------------------



* Testing the autoRow and index option
$onEcho > data.csv
1,3
1,4
2,5
2,6
$offEcho

$onEcho > testAutoRowIndex.gms
* 7. This should fail, since there are duplicates
$log
$log 7. Without autoRow and with index
$call csv2gdx data.csv id=d useHeader=N index=1 value=lastCol colCount=2 trace=0
$ifE errorLevel=0 $abort Error expected: CSV2GDX should have failed!
$log Expected Error: Duplicate records/keys!
$log


* 8. Using the autoRow option, unique UELs should be generated automatically
*    to prevent duplicate records/keys
$log
$log 8. With autoRow and index
$call csv2gdx data.csv id=d useHeader=N index=1 value=lastCol colCount=2 autoRow=rowPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log

* Check the results
Set dim1 / rowPrefix1*rowPrefix4 /, dim2 / 1*2 /;
Parameter d
/
rowPrefix1.1 3
rowPrefix2.1 4
rowPrefix3.2 5
rowPrefix4.2 6
/;
$gdxOut temp.gdx
$unLoad dim1 dim2 d
$gdxOut
$offEcho

$call gams testAutoRowIndex.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoRowIndex!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoRowIndex!
*-------------------------------------------------------------------------------



* Testing the autoCol and useHeader=Y option
$onEcho > data.csv
1,1,2,2
3,4,5,6
$offEcho

$onEcho > testAutoColuseHeaderY.gms
* 9. This should fail, since there are duplicates
$log
$log 9. Without autoCol and with useHeader=Y
$call csv2gdx data.csv id=d useHeader=Y values=1..lastCol colCount=4 trace=0
$ifE errorLevel=0 $abort Error expected: CSV2GDX should have failed!
$log Expected Error: Duplicate records/keys!
$log


* 10. Using the autoCol option, unique UELs should be generated automatically
*     to prevent duplicate records/keys. The Header row is skipped.
*     Note that lastCol cannot be used in the values option. CSV2GDX maight fail!
$log
$log 10. With autoCol and useHeader=Y
$call csv2gdx data.csv id=d useHeader=Y values=1..4 colCount=4 autoCol=colPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log


* Check the results
Set dim1 / colPrefix1*colPrefix4 /;
Parameter d
/
colPrefix1 3
colPrefix2 4
colPrefix3 5
colPrefix4 6
/;
$gdxOut temp.gdx
$unLoad dim1 d
$gdxOut
$offEcho

$call gams testAutoColuseHeaderY.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoColuseHeaderY!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoColuseHeaderY!
*-------------------------------------------------------------------------------



* Testing the autoCol and useHeader=N option
$onEcho > data.csv
1,1,2,2
$offEcho

$onEcho > testAutoColuseHeaderN.gms
* 11. Using the autoCol option, unique UELs should be generated automatically
*     to prevent duplicate records/keys. If there is a header row, useHeader=N
*     might fail, if the data cannot be interpreted correctly (for the value columns).
$log
$log 11. With autoCol and useHeader=N
$call csv2gdx data.csv id=d useHeader=N values=1..4 colCount=4 autoCol=colPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log


* Check the results
Set dim1 / colPrefix1*colPrefix4 /;
Parameter d
/
colPrefix1 1
colPrefix2 1
colPrefix3 2
colPrefix4 2
/;
$gdxOut temp.gdx
$unLoad dim1 d
$gdxOut
$offEcho

$call gams testAutoColuseHeaderN.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAutoColuseHeaderN!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAutoColuseHeaderN!
*-------------------------------------------------------------------------------



* Testing the autoRow, autoCol, index and useHeader=Y option
$onEcho > data.csv
a,b,b
I,1,5
I,2,6
II,3,7
II,4,8
$offEcho

$onEcho > testAlluseHeaderY.gms
* 12. This should fail, since there are duplicates
$log
$log 12. Without autoCol, autoRow, with index and useHeader=Y
$call csv2gdx data.csv id=d useHeader=Y index=1 values=2,3 colCount=3 trace=0
$ifE errorLevel=0 $abort Error: CSV2GDX should have failed!
$log Expected Error: Duplicate records/keys!
$log


* 13. Using the autoCol and autoRow option, unique UELs should be generated automatically
*     to prevent duplicate records/keys. If there is a Header row, useHeader=N
*     will fail.
$log
$log 13. With autoCol, autoRow, index and useHeader=Y
$call csv2gdx data.csv id=d useHeader=Y index=1 values=2,3 colCount=3 autoRow=rowPrefix autoCol=colPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log


* Check the results
Set dim3 / colPrefix1*colPrefix2 /, dim2 / I, II /, dim1 / rowPrefix1*rowPrefix4 /;
Parameter d
/
rowPrefix1.I.colPrefix1  1
rowPrefix1.I.colPrefix2  5
rowPrefix2.I.colPrefix1  2
rowPrefix2.I.colPrefix2  6
rowPrefix3.II.colPrefix1 3
rowPrefix3.II.colPrefix2 7
rowPrefix4.II.colPrefix1 4
rowPrefix4.II.colPrefix2 8
/;
$gdxOut temp.gdx
$unLoad dim1 dim2 dim3 d
$gdxOut
$offEcho

$call gams testAlluseHeaderY.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAlluseHeaderY!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAlluseHeaderY!
*-------------------------------------------------------------------------------



* Testing the autoRow, autoCol, index and useHeader=N option
$onEcho > data.csv
I,1,5
I,2,6
II,3,7
II,4,8
$offEcho

$onEcho > testAlluseHeaderN.gms
* 14. This should fail, since there are duplicates
$log
$log 14. Without autoCol, autoRow, with index and useHeader=N
$call csv2gdx data.csv id=d useHeader=N index=1 values=2,3 colCount=3 trace=0
$ifE errorLevel=0 $abort Error: CSV2GDX should have failed!
$log Expected Error: Duplicate records/keys!
$log


* 15. Using the autoCol option, unique UELs should be generated automatically
*     to prevent duplicate records/keys.
$log
$log 15. With autoCol, autoRow, index and useHeader=N
$call csv2gdx data.csv id=d useHeader=N index=1 values=2,3 colCount=3 autoRow=rowPrefix autoCol=colPrefix trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded!
$log


* Check the results
Set dim3 / colPrefix1*colPrefix2 /, dim2 / I, II /, dim1 / rowPrefix1*rowPrefix4 /;
Parameter d
/
rowPrefix1.I.colPrefix1  1
rowPrefix1.I.colPrefix2  5
rowPrefix2.I.colPrefix1  2
rowPrefix2.I.colPrefix2  6
rowPrefix3.II.colPrefix1 3
rowPrefix3.II.colPrefix2 7
rowPrefix4.II.colPrefix1 4
rowPrefix4.II.colPrefix2 8
/;
$gdxOut temp.gdx
$unLoad dim1 dim2 dim3 d
$gdxOut
$offEcho

$call gams testAlluseHeaderN.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with testAlluseHeaderN!
$call gdxdiff data.gdx temp.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running testAlluseHeaderN!
*-------------------------------------------------------------------------------